Bop

Tokenizer

A byte-level BPE tokenizer with exact HuggingFace tokenizers semantics. Construct with Tokenizer(path) (= from_file), from_pretrained, or from_gguf; use with encode and decode.

Immutable after loading and safe to share across tasks and threads (the piece cache is task-local).

from_file(path) -> Tokenizer

Load a tokenizer from a HuggingFace tokenizer.json file. Components outside the supported byte-level-BPE subset error loudly at load — nothing mis-tokenizes silently. Tokenizer(path) is a shorthand.

from_pretrained(repo; revision = "main")

Load a tokenizer from a HuggingFace Hub repo's tokenizer.json (e.g. from_pretrained("Qwen/Qwen3-0.6B")). Sends ENV["HF_TOKEN"] as a bearer token if set (needed for gated repos). No local caching.

from_json(parsed) -> Tokenizer

Build a tokenizer from an already-parsed tokenizer.json object, as returned by JSON.parse (or JSON.lazy).

encode(tokenizer, text; add_special_tokens = true) -> Encoding

Tokenize text exactly as HuggingFace tokenizers would: added/special tokens are extracted first, the remaining segments are normalized, pre-tokenized, and BPE-merged. With add_special_tokens = true (the default) the post-processor template (e.g. a BOS prefix) is applied.

text may also be a raw byte buffer (AbstractVector{UInt8}, interpreted as UTF-8), encoded without copying.

decode(tokenizer, ids; skip_special_tokens = true) -> String

Map (0-based) token ids back to text. Special tokens are omitted unless skip_special_tokens = false. Unknown ids error.

Encoding

The result of encode. enc.ids holds the token ids (0-based, matching HF); enc.tokens the corresponding token strings, materialized lazily on first access.

encode_batch(tokenizer, texts; kwargs...) -> Vector{Encoding}

encode element-wise; keyword arguments are forwarded.

decode_batch(tokenizer, batches; kwargs...) -> Vector{String}

decode element-wise; keyword arguments are forwarded.

from_gguf(path_or_metadata) -> Tokenizer

Load a tokenizer directly from GGUF metadata — either a path to a .gguf file or a metadata dict from gguf_metadata. Requires tokenizer.ggml.model == "gpt2" (byte-level BPE) and a tokenizer.ggml.pre name present in PRE_TOKENIZERS; anything else errors loudly.

gguf_metadata(path) -> Dict{String,Any}

Read the metadata key-values of a GGUF file. Tensor data is never touched — reading stops after the KV section, so a file truncated there parses fine.

PRE_TOKENIZERS :: Dict{String,PreSpec}

The GGUF pre-tokenizer name table. GGUF stores a name (e.g. "qwen35") where tokenizer.json embeds the actual split pattern; this maps the names llama.cpp's converter emits to their PreSpecs. Names are bound via llama.cpp's tokenizer-checksum table (hence "dbrx" covering Phi-4 and OLMo-2, and "gpt-4o" covering gpt-oss); every entry is generated from the family's tokenizer.json and pinned by tests against the paired asset. To support a new family, add its entry and a differential test against that family's tokenizer.json.

Note: GGUF cannot carry added-token lstrip/rstrip (Phi-4 sets them in tokenizer.json) — the GGUF path diverges from HF there, as does llama.cpp itself.

PreSpec(splits, normalizer, ignore_merges, use_regex)

Pre-tokenization recipe for one GGUF tokenizer.ggml.pre name: splits is a vector of (pattern, keep) Split stages (keep ∈ (:both, :gaps, :matches)), normalizer an optional Unicode form (e.g. :NFC), and use_regex selects the built-in GPT-2 ByteLevel pattern instead of explicit splits. Fields must be transcribed verbatim from the family's tokenizer.json.