Token Visualizer
Shows how this encoding splits text: leading spaces, camelCase, CJK. Blocks are OpenAI BPE; Claude counts are scaled. Not a generic token counter.
Enter Text
How this encoding splits text
- • Not a token counter. The colors are cuts. The number is a side effect.
- •
"Hello"and" Hello"are different ids. The space is fused. - • Claude/Gemini rows scale the count. The blocks are still OpenAI BPE.
- • CamelCase, digits, CJK, and emoji fragment. English words often do not.
Features
- Colored blocks are this encoding's slices, not a word or character count
- Hover shows the integer id. "Hello" and " Hello" are different ids
- Claude/Gemini rows scale the count; the cuts you see are still OpenAI BPE
- CamelCase, URLs, digits, and CJK fragment. A counter that only prints N hides that
Common Use Cases
- See why a rhyme or spelling prompt fails: the word is three ids, not one
- Catch a leading space that changed the id before you blame the model
- Compare these splits to the token-counter page, which only prints a number
- Watch emoji and CJK eat more ids than English of the same length
How this encoding splits text, not a generic token counter
A token counter prints one integer. This page shows where this encoding cuts. The colored blocks are OpenAI BPE (the gpt-tokenizer port used here). Common English often lands on one id. A leading space is fused into the next piece, so "Hello" and " Hello" are different ids. CamelCase, URLs, and rare names fall back to subwords or bytes. CJK and emoji usually take more ids than the same number of Latin letters.
The model dropdown changes which published window and tokenizer factor you are thinking about. For Claude and some others, the count is scaled because those vocabularies are denser. The colored boundaries are still OpenAI BPE. If you need Anthropic's actual cuts, this is an estimate of length, not a picture of their tokenizer. That is the failure mode a generic "token visualizer online" copy hides.
Case is part of the id. Apple is not apple. Digits often split. This is not a word counter and not a substitute for the provider's tokenizer in production billing.
Examples
"Hello" vs " Hello"
Same letters. Different token id because the space is fused.tokenVisualizerTool
Usually several ids, not one identifier.東京 vs Tokyo
Expect more ids on the Japanese side in this BPE.Frequently Asked Questions
No. The counter page answers "how many." This page answers "where does this encoding cut." Two strings with the same character length can have different splits. Use the counter for a budget. Use this when a model misspells, fails to rhyme, or treats a leading space as a new token.
The blocks are OpenAI BPE. Claude rows apply a tokenizer factor to the count because Anthropic's vocabulary is denser. The picture is still OpenAI slices. Do not treat the colors as Claude's official tokenizer.
Modern BPE usually attaches a leading space to the next word. " Hello" is not "Hello" plus a space token. That is why copy-paste from a document with a leftover indent changes behavior.
Yes. Token ids are case-sensitive. Apple and apple are different entries. A case change can also turn one id into several.
Tips
- Hover a block for the integer id and the raw string, including the fused leading space.
- If Claude is selected, trust the scaled count more than the colored cuts.