← back to hub ↑ parent

Tokenizer Visualization {{ tokenizerEngine }}

Paste text to see how different tokenizers segment it.

Tokens {{ tokenizerTokens.length }} total · {{ tokenizerWordCount }} words · {{ tokenizerCharCount }} chars

{{ i }} {{ tok.text }} #{{ tok.id }}
Tokens will appear here.