By Vero SecurityBuilt and funded by Vero Security, the AI Security Engineering platform.Read the charter

Tokenization

Tokenizers such as byte pair encoding split text into subword pieces drawn from a fixed vocabulary. Tokenizer quirks explain many odd model failures and enable attacks like invisible Unicode tag characters, which a human reviewer cannot see but the model still reads.

Read in this order

  1. VideoLet's build the GPT Tokenizer (Andrej Karpathy)Start here: build a tokenizer from scratch and see where the quirks come from.
  2. ArticleASCII Smuggler Tool: Crafting Invisible Text and Decoding Hidden Codes (Embrace The Red)A working attack that hides instructions in invisible characters, plus how to detect it.
  3. Open sourcetiktoken (OpenAI)Try a real tokenizer yourself and inspect how any text gets split.

Save your progress

We email you a sign-in link. No password.

By signing in you agree to the terms. The privacy policy explains how we use your details.

Join the AI security engineering community

Free. Roadmap updates, chapters near you, and first word when credentials open.

We’ll email you community updates. Unsubscribe anytime. By sending this you agree to the terms. The privacy policy explains how we use your details.