Tokenizers such as byte pair encoding split text into subword pieces drawn from a fixed vocabulary. Tokenizer quirks explain many odd model failures and enable attacks like invisible Unicode tag characters, which a human reviewer cannot see but the model still reads.
Read in this order
- VideoLet's build the GPT Tokenizer (Andrej Karpathy)Start here: build a tokenizer from scratch and see where the quirks come from.
- ArticleASCII Smuggler Tool: Crafting Invisible Text and Decoding Hidden Codes (Embrace The Red)A working attack that hides instructions in invisible characters, plus how to detect it.
- Open sourcetiktoken (OpenAI)Try a real tokenizer yourself and inspect how any text gets split.