Chat models receive one token stream in which system, user, assistant, and tool messages are marked with special formatting. Models are trained to give system instructions priority, but nothing in the architecture enforces a hard boundary between instructions and data, which is the root cause of prompt injection.
Read in this order
- OfficialChat templates (Hugging Face Transformers)Start here: see how roles are flattened into one token stream by a template.
- PaperThe Instruction Hierarchy: Training LLMs to Prioritize Privileged InstructionsThe research on training models to rank instructions by source, and its limits.