Model providers train models to rank system and developer instructions above text found in tool results. Knowing how instruction hierarchy, structured queries and preference tuning work helps you judge vendor claims and pick models for high-risk agents.
Read in this order
- PaperThe Instruction Hierarchy: Training LLMs to Prioritize Privileged InstructionsStart here: the instruction hierarchy idea and how it is trained.
- PaperLessons from Defending Gemini Against Indirect Prompt InjectionsLessons from a vendor defending a production model, including what failed.
- PaperSecAlign: Defending Against Prompt Injection with Preference OptimizationPreference tuning against injection, with measured attack success rates.