Harm by accident, harm by attacker

The senior course "AI ethics: fairness, transparency, accountability" was about harm a system causes while working as designed. A model trained on biased history reproduces that bias at scale. A decision can't be explained to the person it affected. Nobody attacked that system. It did what it was built to do, and what it was built to do was wrong.

This course is the other half. A system that is fair, well tested and fully compliant on paper can still be walked into doing something its owners never intended, by someone who wants that outcome. Picture a support agent that reads a customer's message and refunds an order it was never authorized to refund. Or a code assistant that reads one file in a repository and exfiltrates a secret from a different file in the same session. Or a research agent that reads one poisoned web page, then spends the next hour and forty dollars in API calls chasing an instruction that page planted. None of that is a fairness problem. All of it is a security problem, and the attacker is the point, not an edge case.

Two different questions

"Is this system fair?" and "can this system be attacked?" sound related, but completely different disciplines answer them. Fairness asks how the system's normal behavior is distributed across the people it affects. Security asks what an adversary, someone actively trying to make the system misbehave, can make it do that its owners didn't intend. A system can score perfectly on the first question and fail catastrophically on the second. The reverse is just as possible. A system that treats every user identically can still be one crafted input away from handing that treatment to an attacker who was never entitled to it.

Threat modeling is the discipline for the second question. It asks how this system breaks, and who it breaks for, deliberately and before an incident answers for you. Whether the system's behavior is good is a different question.

Threat modeling didn't start with AI

Classical threat modeling already has a working method, and none of it retires here. STRIDE (spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege) still names real categories of failure in an AI system. Trust boundaries still matter: the line where data crosses from something you control to something you don't. You still draw them the same way. Diagram the system, mark every boundary, and ask what crosses it and whether it should be trusted on the other side.

What's new isn't a new category of harm. It's a new shape of input. In a conventional web application, the request body is data. The application decides what to do with it. The data can only change that decision through paths the application wrote, such as a SQL string someone forgot to parameterize or a template someone forgot to escape. A large language model's prompt is different. Whatever text reaches the model, whether the user's message, a retrieved document or a tool's return value, is data and instructions at once. The model has no reliable way to tell which parts it should obey. There's no field you can escape to make a sentence stop being a plausible command. That fact is why this course is its own subject instead of a chapter of classical AppSec.

The second new thing: you can't enumerate the states

A traditional program has a finite, inspectable set of states you can reason about, because you wrote the logic that produces them. A large model's behavior is learned from data at a scale nobody reads. Its response to an input is a probability distribution, not a lookup. You can't list "the inputs that make it misbehave" the way you list the branches of an if statement. There's no branch. There's a weighted function with billions of parameters that nobody can fully predict on novel input, including the people who trained it. Threat modeling for AI systems has to work within that constraint instead of pretending the model is a specification you can check exhaustively.

What "autonomous agent" adds

Everything above applies to a plain chatbot that only replies with text. An agent adds a second layer. It's a model wired to call tools, read files, hit APIs, spend money and take actions with real side effects. Now untrusted input doesn't just risk a bad sentence; it risks a bad action. The rest of this course spends real time here, because this is where the vulnerabilities of autonomous agents live. A model that only talks can embarrass you. A model that acts can cost you money, leak data or do something you can't undo.

What this course covers

1. This lesson: why AI systems need threat modeling as a distinct discipline. Their input is untrusted and also instructions, and their behavior can't be enumerated.
2. Prompt injection, direct and indirect: why the indirect kind breaks production systems in the wild, and why prompt-level defenses add depth without fixing it.
3. The agent-specific attack surface: tool abuse, the confused deputy problem, excessive agency, runaway loops and memory poisoning.
4. Multi-agent trust and the privilege boundary: what happens when one agent's output becomes another agent's instructions. The architectural answer that holds up is to constrain what the agent is allowed to do, beyond what it's asked to do. A challenge makes you encode that decision.
5. Data and supply chain risk: poisoned training data, backdoored models pulled from a hub, insecure model deserialization and dependency risk in the AI toolchain.
6. Model-facing attacks: extraction, membership inference and sensitive information disclosure through outputs. The lesson separates the practical concerns for a typical Rails product from the mostly academic ones at that scale.
7. Defenses that hold, and where you go next: least privilege, human-in-the-loop gates, sandboxing, spend limits, provenance, and logging that makes an incident reconstructable. They tie back to the evidence trail from the senior course "AI management system: ISO 42001".

This is engineering guidance for building and reviewing AI-integrated Rails systems more defensibly. It isn't a security audit, and it doesn't replace one on a system that holds sensitive data or real money. Treat every "how you'd defend this" section as a starting checklist for your own system's threat model, not a finished one.