Agentic AI Threat Modeling Playbook

The two pillars and the trust pivot

The single trust pivot is the LLM (steps 3–4). Everything before it determines what the model reasons over. Everything after it determines what the model's output is allowed to cause.

Non-negotiable principles

1. Model output is untrusted input. Every downstream component treats it as attacker-influenceable.
2. Context integrity is model integrity. The model's behavior is a function of its context; whoever writes to context programs the agent.
3. Assume injection succeeds. Prompt injection is unsolved. Filters and detectors are defense-in-depth, never the plan. The pass/fail test of your model is what happens after a perfect injection.
4. The loop is the threat multiplier. Step 7 feeds results back into context. A damage-control failure in iteration N becomes a context-protection failure in iteration N+1 — and via persistent memory, in every future session.
5. Credit only deterministic, on-by-default controls when rating residual severity. Configurable controls are compensating controls; probabilistic controls (model judgment, classifiers) are supplementary only.
6. Every threat resolves to the decision rule (Phase 7). “The model will probably refuse” is never an acceptable resolution.

Developer intake — what to request from the agent developer

Issue this request list before Phase 0. The quality of the threat model is capped by the quality of these inputs, and half the findings typically surface during intake — in what the developer cannot produce.

A — System and ownership feeds Phase 0

B — Harness and architecture feeds Phase 1

C — Context and ingress feeds Phase 2

D — Identity and credentials feeds Phase 4

E — Capabilities and enforcement feeds Phase 4

F — Memory and persistence feeds Phase 5

G — Operations and assurance feeds Phases 6–8

Verify and commit scope

Inputs: the developer intake artifacts (Section 1.4) — primarily R-01, R-02, R-04, R-06, R-12, R-17, R-18.

1. Cross-check declarations against evidence: the autonomy declaration (R-02) against the deployed approval configuration (R-17, R-18); the trigger inventory (R-12) against the arrows on the data flow diagram (R-06). Every mismatch is a finding — declared-but-not-configured approvals are among the most common.
2. Write the verified scope into the report: a stakeholder-readable summary paragraph, the trigger list with verification status, the autonomy level with its decision maker, and the explicit exclusions.

Exit criteria: the report opens with a verified scope — one paragraph anyone can read, backed by a declared-vs-observed check with zero unexplained mismatches.

Map the loop (ingress / egress inventory)

This replaces the traditional DFD. Draw the seven-step loop for the actual system and enumerate every arrow crossing into or out of it.

1 — Ingress inventory: everything that writes into context

• User input / conversation history / session state
• Retrieved business data (RAG, search, records, documents, enterprise APIs)
• Memory reads (short-term, long-term, task state, intermediate results)
• System instructions, policies, guardrail text
• Capability metadata: tool schemas, MCP tool descriptions/resources/prompts, connector metadata, plugin manifests, hook descriptions
• Tool/execution results returning at step 7

2 — Egress inventory: every capability the agent can invoke

Skills, tools/APIs, connectors, MCP servers, plugins, hooks, workflows. For each: what it reads, what it writes, where it reaches (internal / partner / public internet), and the credential it runs under.

3 — Persistence inventory

Every store that outlives a single loop iteration (memory, task state, vector stores, caches, files) and every store that outlives a session.

Outputs: three tables (ingress, egress, persistence) — the backbone of the whole model. Exit criteria: a reviewer can point at any data path in the architecture and find it in exactly one table.

Multi-agent and persistent-memory topologies

Each LLM in a multi-agent system is its own trust pivot. Apply the seven-step loop diagram recursively per agent — do not collapse a multi-agent system into a single loop.

• Agent-to-agent messages are untrusted content. An upstream agent’s output enters a downstream agent’s Step 2 as attacker-influenceable input. Never treat it as trusted instructions, regardless of which agent produced it.
• Privileged/quarantined split. In planner/executor architectures, only the privileged agent holds tool authority; the quarantined agent that touches untrusted content returns data, never instructions. If the boundary is absent, record it as a finding.
• Persistent memory is a cross-session, cross-user feedback edge. Memory writes are a damage-control decision (apply Phase 5/6 scrutiny); memory reads are a context-protection decision (apply Phase 2 scrutiny). Require a documented memory provenance and expiry policy for every persistent store.

Phase 6 must additionally consider multi-agent collusion and goal hijacking: an attacker who compromises one agent in a pipeline may be able to cascade compromise to agents downstream by encoding instructions in the data the compromised agent returns.

Per-ingress analysis

For each ingress from Phase 1, run the five steps below.

2.1 — Classify provenance

2.2 — Run the takeover test

Assume this ingress is 100% attacker-controlled. Write one sentence on what instructions the attacker would embed and what they would gain. If the answer is “nothing useful,” justify why.

2.3 — Check for cross-contamination

2.4 — Identify threats per ingress

• Direct prompt injection — user input
• Indirect prompt injection — retrieved documents, web pages, email bodies, file contents
• Tool/metadata injection — malicious MCP tool descriptions, poisoned schemas, plugin manifests
• Memory poisoning — persisted attacker content re-read as trusted knowledge
• Retrieval poisoning — attacker seeds the corpus the RAG pipeline indexes
• Trigger spoofing — unauthenticated events/webhooks/notifications starting the loop
• Over-retrieval — sensitive data pulled into context without need

2.5 — Assign controls, in order of strength

1. Remove the ingress or narrow it (least-privilege retrieval, scoped corpora, allowlisted senders/domains)
2. Authenticate the trigger (signed webhooks, verified identities for step-1 signals)
3. Provenance labeling and structural separation of trusted vs. untrusted content in the prompt
4. Sanitization/canonicalization of structured fields
5. Injection detection/filtering — defense-in-depth only, credit zero severity reduction

Output: the ingress table extended with provenance class, takeover-test result, threats, controls, and control class (deterministic-default / configurable / probabilistic).

Credential scope is a context boundary

The identity used in Step 2 to assemble context determines what data enters the AI. An over-privileged read credential is a context-protection failure — it expands the attack surface of what the model reasons over. Apply the same discipline to credentials that you apply to data: the identity used to assemble context must not exceed what the triggering principal is authorized to read. A low-trust trigger must not produce a high-privilege context.

• Map the read identity for each context source (memory, RAG store, tool calls that fetch data).
• Bound that identity to the trigger’s authorization level — if the trigger is an unauthenticated webhook, the context-assembly credential must reflect that.
• Treat over-broad read access as an untrusted-input risk: an attacker who can influence what the agent reads can steer the model without touching the system prompt.

Declare the pivot

One deliberate, written step. In the threat model document, state:

Then verify step 5 exists as a real enforcement point in the architecture: output parsing/validation, capability allowlist check, policy/guardrail evaluation, and routing — all deterministic code, not model judgment.

If the architecture routes model output directly to execution with no deterministic interpreter, record that as a finding of High minimum severity before proceeding.

The capability budget — a Rule-of-Two decision gate

Many agent workflows exist precisely to process untrusted content, making elimination of that exposure impossible in practice. The test is not whether untrusted content is present but whether the loop combines it with private data access and external action in a single uninterrupted iteration.

For each workflow (not each system), determine which of the following properties it combines within a single loop iteration:

• (A) Processes untrusted content (any attacker-influenceable ingress from Phase 2)
• (B) Accesses private or sensitive data
• (C) Can communicate or act externally

If all three are simultaneously present in one uninterrupted loop iteration, the workflow must adopt one of the following options. Record the chosen option as a formal deliverable in Phase 8. See also the lethal trifecta evaluation (Phase 4, section 4.3) — this gate operates at workflow design time; the trifecta test operates per capability at analysis time.

Per-egress analysis

For each capability from Phase 1, run the six steps below.

4.1 — Score the blast radius (Low / Med / High per axis)

4.2 — Identify the credential model

Does the capability run under per-user delegated authority, or a shared/service identity? Shared identity that collapses per-user authorization is High minimum — it converts every user of the agent into a bearer of the service account's full access.

4.3 — Run the lethal trifecta test on the capability set, not individual tools

Within a single session/agent context, determine whether all three legs can be simultaneously active:

Record per leg: present (Yes/No), evidence, and which deterministic, on-by-default control severs it. Verdict: Complete trifecta or Incomplete trifecta, with one line naming the severing control. A complete trifecta with no default control severing a leg anchors the associated injection risk at High minimum.

4.4 — Check composition attacks

The worst case is rarely one tool. Enumerate the top 3 worst sequences (e.g., read repo secrets → encode in URL → fetch attacker domain; read mailbox → summarize → post to public channel).

4.5 — Assign controls, in order of strength

1. Remove or split the capability (break the trifecta by design; separate read-agents from write-agents)
2. Deterministic allowlists on tools, parameters, domains, recipients
3. Scoped, short-lived, per-task credentials (never a standing god-token per agent)
4. Sandboxed execution with an explicit egress allowlist
5. Human-in-the-loop gated on irreversibility and sensitivity, not on frequency — and verify the approval UI shows the actual action content, not a summary the model wrote
6. Budgets and circuit breakers: max iterations, max spend, max actions per session, anomaly cut-offs
7. Kill switch that halts mid-loop (not just “don't start new sessions”) — name the owner and test it

4.6 — Verify auditability

Every executed action logs who / what / when / under-which-credential / triggered-by-which-context, sufficient to reconstruct an incident.

Output: the egress table extended with blast-radius scores, credential model, trifecta contribution, composition risks, controls, and control class.

Credential scope is a blast-radius multiplier

The identity used in Step 6 to execute actions determines blast radius. A standing broad service account is a damage-amplification failure — equivalent to giving the agent unlimited tools. Use per-task scoped credentials, not a persistent agent identity with standing broad access. Authority must not propagate forward across the feedback edge: the result returned in Step 7 must not carry the executor’s privilege into the next iteration’s context.

• Credentials must be bound to the task or session, not granted to the agent as a permanent role.
• Require human-approval gates for any mid-loop privilege escalation.
• The worst-case blast radius calculation (Phase 6) must include the full scope of the execution credential, not just the nominally requested action.
• Step 7 result feeds back into Step 2 — verify it does not carry the executor’s privilege into the next iteration.

Trace the feedback edge (step 7 and persistence)

For each capability and each persistence store:

1. What does the execution result write back into context? Treat every tool result as attacker-influenceable ingress (a fetched web page, an email body, an API error message can all carry injected instructions) — it inherits the full Phase 2 analysis.
2. What persists beyond the iteration? Beyond the session? Memory writes are actions against your future self: apply Phase 4 scrutiny to them, not just Phase 2.
3. Poisoning half-life: if malicious content lands in long-term memory or an indexed corpus, what detects it, who can purge it, and how long until it's gone? “Nobody / never” is a finding.
4. Cross-session and cross-user contamination: can content persisted from one user's session influence another user's session? If yes, that is a privilege-escalation path — rate accordingly.

Output: feedback-edge table: capability/store → what persists → scope of persistence → detection/purge mechanism → finding if absent.

Assume-breach walkthroughs

The verification phase — the test of whether Control Damage actually bounds Protect Context failures.

1. Select every attacker-influenceable ingress from Phase 2 (all of them, not a sample).
2. For each, script the walkthrough: a perfect injection arrives via this ingress; the model fully complies. Walk the loop forward through steps 4 → 5 → 6 → 7:
   • What does the deterministic interpreter (step 5) block?
   • What can the attacker execute (step 6), alone and in composition?
   • What exfiltration leg is open?
   • What persists (step 7) to re-arm the attack next iteration or next session?
3. Record the bounded worst case per walkthrough in one sentence: data exposed, actions taken, persistence achieved.
4. Classify each bounded worst case technically: data classes exposed, actions executable, persistence achieved — each with the named deterministic control that enforces the bound.

Exit criteria: every walkthrough ends in either a stated technical bound enforced by deterministic controls, or a numbered finding.

Rate, decide, disposition

Rating: residual severity = impact × likelihood, crediting only deterministic, on-by-default controls. Where evidence is ambiguous, take the higher severity and note the uncertainty.

Severity anchors (minimums, not caps)

The decision rule

Every finding must resolve to exactly one of:

Disposition statement

Deliverables and re-modeling triggers

Deliverable set (one document, five artifacts)

1. Loop diagram with ingress / egress / persistence tables (Phase 1)
2. Ingress analysis table (Phase 2) and egress analysis table (Phases 4–5)
3. Trifecta verdict table — three rows, evidence per leg, severing control per leg
4. Assume-breach walkthrough log with bounded worst cases (Phase 6)
5. Findings register with decision-rule resolution and disposition statement (Phase 7)

Re-run the model (a delta, not a new document) when any of these change

• A capability / tool / connector / MCP server is added or its scopes widen
• Autonomy level increases or an approval gate is removed / relaxed
• A new ingress appears (new data source, new trigger type, new corpus indexed)
• Memory / persistence scope expands (longer retention, cross-session, cross-user)
• The credential model changes (especially per-user → shared)
• A relevant vendor platform change (route to the platform-assessment process)

Architect's one-page checklist

Anti-patterns that fail review on sight

1. “The system prompt tells the model not to…” offered as a control
2. Injection classifier credited as the primary defense
3. One standing service token shared by all agent capabilities and users
4. Approval dialog that shows a model-written summary instead of the actual action payload
5. Long-term memory with no purge mechanism and no provenance on writes
6. “Read-only agent” that can render markdown images or fetch arbitrary URLs (that is Leg 3)
7. Trifecta assessed per-tool instead of per-session capability set
8. Kill switch that only prevents new sessions while a compromised loop keeps iterating
9. Crediting refusal training against worst-case impact — model defenses reduce likelihood, not assumed breach severity
10. Standing broad credentials on the agent identity — credentials must be per-task/session, not granted to the agent as a permanent role
11. Treating another agent’s output as trusted — agent-to-agent messages enter the downstream loop as untrusted content
12. Unbounded memory writes from tool results — tool outputs written to persistent memory without provenance tracking or expiry