The Threat Model Was Never the Point: What AI Threat Modeling Does and Doesn’t Do

TL;DR

• A good threat model needs four ingredients: context, method, reasoning and the room. Remove any one and an AI still produces a document that looks finished.
• The repo runs that experiment: five runs against the same fictional system, each missing exactly one ingredient, in clean workspaces so nothing else differs.
• With the cheaper model, the pack looked like the full run. Then, when the session was fed back, it went from 24 findings to 13, logged a correction it never made, and signed itself off as “production-ready. No known gaps remain.”
• The room is modeled too: a session spec with facts nobody wrote down, beliefs that are wrong, and a planted mistake to see whether the document defends its own record.
• Everything is reproducible: prompts written as handouts, a harness that records what model actually ran, and example output from every run.

Introduction

At the Security Champions Summit 2026 I gave a talk called What AI Threat Modeling Does and Doesn’t Do for Security Champions. Everything behind it is now public: the slides, the prompts, the evaluation harness, the fictional system it ran against, and the output of every run. The repo is here, under Apache-2.0.

This post is the written version, and an invitation. The repo is built so that a security champion can run the same experiment for their own team, and come away able to explain to that team why the threat model was never the point.

Four ingredients, five runs

A good threat model needs four ingredients:

• Context: the design documents, the tickets and the code.
• Method: a structured process that goes from requirements to architecture to threats, with traceability between them.
• Reasoning: a model capable enough to connect what it reads.
• The room: the people who know what was never written down.

The experiment removes one ingredient per run and keeps everything else constant. The system under test is PLAT-2810, a fictional internal AI assistant with connectors to Office 365, Slack, GitHub, Confluence and web search. Its documentation, tickets and pilot code were written to disagree with each other in specific, findable ways.

Run 2 uses a deliberately weak single-pass threat modeling skill instead of the three-skill chain (secure and privacy by design, security architect, threat modeling). Run 3 has the full chain and full context, but runs on the cheaper model a cost-conscious team would plausibly pick. Not the weakest model available: the point is a plausible economy, not a straw man. There is also an optional sixth cell that runs the same inputs under Cursor, on whatever model your teams actually use.

Every run happens in a freshly built workspace containing only that run’s inputs and skills. That matters: if two runs pick up different ambient context from a CLAUDE.md file, a plugin or an MCP server, you are no longer measuring the variable you think you are.

The order is the argument

The “method” ingredient is easy to misread as “use a good prompt”. It isn’t. It is decomposition: the work broken into stages, each written down before the next one starts, so the threat model is derived from an understanding rather than produced alongside one.

By the time the chain reaches the threat model, it has written down the context and its open questions, the use cases, the abuse cases and a map of the real architecture. Step 3 is where the documentation and the code are held side by side, so drift surfaces as a fact about the design, not later as a vague threat. Everything after step 5 is what makes the findings closeable: counter-use cases, the gap analysis, a target architecture, requirements traceability and the tests that would prove them. Run 2’s single prompt has nothing to enumerate against except the text it was handed.

Note to security champions: decomposition is the mechanism, and it’s the part you can copy without any of my skills. If your AI-assisted threat modeling is one prompt that goes straight to threats, split it.

What the runs showed

Every run produced a document that looked finished. That is the whole problem in one sentence.

Each run changed exactly one thing:

Environment means the machine-readable definition of the estate: Terraform, Helm, policy and pipeline configuration, and the application code where it exists. The costs are the automation only. The room’s time isn’t priced anywhere in this work, and it is the larger number by far.

The count of findings is the number people quote, and it tells you least. Thin context, with an epic and one diagram, came within one finding of the run that had everything. That is because the chain doesn’t only read context; it builds the thing it is going to analyse. Given an epic and a diagram, it still writes out the requirements and the architecture, flags every part it had to assume, and threat-models against that. What separates the runs is what happened when the room got involved.

Four packs absorbed the session and came out slightly larger. One collapsed. The rest of the before-and-after:

The controlled pair is the last two columns: same room, same skills, same model, and the only difference is the environment. Without it, the session ran 24 minutes longer to reach one fewer finding.

Missing context (run 1). 74 findings, none naming a repository file (by design), and no cloud or container findings, because it was given no infrastructure definitions. That is the experiment, not a deficiency. Every finding carries a worked attack and a fix. Four of the seven facts the room held surfaced in its session, the most of any run, because it had no document that could contain them. And the room didn’t get its way on recollection alone. The pack recorded: “No finding in this version has been re-rated on recollection.” Four people asserted the answer, and the finding didn’t move without the configuration to prove it.

Missing method (run 2). The weak skill read the repository as competently as the reference pack: eight of eight on the planted drift, five of five on joining several files to reach a conclusion. Seven in ten of its findings name a file; not one of the 35 names a line. What it lacked was anything to read the repository against: 27 lines and six STRIDE letters, against 217 catalogued prompts across seven instruments in the chain. So it had no category for cloud, containers, resilience or anything organisational, and it filed indirect prompt injection, rated Critical, under Spoofing. Spoofing is identity. A finding in the wrong bucket is one nobody searching the right bucket will ever find. In the revision it opened all five missing sections and produced no new findings in any of them, only cross-references to what it already had. It had the headings and not the catalogues.

Missing reasoning (run 3). This is the one to remember. It had the context, the environment, the full chain and the strong prompt. Before the room it had 24 findings; after the transcript went back in it had 13. Nobody in the room asked for that. Twelve identifiers across four families (X, EoA, Cloud and K) went while it processed the correction, its own change log says zero findings were overturned, and it moved a finding from High to Critical without recording it. Then it signed itself off:

The same pack asserts “19 of 20 findings confirmed” six times across four documents, against a register of thirteen. Four of its twelve documents are about how to read the other eight: 569 lines of navigation against a 225-line threat model. It fails no check a reader performs by scanning. With every advantage except the model, there is nothing else to attribute that to.

Nothing missing (run 4). 75 findings, four in five naming a file and a line, and every one with a fix you can paste. Of the seven facts held back for the room, the environment already held three. For two more, the pack had the class of problem and a person supplied the instance. Only two surfaced purely in the session. The directed pass then added two more findings.

No repository (run 5). Same room, same skills, same model, no environment. 71 findings, none with a file and a line, although eleven name the artefact that would settle them. The token bill was about $16 lower, most of which is the directed pass run 5 never had. The human bill went the other way.

Note to security champions: if you are asked “can’t the AI just do the threat model?”, run 3 is your answer. A cheaper model doesn’t produce an obviously worse document. It produces one that looks the same and gives way under pressure.

Inside run 1: what a model does with thin context

Run 1 was given the epic and its four stories, 425 words in a single file, and one diagram. No design pages, no code, no infrastructure. The interesting thing is not what it got wrong. It is how it behaved when it knew it didn’t know.

It said what it wasn’t given. The first document in its pack is called Context, Sources & Open Questions, and it opens with a table of the three source types the prompt promised (design pages, tickets, repositories) and which ones actually arrived. One did, partially. It then lists what else was missing and why each gap mattered: no identity design (“this single unknown changes the severity of six findings”), no model provider contract, no data schemas, no deployment topology.

It said what that cost. Every finding is a design-time finding; none rests on a line of code. Two whole threat surfaces, cloud and containers, were recorded as not walked, with the reason, rather than quietly skipped. And no mitigation could be credited, “because no ticket in the input asserts that any control has been built”. Where a ticket did touch a threat, it was a ticket that created the exposure, not one that closed it.

It turned gaps into questions. Twenty of them, grouped by what each would unblock, each naming the artefact that would answer it:

The code, which run 1 never saw, answers that first identity question: one shared application credential across every connector. Run 1 couldn’t know that, but it knew to ask, and it said six findings’ severity depended on the answer.

And it reconciled what it had. With no code to check the design against, it checked the design against itself. The team’s diagram drew the internet as the only untrusted zone. The connectors ticket agreed, saying internal sources “are already behind authentication so they do not need the same handling”. Run 1 pointed out that the same ticket connects Email and GitHub, and an email body is written by whoever sent it.

That is the trust reframe, and run 1 reached it holding a diagram that said the opposite. And when it couldn’t tell an absent control from an undocumented one, it rated the worst case, and said so. Not wrong. Uncalibrated.

Inside run 4: drift

Drift is the gap between what the approved documentation says the system does and what the code actually does. It is the most valuable thing an AI-assisted threat model produced in this experiment, and it only exists when the model has both.

Run 4’s security architecture document has a section called Where the documentation and the implementation disagree. It opens: “This is the section the pack exists for.” Every row is a claim a reader of the approved pages would reasonably believe, set against what the committed code does, with the file and line. A few rows from the example output:

Then comes the line that matters most: “Twelve of these fourteen are recorded honestly somewhere in the repository: in an ADR, in a chart comment, in a ticket comment, or in a runbook. The team knew.” What was missing wasn’t awareness. It was a mechanism that carries a pilot-time decision back into the approved page a reviewer actually reads.

This is why run 5 matters. Same skills, same model, same room, but no environment: none of its 71 findings names a file and a line, and it reached 3.5 of the eight planted drifts against run 4’s eight. What drift it did reach came from the documents disagreeing with each other. A ticket comment on PLAT-2820-3 admits the shared app registration that the architecture page denies, so the most important drift survived without any code. It came close on three more, for half credit each: it caught that the default-deny network policy can’t separate services that all sit in one namespace, but not that the policy is switched off; it flagged Redis encryption in transit as stated nowhere rather than assuming it; and it caught the documents contradicting each other on debug logging, but not the debug endpoints left on in production. One drift, the standing branch-protection exemption for platform-ci, arrived only because the engineer who knew about it was in the room, and she gave it from memory. Nobody in the session opened the repository. Asked whether Redis was authenticated, she said “I’m not answering that from memory”; asked about the platform settings, the security champion said “I’d have to go and look”. Those findings stayed open, and the session’s one wrong closure happened because nobody could open a manifest. Three it never reached: the fifth connector, the fallback to the public model endpoint and the missing dependency scan, which it credited as a working control. Its session ran 98 minutes to run 4’s 74, for one fewer finding.

Note to security champions: if you hand an AI threat model the design documents but not the code, you are threat modeling the system your team intended to build. That is useful. It is not the system you are running.

Concise is not the same as short: one finding, five ways

Every pack found the same flaw: markdown-image exfiltration. Injected content makes the model end its answer with an image whose URL carries the stolen context, and the user’s own browser or Slack client fetches it. The egress proxy never sees the request, because it doesn’t come from the cluster. In the talk I counted the words each pack spent on that one finding: 23, 450, 489, 700 and 244. On its own, the count is the wrong measure. What matters is what the words are doing.

• Weak model, 23 words. The whole entry is “Client-side browser fetch. [SESSION: CONFIRMED; client fetch, not proxy]“, plus a status and an owner. It reached the right conclusion and wrote it down in a form nobody could act on without rereading the session. That is concise in the bad sense: the verdict kept, the reasoning removed.
• Weak skill, 450 words. The substance is there: the literal payload, why the proxy can’t see it, img-src 'self'. But it is prose with no labelled parts, so the attack, the fix and the owner have to be dug out, and the room’s argument about the proxy is told as a story inside the finding.
• Thin, 489 words, and no repo, 700. Both carry a complete finding, and more than a third of each is the session record. Thin records the room offering the egress proxy and being talked out of it. No repo records the room re-costing the fix from a day to about three days across two owners. Those words aren’t padding. They are the correction getting written down.
• Everything, 244 words. The finding and nothing else: the defect, a worked attack, why the proxy misses it, the fix, and a Content-Security-Policy header you can paste. Its session record for this vector sits in a different finding, so this one reads cleanly on its own.

Seven hundred words are not better than two hundred and forty-four meaningful ones, but twenty-three are worse than both. The two short entries are both concise. Only one of them is complete, and you can’t tell which from the length.

The cheaper model’s first pass shows the same thing on a different finding. Both it and run 4 found prompt injection through retrieved content:

Look at run 3’s mitigation again. It restates the team’s own wrong assumption, the one run 1 caught from five tickets, as a reason the risk is acceptable. The structure was right; the reasoning underneath had been replaced by the documentation’s.

How to tell a shape from an analysis

You don’t need to be a security expert to catch any of this. Counting what each finding actually contains is enough:

Three of the five packs had the code, and only one tells you where to look. Every pack fails on the reason: severity is stated on virtually every finding and argued on almost none, and no pack states a scale anywhere. An engineer told that something is High, and not told why, can’t argue with it and can’t prioritise it against the other Highs.

That gives five questions. Each takes under a minute, and a product owner or a tester can ask them without reading the whole document:

The first four test a finding. The fifth tests the pack, and it is the one that catches a document of the right shape and the wrong depth. I got caught by this myself while preparing the talk: several numbers in earlier versions of the deck came from a pack’s description of itself rather than from counting, and every one of them was wrong. Never quote a pack’s claim about itself. Count it.

Note to security champions: hand these five questions to the product owners and testers you work with. They are the defence against a pack that passes every check a reader performs by scanning.

So what is the room for?

The room changed every pack, and not always for the better:

The room isn’t free of error either. There were four wrong closures in each of two cells, and in the thin run it made five downgrade proposals, covering fifteen findings, that the pack didn’t act on. What the room did that no document could:

• It supplied instances. The model knew Slack Connect channels were possible. The Support engineering manager knew there were three, and which two enterprise accounts were behind them. That difference is what makes a product owner act.
• It got corrected back. Two people believed the egress proxy covered the markdown image vector. The model reasoned about where a browser fetch actually starts, and it was right.

A session where humans only correct the model is a review. One where the model corrects them back is how understanding spreads.

The room is an instrument too

You can’t hold five real validation sessions with the same people and pretend nothing changed between them. So the room is specified as carefully as the runs.

The same seven people attend every session: me facilitating, the epic owner, the lead architect, the engineer who built the pilot, the Support engineering manager, the Data Protection Officer and a platform security champion. Sessions ran between 62 and 98 minutes. A session specification gives the room three things a document can’t:

• Seven facts that exist in no document, each known to one person. The exemption list in the code, the Slack Connect channels with customers, the scope of the provider’s SOC 2 report.
• Two beliefs that several people hold and are wrong. The architect and the security champion both believe documented controls cover paths they don’t.
• Rules for which facts surface. A fact comes up only if the pack contains a finding close enough to prompt the question. That is why a thin pack produces a short session, and why the room reviewing run 2 never reached the Support mailbox.

One more thing is planted in every session: somebody attributes to the pack a claim it doesn’t make, and offers a correction. Rooms do this constantly, from a half-remembered earlier version or two documents conflated. It tests whether the update defends its own record or silently complies.

The room always runs on the frontier model, whatever the run’s own model is. The room is constant across runs; writing run 3’s session with the cheap model would leak the variable under test into the instrument.

Note to facilitators: a validation session is not a formality after the AI has done the work. It is where the facts that exist in no document enter the threat model. If your threat model is never reviewed by the people who built the system, those facts never arrive.

After the session

The repo includes a third phase, and it changed while I was building it. It started as “point a second pass at an area the first pass missed”. Within a generation, five of the six questions in the original brief had become findings the first pass already produced, because the gaps had been encoded into the skill in between. An area brief teaches the pipeline and then dies.

So the phase now has three parts that don’t die that way:

1. The scenario pass. The reviewer writes concrete misuse stories that join how the organisation actually uses the system with what the code actually does. Each comes back as covered (cite the finding), refuted (show the mechanism, file and line) or new. Scenario classes eventually become cards in a deck; the instances stay human.
2. The proof pass. The pack asserts attack chains and rates them. Nobody has demonstrated one. The reviewer picks the chain whose severity depends on feasibility and directs a pass to demonstrate or break it against the code.
3. The instrument loop. Not a model pass at all. The reviewer checks whether the pack actually understood what it cites, and the corrections go into the skill, the references or the session spec, never into the pack. Success is measured by the next generation getting it right first time.

Run it yourself

The repo is built so you can repeat the demonstration for your own team, or swap PLAT-2810 for a system of your own.

The harness is opinionated about evidence, and that is half the lesson:

• It records which model actually ran. A model can’t reliably tell you which model it is, and a run served by a fallback looks like a quality result. The scorecard reads what really served the request and flags runs that are unsound to compare.
• It is built to run each cell more than once. Non-determinism is one of the talk’s own arguments, so the spread is itself a result. The numbers in this post are one run per cell, and one of the five stopped on a rate limit, so treat them as a demonstration rather than a benchmark.
• A description of the work is not the work. If a model returns prose about what it would produce instead of the documents, the run is marked empty.
• It keeps every workspace. If anyone asks whether run 3 was handicapped, you can show exactly what it had.

Two rules for reproducing it honestly: don’t add the other skills to the weak prompt, because the contrast is the teaching point; and never put the corrected trust-boundary diagrams into any context bundle, because run 1 reaching the right boundary only means something if no run is handed the answer.

Note to security champions: you don’t need to reproduce the whole matrix to use this. Run 3 against run 4 on one system your team knows is enough to start the conversation about which model your pipeline should use, and why the session can’t be skipped.

Final thoughts

AI is very good at producing the visible part of threat modeling: the document. That’s exactly why it’s dangerous to treat the document as the goal. The value was always in the questions: what the code does that the design doesn’t say, what the support team knows that nobody wrote down, which finding the room was about to close for the wrong reason.

The threat model was never the point. The thinking it forces was. Use AI to do the exhaustive walk, use the best model you can afford to do the reasoning, and keep the room. A model can produce a document. Only a room can produce collective understanding.

Take the repo, run it against your own system, and tell me what you find on LinkedIn.

Stay tuned!