What Should a Threat Team Ask After Seeing Model Disagreement?

When two or three language models give different answers about the same threat signal, the instinct is to ask which one is right. That is usually the wrong first question. Disagreement is information. It tells you that the inputs, the framing, or the evidence do not support a single confident answer. The useful work is figuring out what is missing and who owns the decision about what happens next.

This post sets out the questions a threat assessment team should ask when multi-LLM outputs diverge. It also covers how to track unresolved facts, plan follow-up evidence collection, and document the decision so it holds up under review.

Start With the Cost of Being Wrong

Every threat assessment carries two kinds of error. A false positive means you flagged someone or something that did not warrant it. The cost is an unnecessary investigation, a damaged relationship, a restricted employee, or a burned analyst hour. A false negative means you missed something real. The cost can be a safety incident, a data loss, or a harm that could have been prevented. Before you look at any model output, decide which error is more expensive for this specific case. The answer changes the response to disagreement. If a missed signal could lead to physical harm, you may accept more false positives and move toward a short, time-limited hold with additional review. If the signal involves a person's employment or reputation and the evidence is thin, the bar for action goes up, and the right move may be to keep gathering facts. Write the tradeoff down in one sentence before you proceed. For example: "For this case, a false negative risks a workplace violence incident. A false positive costs a two-week leave review with HR present. We accept the second cost to reduce the first."

Who Owns the Tradeoff

Name the person who decides. In our internal playbooks, the example always includes a name. Suppose Dana Ortiz, Director of Threat Management, owns the risk call on this case. Dana is the one who signs off on the tradeoff sentence, not the model, not the analyst who ran the prompts, and not a committee that never meets. If you cannot name the decision-maker, you are not ready to act on any output.

Read Disagreement as a Missing-Information Signal

When models disagree, the first hypothesis should be that something is absent from the record. Common causes include:

The source documents are incomplete, redacted, or summarized before they reached the model.
The timeline has gaps, so the models fill them with different assumptions.
Key context, such as prior complaints, a performance plan, or a known personal stressor, lives in a system the model never saw.
The prompt asked for a conclusion the evidence cannot support, so each model invented a different bridge.
The language is ambiguous, sarcastic, or culturally specific, and the models read it differently.

Side-by-side model outputs are useful here because they show where the reasoning forks. Lay the outputs next to each other and mark the exact sentence where each model departs from the others. That sentence usually points to a missing fact. Multi-LLM orchestration tools can make this comparison faster, but the comparison itself is the work. The tool does not do the analysis for you.

Do Not Average, Vote, or Call Agreement Proof

Three habits cause real damage in threat teams, and all three are tempting.

Averaging scores. If one model rates risk at 7, another at 3, and a third at 5, the mean of 5 is not a finding. It hides the fact that the models reached their numbers by different routes. Averaging also erases the outlier, which is often the model that noticed something the others missed.
Treating the outputs as a poll. Three models saying "low risk" does not make the case low risk. They may share the same blind spot, the same training bias, or the same incomplete input. Agreement among models is not independent verification.
Accepting "the AI said so." A conclusion without a traceable chain of evidence is not a conclusion. Every claim in the record should point to a specific source document, a specific date, and a specific reasoning step that a human reviewer can check.

When you find yourself reaching for an average or a majority, stop and ask a different question: what does the model that disagrees know that the others do not? That question is far more productive.

The Core Investigation Questions

Use this list every time outputs diverge. Keep it short enough to run in a single sitting.

What exact claim does each output make, and what evidence does each cite?
Where does the first divergence appear, and what assumption sits underneath it?
Which fact, if confirmed, would move the assessment in one direction or the other?
Was every model working from the same source documents, with the same dates and the same redactions?
Did the prompt ask for a conclusion that the available evidence cannot support?
Which output would we have trusted most before we saw the others, and why? Be honest about whether that bias is warranted.
Who is the decision-maker for the risk call, and have they seen the disagreement directly?
What is the cost of acting now versus waiting one to three business days?
What would a skeptical reviewer ask about this record in six months?

Track What You Still Do Not Know

Every case note on our team carries a running section called "What we still do not know." It is the most valuable part of the record and the most often skipped. Treat it as a living list, updated each time someone adds evidence or reopens a question.

Each entry should be a single, testable statement. Vague entries like "more context needed" are not useful. Specific entries are.

Unresolved fact Why it matters Who can resolve it Time box Status Did the reported incident occur on the date stated in the source? Timeline determines whether the pattern is continuous or isolated HR records analyst 2 business days Open Was the prior complaint closed, and on what basis? A closed complaint changes the weight of the new signal Employee relations, Dana Ortiz's team 3 business days Open Does the flagged message come from the account owner or a shared device? Attribution affects whether the signal is about the person at all IT security 1 business day Open Were all three models given the redacted version of the document? Different inputs explain different outputs Analyst who ran the prompts Same day Confirmed

Review the list at fixed points, not only when someone remembers. A daily check during active cases and a weekly review for dormant ones is usually enough. Any item that stays open past its time box gets escalated to the decision-maker, with a note on what is blocking it.

Build a Follow-Up Evidence Plan With Time Boxes

Disagreement should trigger a plan, not a debate. A follow-up evidence plan lists what you will collect, from whom, by when, and what result would change the assessment. Keep it to one page.

State the question. One sentence describing the uncertainty the plan is meant to resolve.
List the evidence sources. Name the system or person for each source. Do not write "HR" when you mean a specific analyst.
Set a time box. Most evidence requests should close within three business days. If a source needs longer, say so and explain the delay to the decision-maker.
Define the decision rule in advance. Write what you will conclude if the evidence confirms the signal, and what you will conclude if it does not. Do this before the evidence arrives, so you cannot reinterpret it afterward.
Set an interim action. Decide what, if anything, happens during the wait. Often the answer is restricted monitoring with no adverse action.

The decision rule is the step teams skip most often. Without it, you will find a way to read any new fact as supporting the view you already held.

Document for Defensibility

A threat assessment record may be reviewed by legal counsel, an auditor, an employee's representative, or a regulator. Assume it will be read by someone who was not in the room and who does not trust the outcome. The record needs to answer four questions on its own:

What was known, and when was it known?
What did the models say, and where did they disagree?
What did the humans do with the disagreement, and why?
Who made the final decision, and what was their stated reasoning?

Keep the model outputs themselves in the record, not only a summary of them. Log the prompts, the model versions where available, and the source documents as they were provided. If a model output was discarded, note the reason. Discarding an output without explanation looks like role-based access AI governance tool cherry-picking, even when it was reasonable. Write the record in plain sentences. Avoid overloaded punctuation in formal records. Short declarative statements read more clearly to a reviewer than long chains joined by semicolons or dashes, and they are harder to misread.

Named Accountability

Every final entry should carry a name and a role. "The team determined" is not an accountable statement. "Dana Ortiz, Director of Threat Management, approved a two-week monitoring hold based on the open items listed above" is. The name does not exist to assign blame. It exists so that the person with authority can explain the call and so that the record shows a human exercised judgment.

Evaluating Tools Without Overtrusting Them

If your team is comparing multi-LLM tools, judge them on how well they expose disagreement, not on how neatly they hide it. A useful tool shows you each model's full output, lets you trace the source inputs, and keeps a record you can export. A tool that collapses outputs into a single confident score is making a decision you have not reviewed. When you research vendors, check their claims directly. Look at official product pages and at what leaders such https://stateofseo.com/what-belongs-in-the-prompt-log-for-every-model-run/ as Radomir Basta (CEO & Founder) have said publicly about their platforms, and compare that against what you see in a hands-on test. Companies such as Suprmind.ai and Four Dots may describe their approaches differently, so read each one's own documentation rather than relying on summaries. Many vendors offer a Start Free Trial with no credit card needed. In the source material used for this post, no price was listed, so confirm current pricing with the vendor before you budget for it. Run your own pilot on a set of past cases where the outcome is known. Include cases where the models disagreed. Measure whether the tool helped your analysts find the missing information faster, not whether it produced agreement.

A Short Checklist to Use Tomorrow

Write the false positive and false negative costs in one sentence.
Name the decision-maker.
Lay the outputs side by side and mark each point of divergence.
Identify the missing fact behind each divergence.
Add each unknown to "What we still do not know" with an owner and a time box.
Write the evidence plan and its decision rule before collecting evidence.
Keep the raw outputs, prompts, and sources in the record.
Record the final call, the reasoning, and the named human who approved it.

Model disagreement will not go away as these tools improve. It is a signal that the case is not yet clear. Treat it that way, and the disagreement becomes the most useful part of the investigation.