Home  /  Blog

Jaxon blog

RAG and AI judges are what teams reach for when an AI check is wrong. Neither one worked.

When a model gets a document check wrong, teams reach for retrieval or a second model as judge. We ran both on the same 65 compliance rules. Retrieval moved the errors from one pile to the other, the best judge

November 19, 2026

LinkedIn
X
Threads

Alec Hoyland, November 2026. Third in a series on the DSAIL benchmark.

Coming in late? Part 1 explains the three metrics, and part 2 scores four models with and without DSAIL.

Previously, we showed DSAIL scoring 21 to 40 points higher on a compliance benchmark than the same models running on a prompt and structured generation alone. A fair reply is that nobody ships a plain prompt. When a model gets a document check wrong, there are two things a team usually reaches for. One is retrieval, usually called RAG: instead of the whole document, show the model only the passages a search finds relevant to each rule, so it has less to get lost in. The other is a second pass, usually called an LLM judge: have a model review the first model's answers and keep or overturn each one.

Both are reasonable ideas, and both are in wide use. We ran them on the same benchmark, with the same models and the same answer key, and measured what they bought.

The test

The task is the one from the earlier posts: 65 rule evaluations drawn from SEAD-4, the directive that governs national security clearance decisions, applied to eight sample background investigation reports. For each rule the system says TRUE, FALSE or UNKNOWN, and over half the correct answers are UNKNOWN, because the reports often say nothing about what a rule asks. Every configuration runs all 65 evaluations five times, and a rule counts as correct only if all five runs got it right, since a real verdict is issued once.

The baseline is a plain prompt: the rules and the whole report in one request, one verdict per rule. We ran retrieval and the judge on the two models that score 100% with DSAIL, (1) GPT 5.6 Sol with no reasoning, which scores 78.5% as a plain prompt and is the best of any model, and (2) Claude Opus 5.5 at low reasoning, which scores 60.0%.

Retrieval: moving the errors around

The retrieval arm splits each report into short overlapping passages. For each rule it shows the model only the passages a keyword search ranks highest against that rule's text, and asks for that one verdict. We varied how many passages it saw, from one to ten, plus a control that sends the whole report but still asks about one rule at a time.

Passages shown per ruleGPT 5.6 SolClaude Opus 5.5
Plain promptBaseline, whole report78.5%60.0%
169.2%69.2%
373.8%69.2%
569.2%72.3%
1069.2%61.5%
Whole reportOne rule per call72.3%66.2%
Correctness on 65 rule evaluations at each retrieval setting, against the plain-prompt baseline.

For the stronger model, retrieval never beat the plain prompt. Every setting scored below 78.5%, and the best of them was almost five points under. For the weaker model it helped, lifting Claude Opus 5.5 from 60.0% to 72.3% at its best setting, which is still below what GPT 5.6 Sol managed with no retrieval at all. The best retrieval result anywhere in the sweep, 73.8%, trailed the best plain prompt.

The reason the number barely moves is that the dial has two ends. With one passage per rule, Claude Opus 5.5 had very little to go on, so it said UNKNOWN a lot: it left 94% of the open questions open and caught 40% of the violations. Shown the whole report, it caught 97.5% of the violations and left 60% of the open questions open, because it had started answering questions the report never addressed. The dial moves errors from one pile to the other. There is no setting where both piles are small.

One passage per rule
Starved of context, it holds back
  • Open questions left open94%
  • Violations caught40%
Cautious, and blind. It says UNKNOWN often enough to look careful, and misses three violations in five.
Whole report
Given everything, it starts deciding
  • Open questions left open60%
  • Violations caught97.5%
Thorough, and overconfident. It catches nearly every violation and answers two in five questions the report never addressed.
Claude Opus 5.5 at both ends of the retrieval dial. Turning it does not shrink the error, it relocates it.

Retrieval also costs more, because it makes one call per rule instead of one per report. Five passes with Claude Opus 5.5 cost $0.69 as a plain prompt and $2.17 with retrieval over the whole report, and model time per pass went from 48 seconds to 83.

The judge: a second opinion with the same blind spot

The judge arm takes the plain prompt's verdicts, the same ones scored as the baseline above, and hands them to a second pass along with the rules and the report. The second pass returns a verdict per rule, keeping or replacing each one. We tried six judges: each base model reviewed (1) by itself at a higher reasoning setting, (2) by the other model and (3) by a majority vote across its own other four runs, with no model involved at all.

Base modelJudgeBeforeAfterChange
GPT 5.6 SolGPT 5.6 Sol78.5%78.5%0
GPT 5.6 SolClaude Opus 5.578.5%61.5%-17.0
GPT 5.6 SolMajority vote78.5%78.5%0
Claude Opus 5.5GPT 5.6 Sol60.0%63.1%+3.1
Claude Opus 5.5Claude Opus 5.5, more reasoning60.0%60.0%0
Claude Opus 5.5Majority vote60.0%64.6%+4.6
Correctness before and after a judging pass, for six judges over the same two base runs.

The best a judge did was 4.6 points, and that judge was a vote, with no model in it. The best a model judge did was 3.1 points, when the stronger model reviewed the weaker one, and the result, 63.1%, was 15 points below what the judge model scores on its own. A team with access to GPT 5.6 Sol would do better to let it do the whole job.

A model reviewing its own work changed almost nothing. GPT 5.6 Sol judging itself touched 6 of 325 verdicts. Claude Opus 5.5 with more reasoning, judging Claude Opus 5.5 with less, touched 12, fixing seven and breaking five. The judge pass cost about as much again as the base run it reviewed, and in the Claude case more, $0.96 to review a $0.69 run.

The one judge that changed a lot made things worse. Claude Opus 5.5 reviewing GPT 5.6 Sol overturned 61 verdicts, fixed 11 of them and broke 49. Of the 49 it broke, 46 were the same move: the base model had correctly said UNKNOWN, and the judge changed it to TRUE. The judge resolved questions the report had not answered, which is the habit behind two thirds of all plain-prompt errors in this benchmark.

Claude Opus 5.5 reviewing GPT 5.6 Sol: 61 verdicts overturned
Correct UNKNOWN changed to TRUEOther overturns
  1. Verdicts the judge broke
    4649
  2. Verdicts the judge fixed
    11
Of the 49 verdicts the judge broke, 46 were a correct UNKNOWN turned into a TRUE.
A second model with the same blind spot is a second opinion in name only.

A second model with the same blind spot is a second opinion in name only.

DSAIL

DSAIL changes the job. The model never sees a rule. It answers a fixed set of factual questions about the report, each of which accepts UNKNOWN, and the DSAIL service applies the rules to those answers with logic that returns UNKNOWN whenever a needed fact is missing. The previous post describes the mechanism in detail. Here is where it lands against everything else.

Correctness, best to worst run in each arm
DSAILPlain promptRetrieval and judge
DSAIL
six models
83.1%100%
Plain prompt
nine runs
38.5%78.5%
Retrieval
twelve runs
61.5%73.8%
LLM judge
six runs
60.0%78.5%
Each bar spans the worst and best run in that arm, on a 38.5% to 100% scale. The worst DSAIL run beats the best of everything else.
ArmBest runWorst run
DSAILSix models100%83.1%
Plain promptNine runs78.5%38.5%
RetrievalTwelve runs73.8%61.5%
LLM judgeSix runs78.5%60.0%
Best and worst correctness in each arm, all on the same 65 rule evaluations.

The worst DSAIL run, with Claude Haiku 5.5 answering the questions, scored 83.1% and cost three cents for five passes. That beats the best plain prompt, the best retrieval setting and the best judge. Four of the six DSAIL runs scored above 95%, and two scored 100% on every rule in every trial. Four of the six also took less model time per pass than any judge run, because answering short factual questions is quicker than deliberating over rule text.

Retrieval changes what the model reads, and the judge changes who reads it. Neither changes what the model is asked to do, which is to decide whether a written rule is met, and that is where the errors come from. DSAIL takes that decision away from the model and gives it to logic that cannot round an unanswered question up to a verdict.

Every run behind this post is published at jaxon.ai/benchmark. When you want to see how DSAIL handles your own rules, we're happy to run it. Contact us.

Prove Everything

LinkedIn
X
Threads