Alec Hoyland, November 2026. Third in a series on the DSAIL benchmark.
Coming in late? Part 1 explains the three metrics, and part 2 scores four models with and without DSAIL.
Previously, we showed DSAIL scoring 21 to 40 points higher on a compliance benchmark than the same models running on a prompt and structured generation alone. A fair reply is that nobody ships a plain prompt. When a model gets a document check wrong, there are two things a team usually reaches for. One is retrieval, usually called RAG: instead of the whole document, show the model only the passages a search finds relevant to each rule, so it has less to get lost in. The other is a second pass, usually called an LLM judge: have a model review the first model's answers and keep or overturn each one.
Both are reasonable ideas, and both are in wide use. We ran them on the same benchmark, with the same models and the same answer key, and measured what they bought.
The test
The task is the one from the earlier posts: 65 rule evaluations drawn from SEAD-4, the directive that governs national security clearance decisions, applied to eight sample background investigation reports. For each rule the system says TRUE, FALSE or UNKNOWN, and over half the correct answers are UNKNOWN, because the reports often say nothing about what a rule asks. Every configuration runs all 65 evaluations five times, and a rule counts as correct only if all five runs got it right, since a real verdict is issued once.
The baseline is a plain prompt: the rules and the whole report in one request, one verdict per rule. We ran retrieval and the judge on the two models that score 100% with DSAIL, (1) GPT 5.6 Sol with no reasoning, which scores 78.5% as a plain prompt and is the best of any model, and (2) Claude Opus 5.5 at low reasoning, which scores 60.0%.
Retrieval: moving the errors around
The retrieval arm splits each report into short overlapping passages. For each rule it shows the model only the passages a keyword search ranks highest against that rule's text, and asks for that one verdict. We varied how many passages it saw, from one to ten, plus a control that sends the whole report but still asks about one rule at a time.
| Passages shown per rule | GPT 5.6 Sol | Claude Opus 5.5 |
|---|---|---|
| Plain promptBaseline, whole report | 78.5% | 60.0% |
| 1 | 69.2% | 69.2% |
| 3 | 73.8% | 69.2% |
| 5 | 69.2% | 72.3% |
| 10 | 69.2% | 61.5% |
| Whole reportOne rule per call | 72.3% | 66.2% |
For the stronger model, retrieval never beat the plain prompt. Every setting scored below 78.5%, and the best of them was almost five points under. For the weaker model it helped, lifting Claude Opus 5.5 from 60.0% to 72.3% at its best setting, which is still below what GPT 5.6 Sol managed with no retrieval at all. The best retrieval result anywhere in the sweep, 73.8%, trailed the best plain prompt.
The reason the number barely moves is that the dial has two ends. With one passage per rule, Claude Opus 5.5 had very little to go on, so it said UNKNOWN a lot: it left 94% of the open questions open and caught 40% of the violations. Shown the whole report, it caught 97.5% of the violations and left 60% of the open questions open, because it had started answering questions the report never addressed. The dial moves errors from one pile to the other. There is no setting where both piles are small.
- Open questions left open94%
- Violations caught40%
- Open questions left open60%
- Violations caught97.5%
Retrieval also costs more, because it makes one call per rule instead of one per report. Five passes with Claude Opus 5.5 cost $0.69 as a plain prompt and $2.17 with retrieval over the whole report, and model time per pass went from 48 seconds to 83.
The judge: a second opinion with the same blind spot
The judge arm takes the plain prompt's verdicts, the same ones scored as the baseline above, and hands them to a second pass along with the rules and the report. The second pass returns a verdict per rule, keeping or replacing each one. We tried six judges: each base model reviewed (1) by itself at a higher reasoning setting, (2) by the other model and (3) by a majority vote across its own other four runs, with no model involved at all.
| Base model | Judge | Before | After | Change |
|---|---|---|---|---|
| GPT 5.6 Sol | GPT 5.6 Sol | 78.5% | 78.5% | 0 |
| GPT 5.6 Sol | Claude Opus 5.5 | 78.5% | 61.5% | -17.0 |
| GPT 5.6 Sol | Majority vote | 78.5% | 78.5% | 0 |
| Claude Opus 5.5 | GPT 5.6 Sol | 60.0% | 63.1% | +3.1 |
| Claude Opus 5.5 | Claude Opus 5.5, more reasoning | 60.0% | 60.0% | 0 |
| Claude Opus 5.5 | Majority vote | 60.0% | 64.6% | +4.6 |
The best a judge did was 4.6 points, and that judge was a vote, with no model in it. The best a model judge did was 3.1 points, when the stronger model reviewed the weaker one, and the result, 63.1%, was 15 points below what the judge model scores on its own. A team with access to GPT 5.6 Sol would do better to let it do the whole job.
A model reviewing its own work changed almost nothing. GPT 5.6 Sol judging itself touched 6 of 325 verdicts. Claude Opus 5.5 with more reasoning, judging Claude Opus 5.5 with less, touched 12, fixing seven and breaking five. The judge pass cost about as much again as the base run it reviewed, and in the Claude case more, $0.96 to review a $0.69 run.
The one judge that changed a lot made things worse. Claude Opus 5.5 reviewing GPT 5.6 Sol overturned 61 verdicts, fixed 11 of them and broke 49. Of the 49 it broke, 46 were the same move: the base model had correctly said UNKNOWN, and the judge changed it to TRUE. The judge resolved questions the report had not answered, which is the habit behind two thirds of all plain-prompt errors in this benchmark.
- Verdicts the judge broke4649
- Verdicts the judge fixed11
A second model with the same blind spot is a second opinion in name only.
DSAIL
DSAIL changes the job. The model never sees a rule. It answers a fixed set of factual questions about the report, each of which accepts UNKNOWN, and the DSAIL service applies the rules to those answers with logic that returns UNKNOWN whenever a needed fact is missing. The previous post describes the mechanism in detail. Here is where it lands against everything else.
six models
nine runs
twelve runs
six runs
| Arm | Best run | Worst run |
|---|---|---|
| DSAILSix models | 100% | 83.1% |
| Plain promptNine runs | 78.5% | 38.5% |
| RetrievalTwelve runs | 73.8% | 61.5% |
| LLM judgeSix runs | 78.5% | 60.0% |
The worst DSAIL run, with Claude Haiku 5.5 answering the questions, scored 83.1% and cost three cents for five passes. That beats the best plain prompt, the best retrieval setting and the best judge. Four of the six DSAIL runs scored above 95%, and two scored 100% on every rule in every trial. Four of the six also took less model time per pass than any judge run, because answering short factual questions is quicker than deliberating over rule text.
Retrieval changes what the model reads, and the judge changes who reads it. Neither changes what the model is asked to do, which is to decide whether a written rule is met, and that is where the errors come from. DSAIL takes that decision away from the model and gives it to logic that cannot round an unanswered question up to a verdict.
Every run behind this post is published at jaxon.ai/benchmark. When you want to see how DSAIL handles your own rules, we're happy to run it. Contact us.
