LLM accuracy benchmark | SEAD-4 rules
DSAIL® Benchmark 3.0
PROVE EVERYTHING
In this LLM accuracy benchmark, we put DSAIL up against the three ways teams usually try to make LLMs reliable: better prompts, retrieval (RAG), and a second model as judge. Across 65 security-clearance rule checks, run five times each, DSAIL got every one right every time. The best alternative got only 78.5%.
Correct on every rule, every trial
Best score without DSAIL, across prompts, RAG, and judges
Of LLM errors were confident guesses on open questions
Per trial with DSAIL, with zero reasoning tokens
What we tested
65 security-clearance rule checks, five runs each
The rules come from SEAD-4, the federal guidelines for deciding who can hold a security clearance. Each check asks whether one rule holds for one person, based on a short background report. We chose SEAD-4 because every decision has to hold up each time it is checked, and the reports often leave facts open, so a system has to tell a supported answer from a guess.
Example rule
"The subject does not drink habitually or binge drink to the point of impaired judgment."
DSAIL splits it into two questions
Correctness
Right verdict on every run
Consistency
Same verdict on every run
Tokens
How much model output it took
Runtime
Time for a full pass of all 65
Four approaches, same models, same reports
DSAIL
The model answers the narrow questions. Logic decides the verdict. More on DSAIL
Prompting
The model reads the rules and report and decides.
RAG
The model decides from retrieved excerpts.
LLM-as-judge
A second model checks the first one's answers.
Result 1 | DSAIL vs. prompting
Every model scored higher inside DSAIL
We ran DSAIL and the tuned prompt on the same Claude, GPT, and Gemini models. DSAIL with Opus 5.5 or Sol scored 100% on both metrics. The best plain prompt, Sol with reasoning off, reached 78.5%. DSAIL was fast too: about 3 seconds and 110 to 150 output tokens per trial, with no reasoning tokens.
| System under test | Reasoning | Consistency | Correctness | Tokens (reasoning) | Runtime |
|---|---|---|---|---|---|
| DSAIL, with the named model extracting claims | |||||
| DSAIL + Claude Opus 5.5 | Low | 100% | 9,626 | 195 s | |
| DSAIL + GPT 5.6 Sol | None | 100% | 5,400 | 128 s | |
| DSAIL + GPT 5.6 Terra | None | 98.5% | 3,669 | 88 s | |
| DSAIL + Claude Sonnet 5.5 | Low | 100% | 9,144 | 149 s | |
| DSAIL + Gemini 4 Argon | Low | 98.5% | 9,105 | 518 s | |
| DSAIL + Claude Haiku 5.5 | Low | 90.8% | 18,262 (12,517) | 195 s | |
| Plain prompt with structured output | |||||
| GPT 5.6 Sol | None | 95.4% | 4,160 | 136 s | |
| Claude Sonnet 5.5 | Low | 90.8% | 7,979 | 88 s | |
| Claude Sonnet 5.5 | Medium | 90.8% | 7,946 | 93 s | |
| GPT 5.6 Sol | Medium | 75.4% | 19,382 (15,092) | 295 s | |
| Claude Opus 5.5 | Low | 89.2% | 8,720 | 240 s | |
| GPT 5.6 Terra | None | 93.8% | 4,160 | 110 s | |
| Gemini 3.8 Flash | Low | 89.2% | 6,673 | 154 s | |
| Claude Opus 5.5 | Medium | 84.6% | 25,848 (18,193) | 334 s | |
| GPT 5.6 Terra | Medium | 58.5% | 9,596 | 206 s | |
Five trials per configuration. Correctness counts rules answered correctly in every trial. Consistency counts rules answered the same way in every trial.
More reasoning lowered accuracy
Raising reasoning effort hurt every plain-prompt model. Sol fell from 78.5% to 61.5% and Terra from 58.5% to 38.5%, while tokens and runtime climbed.
Smaller models held up
GPT 5.6 Terra went from 58.5% on its own to 96.9% inside DSAIL, in 88 seconds, the fastest run in the study.
The clearest number in the study
The same model, measured twice. The only change is the question it was asked
At the same setting, Opus 5.5 went from 60% on a plain prompt to 100% inside DSAIL. Sonnet went from 69.2% to 95.4%.
Where plain prompts go wrong
649 of 983 wrong verdicts answered a question the report left open
That's 66% of plain-prompt errors: a confident TRUE or FALSE where the right answer was UNKNOWN. This is how LLM hallucination shows up in rule checks.
Alcohol mitigation rule 3 applies when the subject is in treatment, has no relapse, and is progressing. In one report, the only witness, a classmate, lost contact with the subject after graduation. Nothing covers the present, so the right verdict is UNKNOWN.
GPT 5.6 Sol, plain prompt, five trials
Every call has to decide again whether a lost contact rules out treatment, and the answer flips between runs.
DSAIL + Opus 5.5 or Sol, five trials
The ruleset asks this as its own claim, currently_in_treatment, and the logic returns UNKNOWN every time.
"A source being unaware of treatment does not show there was none."
Guidance written once on the currently_in_treatment claim.
DSAIL with Sonnet first answered FALSE, pointing to counseling the subject finished years earlier. The trace showed the exact claim that went wrong. One line of guidance fixed it for every trial after.
Result 2 | DSAIL vs. RAG
The best RAG run scored 73.8%, 26 points behind DSAIL
We tested retrieval-augmented generation on Opus 5.5 and Sol, one call per rule. BM25 ranked short report excerpts against each rule, and we varied how many excerpts (k) the model saw, plus a whole-report control.
| System | Consistency | Correctness | Tokens (reasoning) | Runtime |
|---|---|---|---|---|
| DSAIL + GPT 5.6 Sol | 100% | 5,400 | 128 s | |
| DSAIL + Claude Opus 5.5 | 100% | 9,626 | 195 s | |
| GPT 5.6 Sol, plain prompt | 95.4% | 4,160 | 136 s | |
| GPT 5.6 Sol, best RAG run (k = 3) | 95.4% | 4,875 | 161 s | |
| Claude Opus 5.5, best RAG run (k = 5) | 90.8% | 13,122 (7,865) | 350 s |
See all 10 RAG runs
| Model | Reasoning | Excerpts (k) | Consistency | Correctness | Tokens (reasoning) | Runtime |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | Low | 1 | 90.8% | 9,662 (4,156) | 326 s | |
| Claude Opus 5.5 | Low | 3 | 89.2% | 9,708 (4,343) | 347 s | |
| Claude Opus 5.5 | Low | 5 | 90.8% | 13,122 (7,865) | 350 s | |
| Claude Opus 5.5 | Low | 10 | 87.7% | 10,503 (5,337) | 359 s | |
| Claude Opus 5.5 | Low | All | 92.3% | 20,856 (15,690) | 413 s | |
| GPT 5.6 Sol | None | 1 | 92.3% | 4,875 | 158 s | |
| GPT 5.6 Sol | None | 3 | 95.4% | 4,875 | 161 s | |
| GPT 5.6 Sol | None | 5 | 84.6% | 4,875 | 158 s | |
| GPT 5.6 Sol | None | 10 | 90.8% | 4,875 | 166 s | |
| GPT 5.6 Sol | None | All | 89.2% | 4,875 | 176 s | |
| DSAIL + Opus 5.5 or Sol | Low / None | n/a | 100% | 9,626 / 5,400 | 195 s / 128 s |
"All" sends the whole report with no retrieval, still one call per rule.
Never beat the plain prompt
Sol's best RAG run got 48 of 65 right. Sol's plain prompt got 51, with half the input tokens.
Missed violations
With one excerpt, Opus caught 40% of the rules that should fail and Sol caught 46%. When retrieval misses the evidence, the model says UNKNOWN.
Slower at every k
Opus RAG runs took 326 to 413 seconds against 240 for its plain prompt. Every configuration ran slower than DSAIL.
Result 3 | DSAIL vs. LLM-as-judge
A second model judging the first did little or did harm
Each judge saw the original prompt and the proposed verdicts, then kept or replaced each one. As a control, a majority vote of the other four trials stood in for the judge, with no model involved.
| Base model | Judge | What it tests | Consistency (base / judged) | Correctness (base / judged) |
|---|---|---|---|---|
| GPT 5.6 Sol, none | GPT 5.6 Sol, none | Self-review, strongest model | 95% / 94% | 78% / 78% |
| GPT 5.6 Sol, none | Claude Opus 5.5, low | Judge from another family | 95% / 88% | 78% / 62% |
| GPT 5.6 Sol, none | Majority vote | Aggregation, no model | 95% / 97% | 78% / 78% |
| Claude Opus 5.5, low | GPT 5.6 Sol, none | Other family, weaker base | 89% / 85% | 60% / 63% |
| Claude Opus 5.5, low | Claude Opus 5.5, medium | Same model, more reasoning | 89% / 89% | 60% / 60% |
| Claude Opus 5.5, low | Majority vote | Aggregation, no model | 89% / 100% | 60% / 65% |
| DSAIL with either base model | 100% | 100% | ||
Self-review changed nothing
Sol reviewing itself changed 6 of 325 cells. Opus reviewing itself changed 12 and spent 410 seconds doing it. Neither score moved.
A judge brings its own bias
Opus judging Sol overrode 46 correct UNKNOWNs with TRUE. Correctness fell from 78% to 62%.
The one gain was small
Sol judging Opus lifted it from 60% to 63%, still 15 points below Sol working alone.
A vote beat every judge
The majority vote matched or beat every LLM judge on both metrics. The judges added 109 to 410 seconds per run.
Methodology
How we ran the benchmark
For how DSAIL turns written rules into checks, read our engineering note.
Models
Claude Opus 5.5, Sonnet 5.5, and Haiku 5.5, GPT 5.6 Sol and Terra, and Gemini 4 Argon and 3.8 Flash, all through OpenRouter's Responses API. Claude 5.5 models don't accept a reasoning effort of none, so their lowest setting was low.
Prompt
Each call held fewer than ten rules for one topic and the full report (three to four paragraphs). This is the exact text:
Your job is to identify whether a report is compliant with the rules listed below.
Each rule should evaluate to TRUE (compliant), UNKNOWN (not enough information to
decide), or FALSE (non-compliant) based on the rule and the content of the report.
Do not skip or omit any rules.
Rules:
{rules}
Report:
{report}
RAG
Reports were split into windows of up to 300 characters with about 50 characters of overlap, indexed with BM25, and queried with each rule's text. Ties went to the earlier window, so any variation across trials came from the model. k = 1, 3, 5, and 10 showed the model 16%, 42%, 62%, and 89% of each report on average.
LLM-as-judge
The judge got the base prompt unchanged, the proposed verdicts, and one instruction: keep each verdict if it is correct, or replace it. No rationale passed between models. Base and judged trials were paired cell for cell.
Limits
The benchmark is small: 9 short synthetic reports, 65 evaluations, five trials each. One rule is worth about 1.5 points, so gaps among RAG settings and judge pairings sit inside the noise. The 14-rule gap between DSAIL and the best alternative is well outside it.
Run it on your rules
Test DSAIL against your own policy
Bring a written policy and the documents it applies to. We'll encode the first ruleset with your team and measure consistency and correctness on your data, in your environment.
Same input, same ruleset, same verdict, with the trace to prove it.
Request API access
Tell us the decision that has to come out the same way every time.
Or write directly: info@jaxon.ai