Four Questions to Ask Before You Believe an AI Accuracy Claim

A deep dive on the three numbers behind the DSAIL^® benchmark: what we chose, why, and why it matters.

Alec Hoyland, November 2026

A vendor tells you their compliance check is 80% accurate. Before you can decide whether that is good, you need to know what was counted.

Those are not pedantic questions. Depending on which way you count, the same system can look like it is ready for an auditor or like it is guessing a fifth of the time, and the vendor gets to pick which number goes on the slide.

We ran into every one of those questions building the benchmark for DSAIL. This post is about the three metrics we settled on, correctness, consistency and completeness: how each is computed, why we defined it the way we did, and what each one catches that the others miss. The next two posts put models on the board and score them this way, so this is the part where we show our work before anyone has a result to argue with.

The benchmark

The task is adjudication against SEAD-4, the Security Executive Agent Directive that sets the adjudicative guidelines for national security eligibility. We took the disqualifying and mitigating conditions from four guidelines (financial considerations, alcohol consumption, criminal conduct and allegiance), wrote each condition out as a numbered rule, and paired the rulesets with eight sample background investigation reports. Thirteen pairings, 39 distinct rules, 65 rule evaluations in all. For each rule the system under test says TRUE (the report shows the rule is satisfied), FALSE (the report shows it is not) or UNKNOWN (the report does not say). The main runs repeat all 65 evaluations five times, so one run is a grid of 65 rules by 5 trials, 325 cells total.

Correctness: right in every trial

Correctness is the share of the 65 rule evaluations where every one of the five trials matched the answer key. Four right out of five is zero credit for that rule.

That is harsher than the usual accuracy, which would average over the 325 cells. We chose it because a compliance decision is made once. The adjudicator who receives a verdict does not get the average of five runs. They get one draw, and what they need to know is how often that draw can be trusted no matter which one it was. Averaging hides the rules a model gets right only sometimes, and sometimes is the case that reaches a file.

The adjudicator does not get the average of five runs.

The gap between the two definitions is itself a diagnostic. For a model that answers the same way every time, the two numbers agree. Where they split, the split is instability.

Consistency: the same answer every time, right or wrong

Consistency is the share of the 65 rules where all five trials gave the same verdict. It never looks at the answer key. A compliance system needs to be consistent in order to be trustworthy. An incorrect system can be fixed, but an inconsistent system is unreliable in a way that is hard to diagnose and harder to trust.

Completeness: did your system answer every rule?

Completeness quantifies the percentage of rules that we actually answered. For example, if 65 rules were provided, 50/65 rules answered (and 15 ignored/forgotten) would result in a completeness score of roughly 77%. Given that we only provided less than ten rules at a time in this benchmark to the systems under test, completeness scores all measured at 100%, but in the future, we intend to scale this benchmark to hundreds of rule evaluations, showing how DSAIL scales algorithmically to large-scale compliance workflows without filling up the context window.

What to ask of a number

Go back to the vendor’s 80%. The four questions at the top of this post are really these three metrics, plus one that sits underneath all of them. (1) Is that 80% of trials or 80% of rules, and was any rule run more than once? (2) When the same document goes in twice, how often does the same verdict come out? (3) Did the system return a verdict for every rule it was given, or only the ones it got to?

None of the three can stand in for the others. Correctness scored strictly tells you what one verdict is worth, and averaged over trials it tells you something softer that is easy to mistake for the same thing. Consistency tells you whether the system is the same system from one run to the next, and it will report a perfect score for a system that is reliably wrong. Completeness tells you whether the first two numbers were computed over the whole job or over the part the system got around to. Read one without the other two and you are trusting the vendor to have picked the flattering one.

There is a fourth question hiding inside the first: what counts as right when the document doesn’t provide enough information? We made UNKNOWN a verdict with the same standing as TRUE and FALSE, and we scored it against an answer key that says UNKNOWN wherever the report doesn’t say. That choice matters because filling a gap with a guess is the error a compliance system is least allowed to make, and a two-valued score cannot see it at all. A system that is never permitted to say “the report does not say” will be graded on how convincingly it guessed. That is what the next post measures. Same 65 rules, same eight reports, four models, and a look at what they do when the report does not say.