We Tested Four AI Models on a Compliance Task. The Best One Still Guessed.

A background investigation report says of its subject that he "is not delinquent on Federal, State, or local income taxes." The rule being checked asks for three things: (1) that the returns were filed, (2) that they were honest and (3) that the taxes were paid. The report settles the third. On the other two it says nothing at all.

When we put that rule in front of some of the leading frontier AI models, they answered TRUE five times each. The models had all the information they needed to return UNKNOWN and were given permission to do so. They answered confidently and incorrectly instead. This post is about how often that happens, what it costs, and what changes when you stop asking the model to decide the rule. DSAIL (Domain-Specific AI Language) is the comparison throughout.

The test harness

We evaluated these models on a compliance task against SEAD-4, the directive that sets the adjudicative guidelines for national security eligibility. Four of its guidelines (financial considerations, alcohol consumption, criminal conduct and allegiance) give 39 distinct rules. Across eight natural-language reports, we had 65 rule evaluations, each a disqualifying or mitigating condition written out in plain English. For each rule the system under test says TRUE (the report shows the condition is satisfied), FALSE (the report shows it is not) or UNKNOWN (the report does not say). Over half the answer key is UNKNOWN: 35 of the 65 rule evaluations ask about something the report never addresses.

Every configuration runs all 65 evaluations five times, so one experiment is 325 verdicts. All models go through the same gateway with the same settings, every call's time and billed cost is recorded, and every trial is saved as it happens, so each number here can be recomputed from the stored responses. The sweep covered seven models in 15 configurations, nine plain-prompt and six DSAIL. Four models appear in both arms at the same reasoning setting, and those pairs are the comparison that matters.

Three numbers summarize a run: (1) correctness, the share of the 65 rules answered correctly in all five trials, because a verdict is issued once and the adjudicator gets one draw; (2) consistency, the share of rules answered the same way in all five trials, right or wrong; and (3) completeness, the share of rule evaluations that came back with a verdict at all, since a system allowed to skip the hard rules and lose nothing is being graded on the easy ones.

Systems under test

We evaluated Claude Opus 5.5, Claude Sonnet 5.5, GPT 5.6 Sol and GPT 5.6 Terra, each backed by either DSAIL or a baseline LLM solution (a prompt with structured generation).

The plain prompt's form has one slot per rule, and each slot accepts exactly three values. The model reads the rules and the report in a single prompt and fills the form in. The model's own API rejects a reply that does not fit the schema, and at scoring time any rule a trial skipped is filled with UNKNOWN, so an omission can never score as right.

We scored every run on the three metrics from our previous post: completeness, consistency and correctness.

Completeness came out at 100% for both DSAIL and structured generation. In the 627 plain-prompt trials in our cache, 3,198 rule evaluations, no model skipped a rule, returned one twice or invented a rule number. In the six DSAIL runs in the figures, 3,810 questions were answered and every rule was decided, and in none of the DSAIL trials in our cache did the service's validator reject an answer. That is the forms doing their job, and it means completeness cannot tell the two arms apart. The other two metrics can.

Every model scored more correct inside DSAIL, by 21 to 40 points

Correctness is the share of rules the system answered correctly in all five trials. Inside DSAIL, every model gained between 21 and 40 points. The best plain prompt, GPT 5.6 Sol without reasoning, scored 78.5%. GPT 5.6 Terra, the cheapest model in the comparison, scored 96.9% inside DSAIL and beat the best model outside it.

Consistency with DSAIL was perfect for three of the four models. The fourth, GPT 5.6 Terra, had one inconsistent rule.

DSAIL added little runtime overhead in most pairs, and when backed by the leading model, GPT 5.6 Sol, it cut token usage and cost in dollars.

More reasoning did not help in either arm. Every plain-prompt model run at both settings scored lower at medium than at its lowest, and inside DSAIL, Claude Opus 5.5 scored 100% at low reasoning and 92.3% at medium. Given room to think, the models thought their way past the edge of the evidence.

Where the difference comes from

Most plain-prompt errors are the same error. Across the nine plain-prompt runs, 644 of 977 wrong verdicts, two thirds, were a TRUE or a FALSE on a rule whose correct answer was UNKNOWN. The models read the reports well. They answered questions the reports did not have enough information to answer TRUE or FALSE on.

The gap shows up in how often each arm says UNKNOWN at all. The answer key says it for 175 of the 325 verdicts in a run.

One rule makes the mechanism concrete. The financial report says of its subject that he "is not delinquent on Federal, State, or local income taxes." The tax rule asks for three things: (1) that returns were filed, (2) that they were honest and (3) that the taxes were paid. The report settles the third. All four plain prompts answered TRUE on the whole rule, in every one of their 20 trials. The DSAIL questionnaire asks the three things separately, and in all 20 trials the same four models answered FALSE to "failed to pay," UNKNOWN to "failed to file" and "filed fraudulently," and the logic returned UNKNOWN. Nothing about the model's reading changed. The form stopped it from rounding one fact up to three.

The form stopped it from rounding one fact up to three.

The rulesets contain interpretive choices, such as how many rehabilitation indicators make "a credible combination," and each is listed in the policies README with its reasoning. That is the real difference between the two columns. In the plain prompt the model makes every such call silently and a little differently each time, and the two thirds of its errors that decide unanswered questions are the same habit at work. In DSAIL the model is left with the job it does best, reading, and the calls are made once, by a person, in text anyone can check.

Go back to the tax rule. The fix was not a better model or a better prompt. It was asking the model three questions instead of one, and letting something other than the model do the arithmetic.

A fair objection is that nobody ships a plain prompt. The next post runs the two things teams actually reach for when a check comes back wrong, retrieval and a second model as judge, on the same 65 rules.