AI Model Validation

What is DEEM?

A check-engine light for AI models.

DEEM (Detect Errors & Evaluate Models) checks that a model still meets the standard it was approved against. Write the standard once. DEEM re-runs it every time the model, its data or its sensors change, and flags the moment it stops holding.

Standard under watch
Aerial vehicle classifier
Monitoring
Accuracy at least 90%, judged at 95% confidence
85%Accuracy range100%
v1Before new camera
0.94 [0.920, 0.960]
✓ PASS
v3After new camera
0.91 [0.887, 0.933]
✕ FAIL
White line = the 90% barv3 flagged for recertification

Example from the DEEM demo. Average accuracy still clears 90%.

The problem

Code stays put. Models drift.

An approval is a snapshot. The model that earned it keeps changing after sign-off: it gets retrained, the vendor ships an upgrade, the data underneath it shifts. DEEM turns the approval into a standard that can be checked again on every change.

Day 0
Approved

Tested, signed, authorized to operate.

Day 30
Model retrained

New weights. Same approval on file.

Day 90
Vendor upgrade

The model behind the API changed.

Day 180
Data drift

A new camera. New inputs. No code change.

Today
Still true?

DEEM answers that on every change.

Watch the demo

See DEEM in action.

A recorded walkthrough of DEEM re-checking an approved model after a camera change, from the written standard to the recertification alert. It then shows how DEEM checks an LLM that has no answer key.

Detect Errors & Evaluate Models

You can't trust what you can't formally define.

Generic benchmarks answer someone else's question. DEEM answers yours: is this model still working for your mission, against the definition of success your team wrote and signed?

01

Specify before you trust

Define valid output, error conditions and acceptable fallback behavior before a model is ever tested. Precise enough to enforce, readable enough for a program manager to approve.

02

Assess consistency

Detect drift and evaluate behavior over time, across model versions and across semantically equivalent inputs.

03

Operationalize your standards

Turn formal definitions into automated, auditable, continuous monitoring that travels with the system from development through deployment.

An example

From spec to signal.

Write the standard once. Then every new model version is checked the same way.

Set up once: steps 1 and 2Then, for each model version: steps 3 to 5
Set up once · Step 1

Write the standard

Say what correct looks like, in terms a program manager and an authorizing official can read and sign.

The standard is written in AISL, an open specification language. It becomes the contract the model is held to.

Standard · aerial vehicle classifierAISL
Accuracy floorat least 90%
Judged at95% confidence
Signed off byProgram manager · Authorizing official
FreeOpen sourceOn GitHub
Standard→Test data→v1 PASS→New camera→v3 FAIL · recertify
Verdicts

DEEM doesn't force an answer.

Most evaluation tools return a yes or a no on every run, even when the test set behind the number is too small or too narrow to support one. DEEM reports that instead of picking a side. You find out the result is thin while you can still add test data and run it again, rather than acting on a PASS that was never backed by enough evidence.

✓ PASS

The model meets the standard, even at the lowest likely value.

✕ FAIL

The model no longer meets the standard. It goes back for recertification before the next release.

? INCONCLUSIVE

Not enough evidence to decide. DEEM says so instead of guessing, so you know to collect more test data.

LLM-specific capability

Check consistency when there's no answer key.

Classifying an image has a right answer you can grade against. Summarizing a report or drafting a response does not, so there is nothing to mark the output correct or incorrect. DEEM tests consistency instead: it asks the same question in several wordings, many times each, and measures whether the answers agree.

Aligned answers suggest well-grounded reasoning. Divergent ones point to uncertainty, sensitivity to phrasing, or a badly worded question.

The pass mark is part of the standard you write. DEEM enforces exactly that standard.

4 wordings × 10 runs40 answers
38 of 40 answers are yes; one no in wording B and one in wording D.
YesNo
95%Agreement rate
✓ PASS

Illustrative example: 38 of 40 answers agree.

Why DEEM

Built for high-stakes AI.

Unlike rule-of-thumb metrics, DEEM creates an auditable standard that travels with the system from development through deployment.

Custom definitions of success

Define acceptable output even when ground truth is fuzzy or task-specific.

Consistency checking

Check that semantically equivalent prompts produce aligned responses, before deployment.

System-level drift detection

Get alerted when behavior no longer meets your standard, and prompt recertification.

Side-by-side benchmarking

Compare model variants, prompting strategies and fine-tuning recipes against the same spec.

Continuous validation

Monitor production behavior over time and catch failures and quality issues automatically.

Automated quality assurance

Regression analysis and performance tracking run on their own. Manual review becomes a choice.

Two formal AI languages, one for each job

DEEM watches the model. DSAIL® checks the answer.

AISL · AI Specification Language

Defines what correct behavior looks like

A formal language for the operational boundaries and correctness criteria of an AI system. DEEM is built on it, so every standard is readable, versioned and auditable.

Read the AISL docs
DSAIL® · Domain-Specific AI Language

Proves each output meets your policy

A separate engine with its own language. DEEM judges the model against your standard; DSAIL® judges a single output against your written policy, returning TRUE, FALSE or UNKNOWN with an auditable trace to the rule that decided it.

Explore DSAIL®
Get started

See DEEM on your model.

Bring one model and the standard it was approved against. We'll show you the first re-check.

Runs where your data does: hosted API, on-premises, air-gapped or cloud.