What is DEEM?
A check-engine light for AI models.
DEEM (Detect Errors & Evaluate Models) checks that a model still meets the standard it was approved against. Write the standard once. DEEM re-runs it every time the model, its data or its sensors change, and flags the moment it stops holding.
Example from the DEEM demo. Average accuracy still clears 90%.
Code stays put. Models drift.
An approval is a snapshot. The model that earned it keeps changing after sign-off: it gets retrained, the vendor ships an upgrade, the data underneath it shifts. DEEM turns the approval into a standard that can be checked again on every change.
Tested, signed, authorized to operate.
New weights. Same approval on file.
The model behind the API changed.
A new camera. New inputs. No code change.
DEEM answers that on every change.
See DEEM in action.
A recorded walkthrough of DEEM re-checking an approved model after a camera change, from the written standard to the recertification alert. It then shows how DEEM checks an LLM that has no answer key.
You can't trust what you can't formally define.
Generic benchmarks answer someone else's question. DEEM answers yours: is this model still working for your mission, against the definition of success your team wrote and signed?
Specify before you trust
Define valid output, error conditions and acceptable fallback behavior before a model is ever tested. Precise enough to enforce, readable enough for a program manager to approve.
Assess consistency
Detect drift and evaluate behavior over time, across model versions and across semantically equivalent inputs.
Operationalize your standards
Turn formal definitions into automated, auditable, continuous monitoring that travels with the system from development through deployment.
From spec to signal.
Write the standard once. Then every new model version is checked the same way.
Write the standard
Say what correct looks like, in terms a program manager and an authorizing official can read and sign.
The standard is written in AISL, an open specification language. It becomes the contract the model is held to.
Point it at the test data
DEEM feeds a fixed test set to the model and collects every output.
The same set runs against every version, so each result is comparable to the one before it.
Run v1
Accuracy comes back at 94%. DEEM reports a range, not a single number: 92% to 96% at 95% confidence.
Even the worst plausible value clears the 90% bar. v1 passes.
The camera changes
A new camera rolls out. Nothing in the model's code changed. The inputs did.
DEEM re-runs the same standard on v3. Accuracy: 91%, range 88.7% to 93.3%.
DEEM decides
A plain average would pass v3 at 91%. DEEM checks the bar at the confidence level in your standard. At that level, the worst plausible value is 88.7%.
v3 fails. The team is alerted and the model goes back for recertification before it becomes a mission risk.
Worst plausible 88.7% is below the 90% bar. Alert sent. Recertification required.
DEEM doesn't force an answer.
Most evaluation tools return a yes or a no on every run, even when the test set behind the number is too small or too narrow to support one. DEEM reports that instead of picking a side. You find out the result is thin while you can still add test data and run it again, rather than acting on a PASS that was never backed by enough evidence.
The model meets the standard, even at the lowest likely value.
The model no longer meets the standard. It goes back for recertification before the next release.
Not enough evidence to decide. DEEM says so instead of guessing, so you know to collect more test data.
Check consistency when there's no answer key.
Classifying an image has a right answer you can grade against. Summarizing a report or drafting a response does not, so there is nothing to mark the output correct or incorrect. DEEM tests consistency instead: it asks the same question in several wordings, many times each, and measures whether the answers agree.
Aligned answers suggest well-grounded reasoning. Divergent ones point to uncertainty, sensitivity to phrasing, or a badly worded question.
The pass mark is part of the standard you write. DEEM enforces exactly that standard.
Illustrative example: 38 of 40 answers agree.
Built for high-stakes AI.
Unlike rule-of-thumb metrics, DEEM creates an auditable standard that travels with the system from development through deployment.
Custom definitions of success
Define acceptable output even when ground truth is fuzzy or task-specific.
Consistency checking
Check that semantically equivalent prompts produce aligned responses, before deployment.
System-level drift detection
Get alerted when behavior no longer meets your standard, and prompt recertification.
Side-by-side benchmarking
Compare model variants, prompting strategies and fine-tuning recipes against the same spec.
Continuous validation
Monitor production behavior over time and catch failures and quality issues automatically.
Automated quality assurance
Regression analysis and performance tracking run on their own. Manual review becomes a choice.
DEEM watches the model. DSAIL® checks the answer.
Defines what correct behavior looks like
A formal language for the operational boundaries and correctness criteria of an AI system. DEEM is built on it, so every standard is readable, versioned and auditable.
Read the AISL docsProves each output meets your policy
A separate engine with its own language. DEEM judges the model against your standard; DSAIL® judges a single output against your written policy, returning TRUE, FALSE or UNKNOWN with an auditable trace to the rule that decided it.
Explore DSAIL®See DEEM on your model.
Bring one model and the standard it was approved against. We'll show you the first re-check.
Runs where your data does: hosted API, on-premises, air-gapped or cloud.