Behaviour that was never tested, only assumed
A model refuses correctly in the demo and complies with the same request phrased differently in production. Without an evaluation suite nobody knows until a customer finds it.
AI safety · Evaluation · Red-teaming · EU AI Act
The question is not whether it works. It is whether you can prove it, afterwards.
Every AI deployment eventually meets an auditor, a regulator or a journalist asking why the system did what it did on a specific day. Safety engineering is building that answer in advance, because it cannot be reconstructed afterwards — and for high-risk systems in the EU, the obligation is already live.
- EU AI Act high-risk obligations, since August 2026
- In force
- How often behaviour must be re-evaluated, not once
- Per release
- The form the evidence has to take
- Written
The cost
What goes wrong, and when
Safety failures are almost never the dramatic scenario. They are mundane, and they surface at the worst possible moment.
Silent regression on a model update
The provider ships a new version, behaviour shifts, and nothing in your pipeline notices because there was no suite to fail. The first signal is a complaint.
No record of why a decision was made
An affected person asks why they were declined. If the reasoning was not captured at the time, it cannot be reconstructed — and "the model decided" is not an answer that survives a regulator.
Classification done too late
A system assessed as high-risk after it is built carries obligations that should have shaped the architecture. Retrofitting documentation to a finished system is the most expensive way to do it.
The build
What we build
The evidence a deployment needs in order to be defensible, produced as a by-product of shipping, not as a scramble afterwards.
Task-specific evaluation suite
Built on your data and your failure modes, run in CI, and able to fail a release. A benchmark score from a leaderboard is not evidence about your system.
Adversarial and red-team testing
Structured attempts to make the system behave badly — rephrasing, role-play, indirect instruction — documented with what held and what did not.
Regression gates on model updates
Provider version changes run the suite before reaching production. A behaviour shift becomes a failed check, not an incident.
Decision records
Inputs, reasoning trace and outcome captured at the time, redacted, and retained. This is what makes an explanation possible months later.
EU AI Act risk classification
Assessed at design time, with technical documentation, logging and human-oversight obligations mapped to real architecture, not to a policy PDF. Annex III high-risk duties have applied since August 2026, so for most systems this is remediation, not preparation.
Human oversight design
Where a person must be able to intervene, what they see when they do, and whether they have enough context to overrule the system meaningfully.
The economics
Where the money moves
Safety work is insurance with an unusual property: it also makes shipping faster, because a suite that can fail a release is what lets you release without a committee.
- 01Deployments that clear review first timeThe largest hidden cost in enterprise AI is a finished system waiting on legal, risk or procurement. Evidence produced during the build removes that queue.
- 02Regression caught in CI, not in productionThe difference between a failed check and a customer incident, which is roughly two orders of magnitude in cost and entirely in reputation.
- 03Documentation as a by-productProduced continuously during the build instead of reconstructed under deadline, which is when it is most expensive and least accurate.
- 04Avoided retrofitClassifying at design time, not afterwards. High-risk obligations change architecture; discovering that late means rebuilding.
Why this firm
We built this control for ledgers first
This is the audit trail argument, moved from the ledger to the model. An auditor asking why a journal entry posted and a regulator asking why a model declined someone are the same question, and both need the answer recorded at the time.
Questions
Answered straight
If the output is used in the EU, generally yes — the same extraterritorial shape as GDPR. Most firms we speak to are in scope through a customer or a subsidiary, not through their own headquarters, and find out late.
No. A public benchmark says something about a model on someone else's task. Evidence about your deployment has to be generated on your data, your prompts and your failure modes, and has to be repeatable on demand.
QA asks whether the system does what it was specified to do. Safety evaluation asks what it does when someone actively tries to make it misbehave, and whether that behaviour is stable across model versions you do not control.
Usually nobody, which is the problem. It falls between engineering, legal and risk, and each assumes another holds it. Part of the engagement is naming the owner and giving them something they can run.
Next
Tell us what your system is allowed to decide alone.
If the answer is hard to state precisely, that is the finding. We start by writing it down, then by building the suite that proves it holds.