Wrong data costs more than missing data

In short

• The validation I built has two stages, but the two stages don’t have equal authority. Deterministic rules make the decision; the LLM only comments, and it can only ever tighten a verdict.
• Every new type of problem the model finds is turned into a rule and measured again on every run. The model discovers; the code measures.
• I don’t write a value I’m not sure of. I leave it empty and pick the reason from a closed list; an empty value without a reason can’t get past the schema.
• An agent saying “I succeeded” doesn’t count as evidence. In runs where the evaluator said FAIL, accepted overclaims dropped from 83 to 0.

The happy path is not a success criterion

Most teams working with GenAI go through the same moment. The demo works flawlessly, and everything is green for the first week. Then the inputs shift a little, a service you depend on slows down one day, the model reads a phrase differently. The system still says “success”, but the data coming out of it is no longer correct.

In this article I’ll describe how I faced that moment in a system that uses LLM agents to produce structured data from unstructured inputs. The system itself isn’t the subject here. The subject is the engineering lessons I learned along the way, which I think carry over to other GenAI systems too.

The question I started with was simple: when conditions change, does the system quietly produce wrong data, or does it break visibly? The second is an outage; the first is a loss of trust. I wanted the second, and I wrote that into the architecture.

Rules compute, the model interprets

When people hear “two-stage validation”, most of them picture rules first and then an LLM judge. Mine is a bit different: the two stages deliberately don’t have equal authority. The first stage delivers the verdict. The second stage only advises.

The first stage is fully deterministic. At least 97% of critical fields must be populated, the share of valid numeric values must not fall below 98%, and duplicate records must not exceed 2%. These thresholds are declared explicitly in each input source’s configuration, and if the error list isn’t empty, there is no PASS. I never defined a result called “mostly passed”.

The thresholds for the data health gate live in a single file, pinned with a hash. If someone quietly loosens a threshold, the hash no longer matches and every check at that gate returns FAIL. Changing a threshold can be legitimate, but it should never be invisible.

For fields like numbers, units, links and codes, I don’t use an LLM judge at all. Their correct answer is well defined; I don’t need a model telling me “I think it’s right”. Before any comparison, a rule-based normalization layer runs: number and currency formats, unit conversions, URL canonicalization, whitespace and case folding.

The architecture limits the model’s authority

In the second stage the LLM is an advisor. But “advisor” isn’t a polite request; it’s a boundary the code enforces. There are four rules:

• It only tightens. The model can move a PASS down to WARN; it can’t turn a FAIL into a PASS.
• It follows the schema. The model’s output must conform to a JSON schema generated from the data model. Output that doesn’t conform counts as FAIL.
• It doesn’t touch numbers. A separate validator checks that the model hasn’t touched the numbers, the evidence fields or the gate decisions. The model may only refine the explanatory text.
• It can’t prove itself. The model can’t mark a hypothesis as “proven” on its own.

Why so strict? Because an LLM is often at its most convincing exactly when it’s wrong. A fluent explanation can easily paper over a wrong number. Taking the numbers out of the model’s hands closes that risk at the design level, which is far more reliable than trying to persuade the model with a better prompt.

If the model finds something, it becomes a rule

So why use an LLM at all? Because rules only catch the errors I already know about. I can’t write a rule for an error type I don’t know; someone has to see it first.

For that, I give the model a small window. I draw a stratified sample of at most 100 rows from the data: a few rows from each category and value band. These rows are wrapped as “untrusted data” before they reach the model; no text inside them is treated as an instruction.

If the model spots a type of problem the rules don’t know about, it doesn’t report it in prose. It writes a rule: an identifier and a SQL condition that defines the violation. The deterministic tool then reruns over the full data with this extra rule, and it is the tool that produces the count.

The model discovers; the code measures.

What I like most about this loop is that the model’s finding doesn’t stay an opinion. It becomes a rule that is measured on the next run, and the one after that. A week later, I don’t need the same model to notice the same thing all over again.

Wrong data costs more than missing data

When I first heard this sentence, it felt counterintuitive. An empty cell sticks out, while a wrong value at least looks “filled”. Then I put the cost of the two errors side by side.

A wrong number silently corrupts averages and totals. The report looks consistent but is wrong, and tracing the error back to its source is often impossible. An empty value, on the other hand, gets counted, filtered, classified and put into a backfill plan. In short, a gap is a work item; a wrong value is a loss of trust.

I turned the principle into code in two places. On the extraction side, in configurations that read a field from more than one surface of the input, the value is left empty if the surfaces contradict each other, and the conflict is logged separately. On the evaluation side, every cell is written with an envelope: either the value and its source, or, if it’s empty, the reason. The reason isn’t free text; it’s chosen from a closed list, in a fixed order of precedence.

The schema enforces two rules: an empty value can’t exist without a reason, and a value and a reason can’t exist at the same time. In other words, a “silent null” can’t get through the system. I should also say this: moving the envelope into the production rows, and a score that grades trust across sources, are still open decisions; today I use the envelope in evaluation runs.

Empty cells then pass through a deterministic classifier. There’s a small but important rule here: to label a field “not in source”, every surface where that field could appear has to have been measured: structural metadata, body text, attachments, service responses. If even one wasn’t measured, I can’t assign that label; the field stays “not measured”. Otherwise I’d be confusing “I didn’t look” with “it isn’t there”.

Five steps to filling a gap

Knowing a field is empty isn’t enough; it also has to be filled. The easiest route would have been to have an LLM guess the gaps. I deliberately didn’t do that. I always start from the first step of the ladder.

This order ensures the cheapest and most reliable fix is always tried first. The third and fourth steps must pass a holdout test: I hold back at least 200 rows that I know are populated and try the method on them. For categorical fields, if agreement doesn’t reach 95%, the method is rejected. Even if LLM inference passes this test, it isn’t written to the main table; it stays in a separate table with its source recorded, pending human approval.

A violation is FAIL, the unmeasurable is WARN, silence is forbidden

This three-part rule may be the thing that has helped me most. If a metric can’t be computed, I don’t give a PASS; I give a WARN labeled “not measured”.

I saw why this matters very clearly while designing a privacy test. The test measured whether personal data leaked to the model. If the content is stopped by a protection layer, it never reaches the model, and the test reports “0 leaks” and gives a PASS. Technically correct, practically meaningless.

The fix was a separate gate that first verifies the content actually reached the model. If it didn’t, the test returns FAIL. What we have here isn’t an unmeasurable metric but a violated precondition: for the test to mean anything, the content has to reach the model. An empty test is not a passing test.

The model’s claim is not evidence

One of the agents runs in a loop where it figures out on its own how to read a new input source. At the end of the loop it finishes by saying “I succeeded” or “I failed”.

When I started measuring, I saw something uncomfortable: the model frequently said “I succeeded” even in cases where the deterministic evaluator said FAIL. In one measurement set, the number of these overclaims was 83.

I set a rule: if the evaluator says FAIL, the model’s “I succeeded” claim is rejected and the run counts as failed. In the same measurement, accepted overclaims dropped to 0.

The model didn’t get more honest; dishonest claims just stopped paying off.

Separation of authority between agents

The data health work is run by a team of agents with six roles. The distribution of authority is deliberately lopsided: there is exactly one role that writes to the repository and exactly one that runs commands in the cloud environment; the rest only read.

Every command from the executor first goes through a fail-closed pre-check. A non-development environment, an unconditional delete or update, a force overwrite, and triggering the scheduler are all rejected. Creating a specific stop file or setting a single environment variable halts the whole team instantly.

This cost me some speed. But weighed against the cost of an agent deleting a table in the wrong environment, it was not a hard trade to make.

Takeaways

1. Trust doesn’t come from believing the model will behave correctly; it comes from making sure that when it behaves wrongly, the outcome stays limited and visible.
2. Make the LLM a partner in discovery, not the owner of the decision, and turn what it finds into rules.
3. Don’t write a value you’re not sure of; write why it’s empty.
4. “Not measured” is a result, not a PASS.
5. Test the model’s claims against evidence; when claim and evidence diverge, evidence wins.

In the next article I’ll walk through twelve techniques that measure these principles, and what I found on the first run.

---

The figures are my own measurements (September–October 2026). Project, domain and organization names are deliberately left out; what’s described here is the method I used and what I learned.