Why accuracy is the wrong way to measure data extraction

The extraction system you rejected for scoring badly on accuracy may be the one that would have paid for itself.

A vendor demo reports 99% accuracy. Then it meets your real documents, and the number stops meaning anything.

The question every team asks about an extraction system is fair: how well is it working? The trouble starts when "how well" silently becomes "what is its accuracy," because accuracy is a metric built for a different kind of problem. Judge an extraction system by it and you can approve a system that does nothing useful, or kill one that would cut your manual work by most of its volume. Both mistakes cost real money. Both come from the same wrong yardstick.

What does "accuracy" actually measure?

Accuracy is the most common evaluation metric for classification problems, because it is simple to state. You count four outcomes:

• True positive - the system flagged something and was right.

• True negative - the system left something alone and was right to.

• False positive - the system flagged something and was wrong.

• False negative - the system missed something it should have caught.

Accuracy is the share of all decisions the system got right: the true positives plus the true negatives, over everything. For a clean classification task - is this email spam or not - that is a sensible score. Every item is one decision, and the right answers and wrong answers sit in the same range.

Extraction is not that kind of problem. And the moment you treat it as one, the metric starts to lie.

Why does accuracy break for data extraction?

Because of the true negatives - everything the system correctly left alone.

Take a short list of ten words and ask a system to pick the word pairs that form a company name. There are two right answers. But once you count every possible combination of those ten words - pairs, triples, longer strings - there are nearly ten million ways to choose. Say the system returns four guesses and gets exactly one right. Plug that into the accuracy formula and it scores 99.99996959% - a rounding error short of perfect - because it gets credit for the millions of combinations it "correctly ignored" and was never going to pick anyway. [1]

Nobody would call that system effective. It found one of two answers. But accuracy counts all those untouched non-answers as wins, and they swamp everything that matters.

Real extraction is worse, not better. A single document carries many fields, each value can attach to many records, and the number of possible value-to-record combinations is enormous. The ratio of true combinations to possible ones is wildly imbalanced, and accuracy rewards that imbalance. It is not measuring the work you care about. It is measuring how big the haystack is.

How can a genuinely useful system score zero?

Flip to the other failure, the one that actually costs you a good system.

Pull "expenses incurred during a financial period" from an annual report, and the answer is not one value - it is a record: the amount, the period, the expense type, all of which have to line up. Now ask the honest question: when is a record correct? If you score field by field, a system can get every individual value right yet assemble them into the wrong records, and the field-level score still looks great. So most teams tighten the rule: a record counts only if every field in it is right. [2]

That rule has a trap. Imagine a system that processes 100 records of 10 fields each, and one field is genuinely hard, so it gets that single field wrong every time. Now every record has one wrong field. Every record fails the all-or-nothing test. The accuracy score is zero. On that number, the business shelves the system and goes back to doing the work by hand. [2]

But look at what the system actually did. It missed 100 fields out of 1,000. It got 900 right. If your real cost is the human time spent extracting, that system just did 90% of the work - and the metric told you to throw it away. A measure that recommends rejecting a system that removes most of your manual effort is not a measure of quality. It is a measure pointed at the wrong thing.

What should you measure instead?

You might reach for precision and recall here, or their blend, the F1 score. They are a real step up from raw accuracy - they ignore the true negatives that inflate it, so the ten-million-combination problem disappears. But they still answer the wrong question for a buyer. They tell you how often the system is right; they say nothing about how much human work is left behind, or what that work costs in time. A system can post strong recall and still hand your team a punishing pile of corrections. What you actually care about is the work removed.

Measure that directly. Two counts, taken on your own documents, capture it honestly: [2]

• Human Touch Points (HTP) - every field a person had to fix to make a record fully correct. Adding a missing value, correcting a wrong one, deleting a bad one - each is one touch point.

• System Prediction Points (SPP) - every field the system delivered that a person never had to touch.

From those two, the number that matters:

System efficiency = SPP / (SPP + HTP)

It is simply the share of the work the system did without a human. In the example above, that is 900 of 1,000 - a system efficiency of 90%, on the exact system the accuracy score wrote off as zero. The higher the system efficiency, the fewer the human touch points, the less time spent per document, and the more directly the system improves your ROI. The metric keeps the link to money built in, which accuracy never had.

One caveat to control for is subjectivity: a person chooses which fields to correct first, and over time people learn to fix the fewest fields they can. Hold the number steady by fixing the correction protocol - the same reviewer, the same document set, the same rule for what counts as corrected - especially when you are comparing one vendor against another.

The second caveat is drift. No single number survives contact with changing documents on its own: regulatory language shifts, business context moves, the people writing the source reports change. So pair the efficiency number with a small, explicit verification effort per document, measured in time and kept an order of magnitude smaller than the extraction work itself. That last part is only possible if you can check the output quickly - which is a question about traceability.

Why a score means nothing unless you can trace it

A number you cannot audit is a claim, not evidence. "The system is 90% efficient" or "the model said so" is worth little to a CDO, an auditor, or a regulator who can ask to see the working. The standard for trustworthy AI inputs is the same everywhere: data that is validated and traceable, never a black box. [3, 4]

That is what keeps verification cheap. When every extracted value points back to the exact place it came from - which page, which section, which word - a reviewer confirms a record in seconds instead of re-reading the source document. Cheap verification is what lets the small per-document check stay small. Without lineage, every spot-check is a re-extraction, the verification effort balloons past the extraction effort, and the efficiency you measured on paper never shows up in practice.

Measurement and traceability are the same discipline. One tells you how much work the system removed; the other lets you trust the answer without redoing it.

Accuracy vs what to measure instead

The left column is the number a vendor hands you. The right column is what tells you whether the system will actually pay for itself.

Do you have to build this yourself?

You can. It means defining the metric, instrumenting every correction your team makes, and - the hard part - building the lineage that makes verification fast enough to trust. That last layer is months of engineering aimed at plumbing, not at your actual work, and most teams underestimate it because the demo lands long before the data foundation is done.

At SageX, that foundation ships as infrastructure. The platform turns your scattered documents into structured, AI-ready data, and every value it returns is traced to its source - which page, which section, which word - so a reviewer can confirm it in seconds. [5] Outputs carry a confidence signal, so the fields most likely to need a human are the ones flagged for review, and the touch points land where they matter. The longer you use it, the better it gets - it learns from every correction, so the human touch points fall over time and the efficiency you measure keeps climbing. It runs inside your own cloud, so your documents never leave your walls. [5] We have five live, revenue-generating deployments [5] with teams who chose to build on the foundation rather than rebuild it.

Once you are measuring the right thing, the next question is how a system earns a high score in the first place: see how we think about getting accurate values out of your documents and whether to build or buy the data pipeline underneath.