How We Think About Accuracy

If you have evaluated more than one document-extraction or financial-spreading vendor, you have heard the pitch: "We're 99.8% accurate." It is a comforting number. It is also, in almost every case, meaningless. Not because vendors are lying about their technology, but because the number itself is far harder to compute and validate than anyone selling it wants to admit.

We have decided to take a different posture. We would rather be the company that tells you the truth about accuracy than the one that quotes you a number we cannot defend. So here is how we actually think about it, grounded in the research, including the research that is inconvenient for us.

The status quo we are being compared to: humans typing numbers

The honest baseline is not "perfect." The baseline is a human analyst reading a PDF and re-keying figures into a spreadsheet. So the first question is not "is the AI perfect?" It is "how good is the thing we are replacing?" The research here is unambiguous: manual data work has a measurable, non-trivial error rate, and it has had one for sixty years.

• A 2024 systematic review and meta-analysis of data-processing methods in clinical research (Garza et al., International Journal of Medical Informatics) pooled error rates across studies: single-data entry ≈ 0.29%, double-data entry ≈ 0.14%, optical scanning ≈ 0.74%. And when humans read and abstracted data from records, the task closest to spreading a financial statement, the error rate jumped to 6.57%.¹
• At the mechanical level, the human-factors literature pegs raw keystroke accuracy at 99.5% to 99.8% (a 0.2% to 0.5% per-keystroke error rate), and accuracy at the level of complex judgments drops to 95% to 98% (Panko, synthesizing decades of human-error research).²
• In a real outpatient lab, 3.7% of manually transcribed results were discrepant from the interfaced source, and 14.2% of those were off by more than 20% (Mays & Mathias, JAMIA, 2019).³

Worth flagging, in the spirit of this post: the famous "1% manual data-entry error rate" you will see repeated everywhere has no clean primary source. It is folklore. The real, measured numbers above are the citable ones, and they vary by an order of magnitude depending on the task.

Why humans make these errors (it is not carelessness)

This is not a story about lazy analysts. It is a story about cognition under load:

• Slips and lapses are structural, not moral. James Reason’s foundational taxonomy (Human Error, 1990) distinguishes mistakes (wrong plan) from slips and lapses (right plan, wrong execution). Re-keying a number you understand perfectly well, and still transposing two digits, is a textbook slip.⁴
• Vigilance degrades. Sustained monitoring and checking is not "easy" attention. It depletes a finite resource and is genuinely stressful (Warm, Parasuraman & Matthews, Human Factors, 2008).⁵ The analyst on their 40th tax return at 4pm is not the analyst from 9am.
• Number entry has its own failure modes. Out-by-ten errors, transpositions, and omissions are well-documented. Per-keystroke error probabilities for number entry are modeled in the 0.5% to 10% range depending on conditions (Thimbleby & Cairns, J. Royal Society Interface, 2010).⁶

And here is the part that matters most for our argument: when a human makes these errors, nothing happens. No flag. No alert. No second look. A transposed digit in cell D47 looks exactly like a correct one. The error enters the credit memo silently and stays there until, if ever, someone catches it downstream. Human-only workflows fail silently. That is their defining weakness.

The measurement problem: why a single accuracy percentage is statistically fragile

Now the harder, more important point, the one that should make you skeptical of every vendor, us included, when we wave a single number at you.

To prove a system is "99.8% accurate," someone has to know the right answer for every field. That right answer is the ground truth. And ground truth is produced by humans. The same humans with the error rates above. This creates problems that do not go away no matter how good your model is:

1. Your measuring stick is itself bent. In a landmark 2021 study, Northcutt, Athalye & Mueller found label errors averaging ~3.3% across ten of the most-used ML benchmark datasets, including ~6% of the ImageNet validation set, datasets that thousands of papers had treated as gold (NeurIPS 2021).⁷ If the canonical, heavily-scrutinized test sets in computer science are 3% to 6% wrong, the hand-built ground truth for your financial spreads is not cleaner.

2. When the ruler is noisier than what it measures, the measurement is unreliable. This is just measurement theory. Spearman’s classic result on attenuation (1904) shows that any measured agreement is biased toward zero by the unreliability of the reference.⁸ Put plainly: if your reviewer’s error rate is comparable to the system’s error rate, the accuracy you compute is mostly noise about your reviewer, not signal about the system. Northcutt’s study showed this concretely: correcting label errors reversed which models ranked best. The winner was an artifact of the mislabeling.

3. Field-level accuracy aggregates into something much worse, and nobody quotes that number. Suppose a system genuinely hits 99% per field. A single financial statement can carry hundreds of line items. Run the math:

• A document with 500 fields at 99% per-field accuracy has, on average, 5 errors (500 × 1%).
• The probability that the entire document is error-free is 0.99⁵⁰⁰ ≈ 0.66%.

So the same system is "99% accurate" and "gets <1% of documents perfectly right." Both true, wildly different impressions. When a vendor quotes you a per-field number, they are quoting the flattering denominator.

4. To actually validate a quoted number, you would have to re-check every figure across a massive dataset, by hand. No one does this. It is the very labor the product exists to eliminate. And even if someone did, point #1 applies: their re-check carries its own error rate. The accuracy number is, mathematically, chasing its own tail, measuring a small error rate with an instrument that has an error rate of comparable size.

This is why we will not lead with a number. Not because our extraction is not strong (the first-pass extraction quality is excellent), but because we refuse to imply a precision to the measurement that the measurement cannot support. Anyone quoting you a confident, decimal-pointed accuracy figure for document extraction and financial spreading is either not thinking carefully about how that number was produced, or hoping you will not.

Our approach: assume errors will happen, and make sure they cannot hide

If you accept the two truths above, that every process, human or machine, produces errors, and that you cannot reliably measure your way to confidence with a single percentage, then the right engineering goal changes. It is no longer "achieve a magic accuracy number." It is:

Errors are inevitable. The only thing that matters is whether they are visible, traceable, and caught before anyone relies on them.

A human-only workflow fails this test completely: errors are invisible by construction. So we built the opposite: a system where the AI does a stronger, faster first pass than a human typically would, and then a human reviews on top of it, armed with tools the human-only workflow never had. Here is what that means concretely.

1. Extract-only, never infer. Our extraction models are explicitly instructed to capture only what is printed on the document, never to calculate, infer, or fill a gap with a plausible guess. A missing value comes back empty, not invented. This is the single biggest guard against the confident-but-wrong number, which is the most dangerous kind.

2. Total source traceability. Every extracted figure carries its source text, page number, and column, and every spread links back to the exact source document and page range. A reviewer clicks any number and sees precisely where it came from. This is the feature a human-only process can never offer. A re-keyed spreadsheet cell has no memory of where it came from.

3. Built-in validation that fails loud. Spreads are checked for internal consistency, so balance sheets must satisfy Assets = Liabilities + Equity, and operating statements must tie out to NOI. Failures surface as explicit validation errors and document concerns. The math checks never sleep, never get to their 40th document of the day, and never get tired.

4. Confidence flagging directs human attention. Every mapped line item carries a confidence score. Low-confidence extractions are flagged for closer review rather than blending in with high-confidence ones. The reviewer’s scarce attention, the resource the vigilance research tells us is finite and depletes, is spent where it is actually needed.

5. Model redundancy. A primary model runs with an automatic fallback, and a dedicated model handles document classification, so a transient failure degrades gracefully instead of silently corrupting output.

6. A real human-in-the-loop workflow with an audit trail. Every spread moves through Draft, Reviewed, and Approved, with the reviewer and approver recorded. Analysts can adjust any value, and every adjustment is logged: original extracted value, new value, who changed it, when, and why. The approved spread, not the raw extraction, is the source of truth.

Why this beats human-only, even before you count accuracy

Stack the two workflows side by side:

The key insight is that we are not asking you to trust the AI instead of the human. We are keeping the human, and giving them a better starting point and better instruments. The analyst no longer transcribes (the error-prone part). They verify a near-complete, fully-cited, self-validated draft. The error-generating step is removed; the error-catching step is amplified.

A subtle but real risk here is automation bias, the well-documented tendency of people to over-trust automated output and stop checking (Goddard et al., JAMIA, 2012, found that when an automated aid was wrong, it raised the odds of a human making the wrong decision by 26%).⁹ We take this seriously, and it is why traceability and flagging are not optional features. A number you cannot trace invites blind trust. A number that links to its source page, and that the system flags when it is uncertain or when the statement does not balance, structurally pushes the reviewer to look, turning the human-in-the-loop from a rubber stamp into a real check.

What we will actually tell you

We will not tell you we are "99.8% accurate." We will tell you this:

• The status quo, humans re-keying numbers, has a real, measured error rate, and worse, it fails silently.
• Any single accuracy percentage for this task is statistically fragile, because validating it requires a noisy human reference whose error rate is comparable to the thing being measured.
• Our extraction first pass is strong and consistent, but the durable advantage is not the first pass. It is that we make every error visible, traceable, and catchable, and we keep a human in the loop with the tools to catch it.
• The number that matters is not a vendor’s accuracy claim. It is how fast and how reliably your analyst can get to a spread they are willing to put their name on, and that is a question of traceability, validation, and workflow, not a decimal point on a slide.

In a market full of confident numbers, we would rather give you a defensible method. That is how we think about accuracy.

References

1. Garza MY, Williams T, Ounpraseuth S, et al. "Error rates of data processing methods in clinical research: A systematic review and meta-analysis." International Journal of Medical Informatics, 2024;195:105749. ↩
2. Panko RR. "Thinking is Bad: Implications of Human Error Research for Spreadsheet Research and Practice." EuSpRIG; arXiv:0801.3114. ↩
3. Mays JA, Mathias PC. "Measuring the rate of manual transcription error in outpatient point-of-care testing." Journal of the American Medical Informatics Association, 2019;26(3):269 to 272. ↩
4. Reason J. Human Error. Cambridge University Press, 1990. See also "Human error: models and management," BMJ, 2000;320:768 to 770. ↩
5. Warm JS, Parasuraman R, Matthews G. "Vigilance requires hard mental work and is stressful." Human Factors, 2008;50(3):433 to 441. ↩
6. Thimbleby H, Cairns P. "Reducing number entry errors: solving a widespread, serious problem." Journal of the Royal Society Interface, 2010;7(51):1429 to 1439. ↩
7. Northcutt CG, Athalye A, Mueller J. "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks." NeurIPS 2021; arXiv:2103.14749. ↩
8. Spearman C. "The Proof and Measurement of Association between Two Things." American Journal of Psychology, 1904;15(1):72 to 101. ↩
9. Goddard K, Roudsari A, Wyatt JC. "Automation bias: a systematic review of frequency, effect mediators, and mitigators." Journal of the American Medical Informatics Association, 2012;19(1):121 to 127. ↩