To read vendor accuracy claims, first find out what the percentage counts: characters, fields or whole documents, and on whose test set. Then ask how blanks and reviewed fields were treated, whether the documents resemble yours, and what the figure is per field type. A claim that cannot answer those questions should be tested on your own documents.
What a vendor accuracy claim usually measures
A vendor accuracy claim is a single percentage that summarises how often an extraction tool got something right, and it can be counting very different things. It might count characters read correctly, fields returned with the right value, or documents with every field right. It also depends on which documents were tested, and the bare number does not say.
Three questions pin down what a figure measures.
What is the unit? Characters, words, fields or documents. Each gives a different number from the same output, as the next section shows.
What is the test set? A figure measured on the vendor's own curated samples, on a public research dataset, or on a customer's real inbox will differ, because the documents differ. Clean, typed, single-layout forms score higher than faxed EOBs from dozens of payers.
What happened to review? Some figures describe the tool's raw output. Others describe the result after a person checked the uncertain fields. Both are legitimate, but they answer different questions: the first tells you how much review you will need, the second tells you how good the final data is with review included.
Two more details change the number. One is how blanks were scored; if every correctly empty box counts as a success, a form with many empty boxes inflates the figure. The other is the match rule: whether "$1,250.00" and "1250" were treated as the same value, and whether near-misses earned partial credit.
A headline percentage has no room for all of this, and that is fine. What matters is whether the vendor can answer when you ask.
Character accuracy vs field accuracy in vendor claims
Character accuracy counts the share of individual characters read correctly; field accuracy counts the share of whole fields, such as a member ID or an allowed amount, returned with exactly the right value. A high character figure can sit alongside a much lower field figure, because one wrong character makes a whole field wrong.
Character-level measures come from OCR research. The OCR-D quality assurance specification defines the character error rate as "the quotient of the edit distance over the length" of the reference and OCR text, where edit distance counts the single-character insertions, deletions and substitutions needed to turn one into the other. It is a sound measure of reading quality. It does not tell you how many fields you can post without checking.
The gap is easy to work out with hypothetical numbers. If each character were right 99% of the time and errors were independent, a 10-character member ID would be fully right only about 90% of the time; the full worked example is in why 99% character accuracy is not enough for medical records. The same effect compounds at the document level: if, again hypothetically, each of 30 fields on a claim is right 99% of the time independently, the chance all 30 are right is about 74%. A "99%" figure can therefore describe characters, fields or documents and mean three very different amounts of review work.
Field accuracy is what a billing team needs, because a field either posts correctly or it does not. When a claim does not say which unit it uses, ask. How to score fields yourself, including blanks and match rules, is covered in the guide to measuring field level extraction accuracy.
Which questions expose a weak vendor accuracy claim?
The questions that expose a weak vendor accuracy claim ask what was counted, on which documents, under which rules, and with or without human review. A strong claim has specific answers; a weak one falls back on the headline number. Asking for the figure per field type is the single most revealing question.
The claim-decoding table
The wording in the first column is example wording written for this post, not quoted from any vendor.
| Example claim wording | What it likely measures | The question to ask | What a good answer includes |
|---|---|---|---|
| "99% accurate" | Unstated unit: characters, fields or documents | "Characters, fields or documents?" | The unit, the match rule and how blanks were scored |
| "99% field-level accuracy" | Fields, possibly including empty boxes | "How were correctly blank fields counted?" | Accuracy on fields with a value, reported apart from blanks |
| "Up to 98% accuracy" | The best result on one test or document type | "What was the range across document types?" | Lowest and highest result by document type, with counts |
| "98% accuracy on medical claims" | One claim form, possibly typed and clean | "Which forms, how many payers, what scan quality?" | Number of documents, layouts and the share of faxes or scans |
| "99.5% accuracy with human review" | Final data after people corrected fields | "What was the raw accuracy and review rate?" | Raw field accuracy, share of fields reviewed and threshold used |
| "Reduces errors by 80%" | A change against an unstated baseline | "Compared with what, measured how?" | The baseline process, its measured error rate and the period |
Questions about the test set
Ask how many documents and fields were in the test, how many document types and layouts, and where the documents came from. Ask whether the test documents were kept separate from any tuning. In machine learning research, the Patterns paper Leakage and the reproducibility crisis in machine-learning-based science names "lack of clean separation of training and test dataset" as a leading cause of results that do not hold up. A figure measured on documents the tool was tuned on will overstate how it does on yours.
Questions about the fields
Ask for results per field type: amounts, codes, dates, identifiers. Ask separately for wrong values and missed values, which information retrieval calls precision and recall. A tool can have a strong overall number while one field type you depend on fails often.
Questions about review and confidence
Ask what share of fields fell below the tool's confidence threshold at the reported accuracy, and what threshold was used. A confidence score is the tool's estimate of how likely a field is to be right; the guide to what an OCR confidence score tells you explains where it comes from. An accuracy figure without a review rate hides the cost of reaching it.
Was the vendor's accuracy tested on documents like yours?
A vendor's accuracy was tested on documents like yours only if the test set matched your mix of document types, payers, layouts and scan quality. Each of those changes the result. A figure from typed, single-layout forms tells you little about faxed EOBs from many payers, handwritten intake forms or multi-page remittances.
Layout is the first difference. Each payer prints its EOB differently, and a tool that has seen a layout many times can do better on it than on one it has rarely seen. Ask which payers and form versions were in the test, and compare that list with your own top payers by volume.
Scan quality is the second. Faxed pages, skewed scans, low-contrast copies, stamps and handwriting all make reading harder. If your inbox has a meaningful share of faxes and the test set had none, the figure was not measured on documents like yours. The guide to how OCR handles low-quality medical scans covers why those pages are harder to read.
Content is the third. Claims with many service lines, EOBs that run to several pages, denied claims with zero payments and forms with crossed-out values all test parts of a tool that simple documents do not.
The direct fix is to test the tool on your own documents. A small labelled set drawn from your inbox, scored field by field, answers the question no vendor figure can. The guide to writing a pilot accuracy test plan for documents, fields and success criteria sets out what to agree before that test starts.
Schema Care does not publish an accuracy percentage. Instead each field it returns carries a confidence score and a link to its spot on the page, and you choose the score a field needs to skip review, so the measure that matters is the one you take on your own documents; the review workflow shows how those pieces fit together.
Vendor accuracy claims that compare unlike test sets
Vendor accuracy claims that compare unlike test sets cannot be ranked against each other, because each percentage depends on its own documents, fields, match rules and review step. One vendor's 97% on a hard, varied set can reflect a better tool than another's 99% on clean samples. Without a shared test set, the numbers are not comparable.
Picture two fictional claims. Vendor A reports field accuracy on 500 faxed EOBs from 40 payers, scored by exact match, with blanks excluded. Vendor B reports character accuracy on 50 typed claim forms from one template. Even if B's number is higher, it was measured in an easier way on easier documents. Neither number tells you how the two tools would do on the same pages.
Several differences make claims incomparable:
- Unit. Characters, fields and documents produce different numbers from the same output.
- Document mix. Layouts, payers and scan quality all move the result, whichever tool is tested.
- Match rules. A tool scored with normalized matching on amounts will look better than one scored with exact string matching.
- Blanks. Counting correct blanks raises a figure; excluding them lowers it.
- Review. A figure after human review is not comparable with raw output.
- Field list. A test of five easy fields cannot be compared with a test of thirty, including codes and amounts.
The way to compare tools is to give them the same labelled documents, score them with the same rules, and read the results per field type at matched review rates. A tool that makes fewer errors at a 20% review rate is doing better than one that reaches the same error rate only by sending half its fields to a person. The guide to how to set a confidence threshold for extraction explains how to pick that review rate from test data.
A side-by-side test also exposes things no headline figure can, such as how each tool scores its own uncertainty and how easy its output is to check. For the basics behind those comparisons, including how OCR reads a page in the first place, browse the medical OCR basics category.