Building a ground truth test set for medical documents means choosing real-looking documents that match your inbox, writing a labeling spec for each field, having two people label every document independently, and settling their disagreements into one correct value per field. The finished set stays locked away from tuning so it can judge any extraction tool fairly.
What a ground truth test set for medical documents contains
A ground truth test set for medical documents contains three things: a fixed collection of documents, a list of the fields to be tested on each document type, and one agreed correct value for every field on every document. Ground truth means the answer key: the values a careful person reads from the page, written in a set format.
The documents are the pages themselves, as images or PDFs, in the same form an extraction tool will receive them. A scan from your own front desk belongs in the set as it is, smudges included. Cleaning up the images before testing measures a tool on documents you do not have.
The field list comes from what you actually post. If your team keys allowed amounts, plan payments and remark codes from EOBs, those are the fields to label. Labelling every box on the form costs time and adds fields nobody uses. Start from the screens in your billing system that people type into.
The correct values sit in a table: one row per field per document, with the document ID, the field name, the expected value, and a note for anything unusual, such as "blank on page" or "unreadable". That table is what an extraction tool's output gets compared against.
Every field also needs a written definition, called a labeling spec, so two people reading the same page record the same value.
Labeling spec template (five fictional EOB fields)
The rows below are a template. Field definitions are written for a fictional provider EOB, and every example value is made up.
| Field name | Definition | Format | Blank rule | Example value |
|---|---|---|---|---|
| Payer claim number | The claim reference the plan printed for this claim, not the practice's own account number | Characters exactly as printed, no spaces | Label "blank" if the EOB prints no plan reference | 2026A0419887 |
| Date of service | The service date on this line; if the line shows a range, the first date | YYYY-MM-DD | Use the claim-level date only if the line has none, and note it | 2026-03-14 |
| Procedure code | The procedure code on this line, without modifiers | Five characters as printed | Never blank on a service line; label "unreadable" if it cannot be read | 99213 |
| Allowed amount | The amount the plan recognised for this line, from the allowed column | Number with two decimals, no symbols | Label "0.00" only if printed as zero; "blank" if the column is empty | 132.40 |
| Plan payment | What the plan paid on this line, from the paid column | Number with two decimals, no symbols | Same as allowed amount | 105.92 |
Write the spec before anyone labels a page, and keep a change log. When labelers find a case the spec does not cover, decide it once, add it to the spec, and re-check the documents already labelled. How each labelled value is then scored against a tool's output is set out in the guide to exact match, normalized match and partial credit.
How many medical documents does a ground truth set need?
A ground truth set needs enough medical documents that each field type you care about appears many times, because results are judged per field type, not per document. The rarest important field sets the size. A field that appears on one claim in five needs five times as many claims as a field on every claim.
The reason is the width of the error range. The NIST/SEMATECH e-Handbook page on confidence intervals for a proportion gives the Wilson interval, which still produces an upper limit when a test finds no errors. Applying it at 95% confidence is plain arithmetic, and the figures here are worked examples, not results from any tool:
| Examples of one field type | Errors found | Upper limit of error rate (95%, Wilson) |
|---|---|---|
| 50 | 0 | 7.1% |
| 200 | 0 | 1.9% |
| 400 | 0 | 1.0% |
A field type with 50 clean examples tells you only that its error rate is probably under about 7%. If you need to know whether a field fails less than 2% of the time, you need roughly 200 examples of it with no errors, and more if some errors turn up.
Turning field counts into a document count
Work backwards from the rarest field you care about. Suppose, as a fictional example, that a procedure modifier appears on about one claim line in five, and you want 200 modifier examples. You need about 1,000 claim lines. If your claims average three lines, that is roughly 330 claims. Common fields, such as patient name, will then have far more examples than they need, which is fine.
A step list for building the set
- List the document types and fields you post today, from your billing or EHR entry screens.
- Write the labeling spec, one row per field, using the template above.
- Pick the rarest important field and work out how many documents give enough examples of it.
- Pull documents from your real inbox across payers, layouts and scan qualities, in proportion to volume, then add extra of the rare cases.
- De-identify the documents before they leave your control.
- Have two labelers record every field on every document, independently.
- Compare their labels, settle disagreements with a third person, and record the final value.
- Freeze the set: store it read-only, with a version number, and keep it out of all tuning work.
Which payers, layouts and scan qualities belong in a ground truth set?
A ground truth set should include the payers, form layouts and scan qualities that arrive in your real inbox, in roughly the same mix. A set built from one payer's clean PDFs will overstate accuracy on everything else. The goal is a set that looks like an ordinary week, plus deliberate extra copies of the hard cases.
Start by counting a recent month of documents by type and source. For EOBs, group by payer, because each payer prints its own layout. For claims, group by form type and by who prepared them. For intake forms, group by version, since practices revise their forms.
Then cover three kinds of variation:
- Layout. Every payer or form version that makes up a meaningful share of volume gets its own slice. Include at least a few documents from the long tail of small payers, because those are where unfamiliar layouts live.
- Scan quality. Include faxed pages, skewed scans, low-contrast copies and pages with handwriting or stamps, in the share you actually receive. Do not clean them.
- Content. Include multi-page EOBs, claims with many lines, denied claims with zero payments and documents with corrections or crossed-out values.
Weight the set so the main results reflect real volume, then report the hard slices separately. If faxes are 10% of your volume, they should be about 10% of the headline figure, with their own row showing how the tool did on faxes alone.
Refresh the set when your inbox changes. A payer that redesigns its EOB or a new referral source with a different form makes an old set less representative. Add a new slice rather than replacing the old one, so you can still compare against earlier results.
Labeling ground truth twice and resolving disagreements
Labeling ground truth twice means two people record every field independently, a comparison flags each field where they differ, and a third person settles each difference against the page and the spec. This catches the slips any single labeler makes, and the disagreement rate shows how clear your spec is.
This is established practice for medical text. In the AMIA paper Building Gold Standard Corpora for Medical Natural Language Processing Tasks, Deleger and colleagues report that "all documents were double-annotated by two annotators", that disagreements were resolved in consensus sessions, and that guidelines were revised based on the issues those sessions raised. To keep one annotator from simply deferring to the other, they had "a third-party to supervise the consensus sessions between the two annotators".
Measuring agreement
Agreement between two labelers can be scored like extraction itself. The JAMIA paper by Hripcsak and Rothschild on agreement and the F-measure shows that the F-measure between two raters equals their positive specific agreement, which makes it usable when the usual kappa statistic is hard to compute. For field labels, the simpler measure is often enough: the share of fields where both labelers recorded the same value under the spec's format.
Running the process
Keep the labelers apart while they work. If one can see the other's values, the second reading stops being independent. Give both the same spec version and the same documents in a different order.
Look at disagreements by field before settling them. If most disagreements are on one field, the spec for that field is unclear, and the fix is a clearer definition, not a tie-breaker. If disagreements are spread thinly, they are ordinary slips.
The tie-breaker decides against the page and the spec, not by picking the more senior labeler. Record the reason for any decision that sets a precedent, add it to the spec, and check whether earlier documents have the same case.
De-identified samples for a medical ground truth set
De-identified samples for a medical ground truth set are documents with patient identifiers removed under one of the methods HIPAA's Privacy Rule allows, so the set can be shared with an outside tool or reviewer without exposing protected health information. The rule sets out two methods, and removing names alone does not meet either.
The rule is at 45 CFR 164.514 on eCFR. Paragraph (b) gives two routes. Under expert determination, a person with appropriate statistical knowledge determines that the risk of identifying an individual is very small and documents the methods and results. Under safe harbor, a listed set of 18 identifiers is removed and the covered entity has no actual knowledge that what remains could identify someone.
Several of those 18 identifiers sit right in the fields a billing team cares about: names, "all elements of dates (except year) for dates directly related to an individual", medical record numbers, health plan beneficiary numbers and account numbers. That creates a testing problem. An EOB with its patient dates and member IDs stripped out cannot test how well a tool reads those fields.
One way to handle this is to replace removed values with fictional ones of the same format, so the field still has something realistic to read. Whether a particular approach meets the rule for your documents is a question for your privacy officer or counsel; this post describes what the regulation says, not whether a given set complies. The HIPAA and PHI category covers business associate agreements and the Privacy Rule in more depth.
Schema Care's free plan fits this stage: it runs on de-identified sample documents, and real patient data goes in only after a business associate agreement is signed, which each paid plan on the pricing section includes.
Leaking ground truth documents into threshold tuning
Leaking ground truth documents into threshold tuning means using the test set to choose confidence thresholds, match rules or field settings, and then reporting results on that same set. The results look better than they will be on new documents, because the settings were fitted to those exact pages. The test set must stay untouched while you tune.
Machine learning research has a name for this. In the Patterns paper Leakage and the reproducibility crisis in machine-learning-based science, Kapoor and Narayanan list "lack of clean separation of training and test dataset" as a leading form of leakage, and note that "using the same dataset for training and testing the model is a textbook example of overfitting". Their examples involve feature selection and pre-processing; a confidence threshold chosen on the test documents is the same mechanism applied to a different setting.
In practice, leakage creeps in quietly. Someone looks at which fields failed on the test set and adjusts the threshold for those fields. Someone adds a cleanup rule because three test documents had a stray character. Each change is reasonable, and each one tunes to the answer key.
Keep two sets instead. A tuning set, built the same way, is where you look at failures, set thresholds and write rules. The test set is opened only to measure the final settings, ideally once per decision. If you must look at test failures, treat the set as spent for that purpose and build a fresh one before you report a number.
The same discipline is what makes a vendor's figure believable or not, which is the subject of the guide to how to read vendor accuracy claims. When your set is ready, the guide to the pilot accuracy test plan for documents, fields and success criteria shows how to put it to work while evaluating a tool, and the full walk-through is in running an accuracy test during a pilot.