Running an accuracy test during a pilot means agreeing in writing which documents, fields and success criteria count, feeding the tool a labelled set that looks like your inbox, and reading the results field by field at several confidence thresholds. A fair test runs long enough to cover your real mix and ends with a clear decision.
What an accuracy test during a pilot should prove
An accuracy test during a pilot should prove whether a document extraction tool can take over a defined share of your keying work at an error rate you accept. A pilot is a time-limited trial of a tool on your own documents before you commit. The test should settle decisions, not produce one impressive percentage.
Write the decisions down before you start. Most buyers need answers to four:
- Go or no-go. Does the tool meet the agreed error rate on the fields you post, for the document types that make up most of your volume?
- Which document types. Is it ready for every type you tested, or only some? A tool can be ready for EOBs and not for handwritten intake forms.
- Where to set the threshold. At what confidence score should fields skip review, and how much review work will that leave?
- What review staffing looks like. How many fields a week will still need a person, and how long each takes.
Each decision needs a measurement. Go or no-go needs per-field-type error rates on fields that skip review. The threshold decision needs those rates at several cut-offs. Staffing needs review volume and time per reviewed field. If a measurement does not feed a decision, drop it from the plan.
A confidence score is the tool's own estimate of how likely a field is to be right. The test checks whether those scores mean what they claim; the guide to what an OCR confidence score measures explains where a field's score comes from and why it is not the same as accuracy.
Be clear about what the pilot cannot prove. A few weeks of documents cannot show how a tool handles a payer layout change next year, or a document type you did not send. List those gaps in the final write-up so nobody reads the result as broader than it is.
Pilot accuracy test plan: documents, fields and success criteria
A pilot accuracy test plan is a short written agreement, made before the first document goes in, that names the documents, the fields, the match rules and the success criteria. Agreeing these in advance stops anyone, buyer or vendor, from choosing the measure after seeing the results. It fits on two pages.
Documents
Name the document types, the share of each, and where they come from. Use a labelled set drawn from your own inbox, not the vendor's samples. The set should include the payers, layouts and scan qualities you actually receive, with the hard cases reported separately. How many documents give a stable result for each field type is worked through in the guide to how many medical documents a ground truth set needs.
Fields
List the fields you post, by document type. For an EOB that might be payer claim number, date of service, procedure code, allowed amount, plan payment and remark codes. Group them into field types, such as amounts, codes, dates and identifiers, because results will be reported that way.
Match rules
For each field type, write the rule that decides right or wrong: exact match for codes and identifiers, normalized match for amounts and dates, and how blanks are scored. Settle disputes about rules now, not when a result is borderline.
Success criteria
Set a target for each field type, on fields that skip review, at a stated confidence level. A fictional example: "Allowed amount and plan payment: upper limit of the 95% error range below 1% on auto-accepted fields. Codes: below 2%. Review rate across all fields: under 30%." The numbers here are placeholders; yours come from what an error costs your team.
Data handling
Say what data the tool may see and when. One approach is to begin with de-identified documents and move to live documents only once a business associate agreement is signed. The HIPAA and PHI category covers business associate agreements and de-identification in more detail.
How long should a pilot accuracy test run?
A pilot accuracy test should run long enough to process enough labelled examples of each important field type and to see at least one full cycle of your normal document mix. The example below uses four weeks, but volume sets the length: a rare document type takes longer to gather.
The deciding factor is examples, not days. If your success criterion for a field type needs a few hundred clean examples before its error range falls below target, the test runs until it has them. The NIST/SEMATECH e-Handbook page on confidence intervals for proportions gives the Wilson interval used for that calculation; with zero errors, the upper limit at 95% confidence is close to 3.84 divided by the sample size plus 3.84, so 200 clean examples put the upper limit near 1.9%.
Calendar time still matters for two reasons. Document mix changes through the week and the month: payment runs, batch mailings and month-end statements arrive on their own schedule. And people need time to settle into the review screen, so the first week's review times will be slower than later weeks.
Four-week pilot plan (example)
The plan below is a fictional example for a billing team that posts EOBs and claims. Document counts and targets are made up.
| Week | Documents | Fields | What is measured | Decision at the end |
|---|---|---|---|---|
| 1 | 150 de-identified EOBs from the labelled tuning set | All EOB fields in the plan | Per-field accuracy at thresholds 80, 85, 90, 95; review time per field | Pick a starting threshold per field type; fix any match-rule disputes |
| 2 | 150 de-identified claims from the tuning set | All claim fields in the plan | Same measures for claims; per-payer breakdown for EOBs | Confirm thresholds for claims; flag weak document types |
| 3 | 300 documents from the locked test set, EOBs and claims mixed | All planned fields | Error range on auto-accepted fields at the chosen thresholds; review rate | Pass or fail each field type against the success criteria |
| 4 | Live documents in parallel with normal keying, after a signed BAA | Amounts and codes | Auto-accepted fields compared with what staff keyed; review hours | Go or no-go per document type; staffing plan |
Weeks 1 and 2 use a tuning set and week 3 uses a separate locked test set, so the thresholds chosen early are judged on documents they were not tuned on. Week 4 runs alongside your normal process, so nothing depends on the tool yet.
On Schema Care, the first stage of a plan like this maps onto the free plan, which runs on de-identified sample documents; the service does not accept real patient data until a BAA is signed, and every paid plan includes one.
Reading pilot accuracy results by field and threshold
Reading pilot accuracy results by field and threshold means building a table that shows, for each field type and each candidate confidence threshold, two numbers: the error rate on fields that skip review and the share of fields sent to review. Raising the threshold lowers errors and raises review work. The right setting is where both are acceptable.
A threshold trade-off table (fictional results)
This table is invented to show the shape of a result. It covers one field type, allowed amounts, across 2,000 labelled examples.
| Threshold | Sent to review | Auto-accepted | Errors among auto-accepted | Error rate on auto-accepted |
|---|---|---|---|---|
| 80 | 200 (10%) | 1,800 | 36 | 2.0% |
| 85 | 320 (16%) | 1,680 | 17 | 1.0% |
| 90 | 500 (25%) | 1,500 | 6 | 0.4% |
| 95 | 900 (45%) | 1,100 | 2 | 0.2% |
Read it from the success criteria. If the target is an error rate under 1% on auto-accepted amounts and a review rate under 30%, a threshold of 90 meets both in this example; even the Wilson upper limit for 6 errors in 1,500 fields works out to about 0.9%. Moving to 95 halves the error rate but nearly doubles the review load. Whether that is worth it depends on what a wrong amount costs you compared with a reviewer's time.
Read every field type separately
Build the same table for each field type. A single threshold that works for amounts may send far too many names to review, or let too many codes through. You may well end up with different thresholds for different field types. The guide to picking a starting extraction threshold from test data covers how to turn a table like this into a setting.
Check the scores themselves
Look at whether higher scores really mean fewer errors. If fields scoring 95 to 99 fail about as often as fields scoring 85 to 89, the scores are not separating good fields from bad, and no threshold will fix that. That is a finding in its own right, and it belongs in the write-up. The threshold demo on a fictional EOB shows how moving the cut-off changes which fields go to a person.
What if the pilot accuracy test fails on one document type?
When a pilot accuracy test fails on one document type, the choice is between three options: drop that type from the rollout, raise its threshold so more of it goes to review, or end the pilot. Which fits depends on how much volume the type carries and whether the failure is spread across all fields or concentrated in a few.
Start by locating the failure. Break the failing type down by field and by source. Three patterns are common:
- One field fails, the rest pass. A single weak field, such as a remark code on one payer's EOB, can often be handled by a higher threshold for that field alone, or by always reviewing it. The rest of the document can still flow.
- One source fails. If one payer's layout or one clinic's scans account for most errors, the type may be fine elsewhere. Route that source to manual keying and roll out the rest.
- Everything on the type fails. If errors are spread across fields and sources, the tool is not ready for that type.
Then weigh volume. A document type that is 5% of your work can be dropped from the rollout with little loss. If it is half your work, dropping it removes most of the benefit, and ending the pilot may be the honest outcome.
Raising the threshold is the middle path, and its cost is visible in the trade-off table. If reaching the target error rate on that type means sending 70% of its fields to review, the tool saves little time on it. Write that number down rather than declaring the type a pass.
Whatever you decide, record it with the evidence: the field-level table, the threshold you would need, and the review rate at that threshold. A vendor may offer to retest after changes; if so, retest on a fresh labelled set, not the one you already showed them. The reasons a vendor's own figure rarely matches a test like this are covered in the guide to how to read vendor accuracy claims.