To set a confidence threshold for extraction, run a labeled sample of your own documents through the system, sort every field by its score, and find the lowest score at which accepted fields are wrong rarely enough for your tolerance. Start there, give costly fields such as amounts a higher bar, and re-check after a month of corrections.
What a confidence threshold for extraction decides
A confidence threshold for extraction is the score a field must reach before it skips human review. A field at or above the line goes straight into your system. A field below it waits for a person to compare it with the page. The threshold sets two numbers at once: errors that slip through, and review work created.
Every extracted field arrives with a confidence score, the system's own estimate of how sure it is about that value. What that number measures, and why it can describe a character, a word or a whole field, is explained in the guide to what an OCR confidence score measures.
Moving the threshold trades one cost for the other. Raise it and fewer wrong values get through, but more correct values sit in the queue waiting for a reviewer to approve them. Lower it and the queue shrinks, but some wrong values are posted without anyone looking. There is no setting that removes both costs, so the job is to pick the point where the errors you accept are ones you can live with and the queue is one your team can clear.
The threshold is not a quality guarantee on its own. It only works if higher scores really do mean fewer errors on your documents. Before you trust any line, it is worth checking that scores are calibrated against a labeled sample, so that a field marked 95% is right about as often as the number claims.
How do you pick a starting extraction threshold from test data?
You pick a starting extraction threshold from a labeled test set: documents where someone has already typed the correct value for every field. Run them through the system, record each field's score and whether it matched the correct value, then find the score above which the error rate drops to a level you accept.
A step list for the first threshold:
- Collect a sample that looks like your real mail. Include the payers, layouts and scan quality you actually receive. A sample of clean, recent PDFs will give you a threshold that fails on faxed pages.
- Label every field. Two people typing the same document independently, then settling disagreements, gives a more reliable answer key than one person working alone.
- Keep the sample separate from anything used to train or tune the system. The scikit-learn guide to probability calibration makes the same point about calibrating a model: performance on the data a model was fitted on "would be better than for novel data", so results measured there are too optimistic.
- Run the sample and record three things per field: its type, its score and whether it was right.
- Sort by score and walk down the list. At each candidate threshold, count the wrong fields that would have been accepted above it and the total fields that would have gone to review below it.
- Pick the lowest threshold where accepted errors meet your tolerance. Write the tolerance down first, for example "no more than one wrong allowed amount per 1,000 accepted", so the choice is a decision and not a feeling.
- Record the threshold, the sample and the date. You will compare against this later.
Example arithmetic, with made-up numbers: a sample yields 4,000 fields. Above 90%, 3,400 fields are accepted and 6 of them are wrong. Below 90%, 600 fields go to review. Above 85%, 3,700 are accepted and 19 are wrong. If your tolerance is about 2 errors per 1,000 accepted, 90% passes (6 in 3,400 is under 2 per 1,000) and 85% fails (19 in 3,700 is about 5 per 1,000).
Tuning thresholds per field type: amounts, codes, names and dates
Tuning thresholds per field type means giving each kind of field its own bar, set by what an error in that field costs. A wrong plan payment posts the wrong money to an account. A misspelled patient name may be caught at the next visit. Those two errors deserve different thresholds, even on the same document.
The table below is an example for a billing team. Every value is illustrative, chosen to show the method rather than to recommend numbers.
| Field type | Example error cost | Example starting threshold | Example share sent to review |
|---|---|---|---|
| Amounts (allowed, paid, patient share) | High: wrong money posted, balance disputes | 95% | 14% |
| Codes (procedure, diagnosis, remark) | High: wrong denial reason or rejected claim | 93% | 11% |
| Dates (service, birth) | Medium: timely filing and eligibility checks | 90% | 6% |
| Names (patient, subscriber) | Lower: often caught by matching on other fields | 85% | 4% |
| Checkboxes (consent, yes/no history) | Varies: high when a consent box drives a workflow | 97% | 9% |
Amounts and codes get the highest bars because they feed money and claim decisions directly. Dates sit in the middle. Names can often take a lower bar because a billing system rarely matches a record on name alone; member ID and date of birth usually back it up, so a small name error tends to surface rather than spread.
Per-field thresholds also change the review queue's makeup. With one global line, a reviewer's queue fills with low-risk names. With per-field lines, reviewer time shifts toward the amounts and codes where an error costs the most.
Thresholds for checkboxes and signatures
Thresholds for checkboxes and signatures work differently because the answer is a mark that is either there or not, rather than a string of characters. The score says how sure the system is that the box is ticked or the line is signed. A stray mark, a fold or a scribble across two boxes can push it toward the middle.
A few rules help with marks:
- Score the decision, not the ink. What matters is "consent given: yes or no", not how much of the box is dark. Ask whether the score you see describes the decision.
- Set a high bar when a mark triggers action. A consent checkbox on an intake form can decide whether a record may be shared. A wrong "yes" there costs far more than a wrong "no" on a history question, so its threshold should be near the top of your range.
- Review both directions. Errors on checkboxes run both ways: an empty box read as ticked and a ticked box read as empty. Your labeled sample should include both kinds of example.
- Treat signatures as presence checks. Extraction can tell you whether something is written on the signature line. It cannot tell you whose signature it is. Route uncertain presence calls to a person, and keep any question of who signed outside the extraction step.
Intake forms carry most of the checkboxes a clinic sees. The patient intake category covers form design and how intake data reaches the record.
How much review volume will a 90% extraction threshold create?
The review volume a 90% extraction threshold creates is the share of fields in your labeled sample that scored under 90%, multiplied by your monthly field count. You can estimate it before going live, because the sample already tells you how scores are distributed on your own documents.
A worked example with made-up numbers:
- Monthly volume: 1,200 EOB pages.
- Fields per page: about 25.
- Monthly fields: 1,200 × 25 = 30,000.
- Share under 90% in the sample: 8%.
- Fields to review each month: 30,000 × 0.08 = 2,400.
- Example review time: 10 seconds per field with the source page highlighted.
- Reviewer time: 2,400 × 10 seconds = 24,000 seconds, about 6.7 hours a month.
Run the same arithmetic at 85% and 95% to see the slope. If moving from 90% to 95% doubles the queue but removes only a handful of accepted errors in your sample, the extra review may not be worth it for every field type, which is another reason to set per-field lines.
The estimate assumes your live documents look like your sample. A new payer, a run of faxed pages or a seasonal spike in handwritten forms will push more fields under the line. Turning the queue estimate into a reviewer headcount, with turnaround targets, belongs to how many reviewers a review queue needs.
When one global extraction threshold is the wrong choice
One global extraction threshold is the wrong choice when your documents or fields differ enough that the same score means different things across them. A mix of clean electronic EOBs and faxed intake forms, or of dollar amounts and patient names, will either flood the queue with safe fields or let risky ones through.
The mixes that break a single line most often:
- Document types with different scan quality. Printed EOBs downloaded from a payer portal and faxed referral packets do not produce comparable scores. A line tuned to one is wrong for the other.
- Field types with different error costs. As the table above shows, a single bar treats a remark code and a first name as equally risky.
- Payers with very different layouts. A layout the system reads well and one it reads poorly can sit in the same batch. If the poor one dominates the errors, consider a separate line for it.
- Handwritten and printed fields on one page. Intake forms often combine printed labels with handwritten answers. The handwritten answers will cluster at lower scores.
The other reason a single line fails is miscalibration that differs by field: if amounts are overconfident and names underconfident, one threshold cannot correct both. The guide to confidence score calibration shows how to spot that. Comparing a page-wide score against each field's own score, and why routing should use the field, is covered in field level vs document level confidence.
In Schema Care, you choose how sure the system must be before a field skips review, and the Group plan adds review thresholds per document type; the current plans are listed on the Schema Care pricing section.