Sampling accepted fields for quality checks means pulling a random set of the fields that passed your confidence threshold, checking each one against the source page, and turning the error count into an estimated error rate range. It finds the errors that skipped review without paying a person to re-read every accepted field.
Why sample fields that already passed the confidence threshold?
Fields that passed the confidence threshold still contain some errors, because a confidence score is an estimate, not a guarantee. A confidence threshold is the score a field must reach before it goes straight into your system instead of to a reviewer. Sampling those auto-accepted fields is the only routine way to see the errors that no person looked at.
An extraction tool can be confidently wrong. A field can score high because the text was crisp and the layout was familiar, and still land in the wrong place. Three patterns are common above the line:
- A clean read of the wrong box. The allowed amount and the billed amount sit side by side on many EOBs. A value read perfectly from the neighbouring column scores well and is still wrong.
- A plausible value from a new layout. When a payer changes its form, a date that used to sit in one spot moves. The tool may read another date with full confidence.
- A format the score cannot see. A diagnosis code missing its decimal point, or a member ID with a dropped leading zero, can look complete to a scorer that rates character clarity.
None of these reach a reviewer, because the threshold sends them straight through. If nobody checks, the first sign of trouble is a denied claim, a misposted payment or a patient balance that does not add up. By then the error has moved into other records.
A sample also tells you whether the threshold itself is still right. A threshold set from last quarter's test documents reflects last quarter's layouts and scan quality. A weekly sample is a running check that the cut-off you chose still separates good fields from doubtful ones.
Schema Care stores a source-page link with every field it extracts, so a person checking a sampled field can jump to the exact spot on the page instead of hunting for it; the human review workflow shows how that link and the review log fit together.
How many accepted fields should a quality sample include?
A quality sample should include enough accepted fields that the error rate range it produces is narrow enough to act on. Sample size controls the width of that range, not the error rate itself. A small sample with zero errors can still be consistent with an error rate high enough to matter.
The arithmetic behind the range is standard. The NIST/SEMATECH e-Handbook page on confidence intervals for a proportion gives the common normal-approximation interval and also the Wilson interval, which it says Agresti and Coull recommended "for virtually all combinations of n and p". The Wilson interval is useful for quality samples because it still gives a sensible upper limit when a sample finds no errors at all, where the simple normal formula collapses to zero on both sides.
Sample size table (example numbers)
The table below is fictional example arithmetic. It uses the Wilson interval at 95% confidence (z = 1.96) and ignores the small correction for sampling a large share of a finite week. The counts are made up to show how the range behaves; they are not measured results from any tool.
| Fields accepted per week | Sample size (n) | Errors found | Observed rate | Estimated error rate range (95%) |
|---|---|---|---|---|
| 2,000 | 100 | 0 | 0.0% | 0.0% to 3.7% |
| 2,000 | 200 | 2 | 1.0% | 0.3% to 3.6% |
| 5,000 | 400 | 4 | 1.0% | 0.4% to 2.5% |
| 5,000 | 400 | 12 | 3.0% | 1.7% to 5.2% |
Three things stand out. A sample of 100 with zero errors only tells you the error rate is probably under about 3.7%, which may be too loose if a 2% rate would worry you. Quadrupling the sample at the same observed rate, from 100 to 400, roughly halves the width of the range, because the width shrinks with the square root of the sample size. And the number of fields accepted per week barely matters once the sample is a small share of them; what matters is how many you check.
Choosing the sample size
Start from the decision, not the formula. Write down the error rate that would make you act, for example "more than 2% of accepted amounts wrong". Then pick a sample big enough that its upper limit can fall below that line when things are fine. If the upper limit of a clean week is still above your action line, the sample is too small to reassure you.
Count fields, not documents. A sample of 20 EOBs might hold 400 fields, but the fields on one page share a scan and a layout, so their errors cluster. Spreading the same 400 fields across more documents gives a fairer picture than checking every field on a few pages.
Stratifying the quality sample by field type and score band
A stratified quality sample splits accepted fields into groups, such as field type and confidence score band, and draws a set number from each group. It puts more checking where errors cost more or are more likely, instead of letting the most common field type fill the sample.
A purely random sample mirrors your volume. If 60% of accepted fields are patient names and dates, 60% of your checking effort goes there, while dollar amounts, the fields that change what gets paid, get a small share. Splitting the sample fixes that.
Two splits do most of the work:
- By field type. Group amounts (billed, allowed, paid, patient share), codes (diagnosis, procedure, remark), identifiers (member ID, claim number) and descriptive fields (names, addresses). Give amounts and codes a larger share, because an error there changes money or meaning.
- By score band. Fields that only just passed the threshold are the ones most likely to be wrong. If your threshold is 90, a band of 90 to 94 deserves a heavier draw than fields scoring 99.
When you stratify, report each group's range separately before combining anything. A combined rate needs each group weighted by its real share of accepted fields, or the heavy draw from the borderline band will make the whole week look worse than it is.
A weekly quality-check routine (example)
This step list uses fictional example numbers for a team accepting about 5,000 fields a week.
- Every Monday, export last week's auto-accepted fields only, with field type and confidence score.
- Split them into four groups: amounts, codes, identifiers and descriptive fields.
- Within each group, split again into a borderline band (threshold to threshold plus 4) and everything above it.
- Draw at random: 60 fields from each amount band, 40 from each code band, 30 from each identifier band and 20 from each descriptive band, for 300 fields in total.
- Have a reviewer check each sampled field against its source page and mark it right or wrong using a written match rule.
- Record errors by group and band, compute each group's range, and log the week's results next to the previous weeks.
- On Wednesday, decide on actions for any group whose upper limit crossed its action line.
The match rule in step 5 matters more than it looks. Whether "$1,250.00" and "1250" count as the same value has to be written down before anyone checks a field. How to write that rule, and when close counts as right, is set out in the guide to exact match, normalized match and partial credit.
Quality assurance for medical data entry: acting on sample errors
Quality assurance for medical data entry is only useful if each sample error leads to a decision. Every error found in the sample points to one of three fixes: raise the threshold for that field type, add a rule that catches the pattern, or fix the source, such as a scan setting. Sorting errors by cause tells you which.
Start by reading the errors, not just counting them. For each wrong field, note the field type, the score, the document type, the payer or form layout, and what went wrong. After a few weeks, patterns appear.
When to raise the threshold
Raise the threshold for one field type when its errors cluster just above the current cut-off. If most wrong amounts scored between 90 and 93 and almost none scored above 95, moving the amount threshold to 94 sends the doubtful ones to review. The cost is more review work, so check how many fields sat in that band last week before you move it. Raising the threshold for every field type because one type had errors is the most expensive way to fix a local problem.
When to add a rule
Add a cross-check when errors are confident and spread across all scores. A threshold cannot catch a field that scores 99 and is wrong. Arithmetic can: most EOB lines follow the sums set out in the guide to EOB service line columns from billed to paid, where billed minus adjustments equals allowed. A rule that holds a line for review when its sums disagree catches swapped columns no matter how sure the tool was. The full set of such rules is in the guide to cross field checks that catch extraction errors.
When to fix the source
Fix the source when errors cluster by document type, payer or scanner. A burst of wrong dates from one payer usually means a layout change. A run of misread digits from one site's scans points to a resolution or contrast setting.
When to re-test
If a sample shows a change you cannot explain, re-run your full test against a fixed set of labelled documents rather than leaning on weekly samples. That set, and how to keep it separate from the documents you tune on, is covered in the guide to building a ground truth test set for medical documents.
Quality sampling mistakes: re-checking what reviewers already saw
The most common quality sampling mistake is drawing the sample from all extracted fields, including those a reviewer already approved or corrected. A quality sample of accepted fields must come from auto-accepted fields only, because that is the group no person has checked. Mixing in reviewed fields makes the error rate look lower than it is.
Reviewed fields have, by definition, had a person's attention. Their error rate after review should be low, and if it is not, that is a reviewer training problem with its own fix. When reviewed fields make up part of the sample, they dilute the errors from the auto-accepted group. A week where 30% of accepted amounts came through review will show a better rate than the auto-accepted amounts deserve.
Other mistakes undo the sample in quieter ways:
- Letting the checker pick fields. A person choosing "a few to look at" picks the easy ones or the ones that look odd. Draw at random, with a fixed method you can repeat.
- Showing the checker the score. A reviewer who sees "99" tends to agree. Hide the score when you can, and ask for a fresh read against the page.
- Changing the match rule mid-week. If one checker counts "1250" as right and another counts it as wrong, the week's rate measures the checkers, not the tool.
- Reading one week alone. A single week's range is wide. Look at the trend across four to eight weeks before you change a threshold.
- Never closing the loop. A sample that finds the same error type every week and triggers no change is a cost with no return.
Sampling is one piece of a wider automation programme for billing teams. The revenue cycle automation category covers where review queues, posting and denials work fit together, and the guide to measuring field level extraction accuracy shows how to score fields when you run a full test instead of a weekly sample.