Inter rater reliability for document reviewers is how often two reviewers make the same call on the same extracted field when they work independently. Measure it by giving both a shared set of fields, counting matching decisions, then correcting for chance with Cohen's kappa. When agreement is low, check first for a field rule with two readings.
What inter rater reliability means for document reviewers
Inter rater reliability for document reviewers compares two independent decisions on one extracted field. Each reviewer sees the same proposed value and the same source page, then approves it or corrects it. The check asks whether they made the same decision and, when both corrected, whether they typed the same final value.
The unit of comparison is the field, not the document. An EOB with 40 fields gives 40 comparisons. A document-level score hides where the trouble is, because one disputed date and 39 easy matches look like a good page. Field-level results show which field, on which document type, splits your team.
The decision being compared needs a fixed set of outcomes. If your queue records only whether a value changed, two are enough: approved as extracted, or changed. If your team also flags unreadable fields, make that a third outcome and score it the same way. A field where both reviewers changed the value but typed different results counts as a disagreement, because the data that reaches your system would differ.
Reliability is not the same as accuracy. Two reviewers can agree perfectly and both be wrong, for example when both follow an outdated rule. Agreement tells you the rule produces one answer in practice. Whether that answer is correct is a separate question, settled against the page and your written field rules.
This differs from scoring a new reviewer against a key. When a trainee is checked against an experienced reviewer or an answer set, one side is treated as right. The guide to what reviewer training for document review must cover handles that one-way scoring. Inter rater reliability treats both reviewers as equals and looks for the rule that let them split.
The same idea appears in two neighbouring tasks. Teams that build a test set have two people label every field and settle the differences, as described in labeling ground truth twice and resolving disagreements. Registry programs check abstractors the same way, covered in how agreement between abstractors is measured. This post applies it to the live review queue.
Percent agreement vs kappa for reviewer reliability
Percent agreement for reviewer reliability is matching decisions divided by fields compared. It is easy to read but inflated when most fields are easy. If two reviewers each approve 95% of fields independently, they agree on about 90.5% of fields by chance alone (0.95 × 0.95 + 0.05 × 0.05). Cohen's kappa removes that expected chance agreement.
The formula is short. In McHugh's paper on interrater reliability and the kappa statistic (Biochemia Medica, 2012), kappa is Pr(a) minus Pr(e), divided by 1 minus Pr(e). Pr(a) is the observed agreement. Pr(e) is the chance agreement, computed from each reviewer's totals for each outcome. The same paper notes that Jacob Cohen introduced kappa in 1960 because percent agreement cannot account for chance agreement, and that the greater the expected chance agreement, the lower the resulting kappa.
Kappa runs from minus 1 to plus 1. McHugh reports Cohen's suggested bands: 0.01 to 0.20 none to slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial and 0.81 to 1.00 almost perfect. She then argues those labels may be too lenient for health-related studies, and that any kappa below 0.60 indicates inadequate agreement.
A worked check on a fictional EOB
The example below is fictional. Two reviewers, Reviewer A and Reviewer B, each review the same 20 fields from an example EOB for a fictional patient, Dana Example, payer claim number 26-0915-7741. Each field is scored as Approve (kept the proposed value) or Correct (changed it). Where both corrected, they typed the same final value.
| B approved | B corrected | A's total | |
|---|---|---|---|
| A approved | 15 | 2 | 17 |
| A corrected | 1 | 2 | 3 |
| B's total | 16 | 4 | 20 |
Step 1, observed agreement. The two agree on 15 + 2 = 17 fields. Pr(a) = 17 / 20 = 0.85, or 85%.
Step 2, chance agreement for Approve. A approved 17 of 20 fields and B approved 16 of 20. The chance that both approve a field at random is (17 / 20) × (16 / 20) = 272 / 400.
Step 3, chance agreement for Correct. A corrected 3 of 20 and B corrected 4 of 20, so (3 / 20) × (4 / 20) = 12 / 400.
Step 4, total chance agreement. Pr(e) = (272 + 12) / 400 = 284 / 400 = 0.71.
Step 5, kappa. (0.85 − 0.71) / (1 − 0.71) = 0.14 / 0.29 = 0.48, rounded to two places.
An 85% match rate sounds solid. A kappa of 0.48 falls in Cohen's moderate band and below the 0.60 line McHugh treats as adequate. The gap exists because most of the 20 fields were easy approvals, so a lot of the agreement would have happened anyway. The three fields they split on carry most of the information, and the section on rule changes below works through each one.
Which number to report
Report both. Percent agreement tells a supervisor how often the queue gets the same outcome. Kappa tells you whether that agreement beats what easy fields would give anyway. Report them per field type, not only overall, since a strong overall figure can hide one field with frequent splits.
How do you run an inter rater reliability check without doubling the queue?
An inter rater reliability check runs without doubling the queue by sending a small, fixed share of items to a second reviewer who works blind to the first decision. Pick the shared items at random within each field type, keep the count steady from month to month, and compare decisions only after both reviewers finish.
A practical starting point, offered as an example rather than a standard, is 20 to 40 shared fields per reviewer pair each month, drawn across your main document types. That is a few minutes of extra work per reviewer, small next to a normal day's queue. Raise the count for a field type when you are testing a new rule for it.
Choose items at random inside each field type. If you let a supervisor pick interesting fields, the sample fills with hard cases and the score drops for reasons that have nothing to do with your normal queue. Drawing a set number from each field type keeps rare fields, such as remark codes, in the sample; the method is explained in stratifying the quality sample by field type and score band.
Keep the second reading independent. The second reviewer should not see the first decision, the first reviewer's name, or any note left on the field. Once one can see the other's answer, the check measures deference instead of agreement.
Pair reviewers in rotation. If the same two people are always paired, they may drift together and score well while the rest of the team reads a field differently. Rotating pairs each month shows whether a rule is understood the same way across the whole team.
The data you need is already in a good correction log. Each review entry should hold the field, the proposed value, the final value, the action and who took it; the section on what an audit trail for reviewer corrections should record lists the full set. In Schema Care, every approval and correction is logged with who made it and when, which supplies the reviewer decisions you compare. The human review steps on the Schema Care homepage show where that record comes from.
Which fields do document reviewers disagree on most?
Document reviewers are most likely to disagree on fields where the written rule allows two reasonable answers. Check dates with ranges or missing years, abbreviations that can be kept or expanded, and partial values split across lines or columns first. A clear numeric field printed once, such as a billed amount, leaves less room for a split.
Dates. A service line may print a range, a date without a year, or a date in a different order from your system. Two reviewers can both read the page correctly and still record different values, because the rule never said which date in a range belongs in the field.
Abbreviations. Payer names, provider names and remark descriptions can be shortened on the page. One reviewer copies "BCBS" as printed; another types the full plan name. Both feel careful. Without a rule that says "as printed" or "expand to the master list", they will keep splitting.
Partial values. Claim numbers and member IDs sometimes wrap onto a second line, or a value sits half inside a box. One reviewer approves what was extracted; another joins the fragments. Neither is careless. The rule simply did not say what to do with a value that crosses a line.
Faint or damaged values. A smudged digit invites one reviewer to correct it and another to flag it. That case has its own post: what reviewers do with illegible fields covers the allowed outcomes when the page cannot settle the value.
Fields with a formatting choice. Leading zeros, dashes inside IDs and modifiers written beside a procedure code all invite two spellings of the same fact. A format rule per field removes the choice.
Turning reviewer disagreements into clearer field rules
Reviewer disagreements become clearer field rules when you ask, for each split, whether the written rule allowed both answers. If it did, the rule has a gap and needs a sentence. If the rule was clear and one reviewer missed it, that is a reviewer error and a coaching note, not a rule change.
Sort splits before acting on them. Put every disagreement from the month in a list with the field, the two decisions and the source page. Group them by field. A field with several splits across different pairs points at the rule. A field where one reviewer is always the odd one out points at that reviewer.
The three splits from the worked example
The three disagreements from the fictional EOB above each led to a different change. All values are fictional.
| Field | Reviewer A | Reviewer B | Cause | Rule change |
|---|---|---|---|---|
| Line 3 date of service | Approved 10/03/2026 | Corrected to 10/05/2026 | Page printed "10/03/26 to 10/05/26"; rule did not say which date goes in the field | First date goes in "from date", last date in "to date" |
| Payer name | Corrected to the full plan name | Approved "BCBS" as extracted | Rule was silent on abbreviations | Record the payer as printed; the master list is matched later |
| Payer claim number | Approved 26-0915-77 | Corrected to 26-0915-7741 | Number wrapped onto a second line; extraction kept the first part | Join a wrapped identifier when the second line has no label of its own |
Each change is one sentence added to the field rules, with the date it took effect and an example image. Re-check the same field types the following month. If the splits stop, the rule worked. If they continue, the sentence is still unclear, and a short team review of five real examples can show why.
Do not settle a split by asking who is more senior. The answer comes from the page and the rule. When the rule cannot decide, write the rule, then decide.
Rule changes also affect what gets a second look. Fields that keep splitting after a rule change are good candidates for a second reviewer, which the post on second level review for high risk fields explains how to route.
When low inter rater reliability is not the reviewers' fault
Low inter rater reliability is not the reviewers' fault when the source page itself cannot settle the value. A faint fax, a stamp over a column, or a printed value that is ambiguous on its face will split careful reviewers no matter how good the rule is. The fix belongs at intake or with the sender.
Recognise this case by its pattern. Splits concentrate on documents from one source, one fax line or one scanner, rather than on one field rule. Both reviewers may have left a note saying the value was hard to read. Changing the field rule does not move the score.
Remove those items from the reliability score, or report them separately. Counting them as disagreements punishes reviewers for page quality and hides the rule gaps you can actually fix. Keep a separate count of unreadable items per source, because that number is useful when you talk to a payer or a scanning vendor.
A second example is a value that is clear but genuinely ambiguous. A date printed as 03/04/26 means different days in different date orders. If the document gives no other clue, no rule written at your desk can settle it. Record it as unresolved, and treat it the same way as an unreadable value.
Be careful with small samples, too. A kappa from 20 fields moves a lot when one decision changes. In the worked example, suppose the date split had gone the other way, with both reviewers approving. Agreement becomes 18 / 20 = 0.90, both reviewers approve 17 fields, chance agreement becomes (289 + 9) / 400 = 0.745, and kappa becomes 0.155 / 0.255 = 0.61. One field moves the result from moderate to substantial. Watch the trend across several months before deciding a reviewer pair or a field rule has a real problem.