A medical document automation pilot checklist sets the success criteria before you start, picks de-identified sample documents that match your real mix, runs them through the software with your reviewers for a few weeks, and measures field errors and review time. At the end, the numbers tell you whether to buy, not the demo.
What should a medical document automation pilot prove?
A medical document automation pilot should prove three things on your own documents: that the software reads the fields you need well enough, that uncertain fields reliably reach a person, and that the total time spent, including review, is lower than keying by hand. Agree how each will be measured before the first document goes in.
Writing the criteria down first keeps the pilot honest. Without them, a pilot drifts into a general impression, and impressions are shaped by the best demo moment rather than the average document. Each criterion needs a measure and a pass line that the team agrees to in advance.
A set of criteria might look like this. The pass lines are examples for illustration; set your own based on what your team needs.
| Criterion | How it is measured | Example pass line |
|---|---|---|
| Field correctness after review | Compare final values with the source page on a sample | No wrong values reach the billing system in the sample |
| Uncertain fields reach review | Check that every error found was flagged or caught by a reviewer | Every error in the sample was below the threshold or corrected in review |
| Review share | Share of fields sent to review at the chosen threshold, by document type | Low enough that review time fits in the team's day |
| Total time per page | Review and exception time per page compared with keying time | Lower than the timed manual baseline |
| Traceability | Pick any value and find its source on the page and its approval in the log | Every value checked can be traced |
Two questions sit outside the numbers and still belong in the criteria. Do the reviewers find the review screen workable? And does the vendor answer questions and fix problems within the pilot's timeline? A short survey of the reviewers and a log of support requests cover both.
Some background helps the team agree on terms. The guide to what medical OCR does and where its job ends explains the reading step, and the post on structured vs unstructured healthcare data explains why some documents produce clean fields and others need more interpretation.
Picking sample documents for a medical automation pilot
Picking sample documents for a medical automation pilot means choosing de-identified documents that match the mix you really receive, in roughly the same proportions, including the difficult ones. A sample of clean, recent, typed pages will make any product look good and tell you little about the faxes and handwriting your team handles every day.
Match your real mix
Start from a count of a typical month's documents by type and source: EOBs from each major payer, claims, lab reports, intake forms and anything else you key. Then draw samples in the same shares. If a third of your EOBs arrive by fax, a third of your sample EOBs should be faxed. Include the troublemakers on purpose: multi-page EOBs, skewed scans, handwritten forms, rare layouts and packets with several documents in one PDF.
Size the sample sensibly
A sample has to be large enough that each document type you care about appears many times, so a single odd page does not decide the result. It also has to be small enough that your team can check the extracted values by hand. A practical approach is to set a number per document type rather than one total, and to keep a separate holdout set that the vendor never sees until the final week.
De-identify before anything leaves your hands
De-identified documents are documents with the information that identifies a patient removed. The HIPAA Privacy Rule describes two ways to reach that standard in 45 CFR 164.514(b) on eCFR. One is a determination by a qualified expert that the risk of identification is very small. The other, often called the safe harbor method, removes a listed set of identifiers, with no actual knowledge that what remains could identify the person.
The safe harbor list runs from (A) to (R), 18 items in all. It includes names, geographic subdivisions smaller than a state, all elements of dates except the year for dates directly related to the person, phone and fax numbers, email addresses, Social Security numbers, medical record numbers, health plan beneficiary numbers and account numbers. On medical paperwork these identifiers sit in headers, claim blocks and margins, so redaction has to cover the whole page, including handwritten notes and fax header lines. This describes what the regulation lists; your compliance or privacy lead decides which method you use and signs off on the sample.
The medical document automation pilot checklist, week by week
The medical document automation pilot checklist below runs over four weeks, with an owner and a done-when line for each item. It assumes de-identified samples, a vendor product already shortlisted and a small team from billing, IT and compliance. Stretch the weeks if your volume or approvals need more time.
Week 0: before the pilot starts
| Item | Owner | Done when |
|---|---|---|
| Write the success criteria and pass lines | Billing manager | Signed off by billing, IT and compliance |
| Time the manual baseline per document type | Billing lead | Average keying minutes recorded for each type |
| Count the monthly document mix | Billing lead | A table of document types, sources and shares exists |
| Agree what data the vendor may receive and when | Compliance lead | Written note: de-identified only until a BAA is signed |
| Review the vendor's security answers | IT lead | Questions answered in writing, with a status for each control |
Week 1: samples and setup
| Item | Owner | Done when |
|---|---|---|
| Pull samples that match the mix, plus a holdout set | Billing lead | Sample and holdout sets saved separately |
| De-identify every sample page | Compliance lead | Every page checked against the chosen method |
| Set up reviewer accounts and the starting threshold | IT lead | Each reviewer can sign in and see a test document |
| Train reviewers on the review screen | Vendor and billing lead | Each reviewer has approved and corrected test fields |
Week 2: first run
| Item | Owner | Done when |
|---|---|---|
| Run the sample set through the software | Vendor | Extracted values and confidence scores returned for every page |
| Reviewers work the review queue | Reviewers | Queue empty, with time spent recorded |
| Check a sample of accepted fields against the page | Billing lead | Errors found are logged by field and document type |
| Log every support request and response time | IT lead | A dated list of issues exists |
Week 3: tune and rerun
| Item | Owner | Done when |
|---|---|---|
| Adjust the threshold based on week 2 errors | Billing manager | New threshold recorded with the reason |
| Rerun the samples at the new threshold | Vendor | New review share and error counts recorded |
| Test traceability on a random set of values | Compliance lead | Each value traced to its page and its log entry |
Week 4: holdout and decision
| Item | Owner | Done when |
|---|---|---|
| Run the holdout set the vendor has not seen | Vendor | Results returned without any tuning on the holdout |
| Compare results with the pass lines | Billing manager | A one-page summary marks each criterion pass or fail |
| Calculate the cost comparison with review time included | Billing manager | ROI worked with pilot numbers |
| Decide: buy, extend the pilot or stop | Whole team | Decision and reasons recorded |
If the decision is to go ahead, the next step is the business associate agreement and a plan for moving from samples to live documents. What a BAA should cover, and why it comes before any live patient data, is covered in the HIPAA and PHI category.
Who belongs on a document automation pilot team?
A document automation pilot team needs people from billing, IT and compliance, plus one person who owns the decision. Billing knows the documents and does the review. IT handles accounts, files and security questions. Compliance decides what data may be shared and signs off on de-identification. Keep the team small enough to meet weekly.
Billing manager (decision owner). Writes the success criteria, sets the threshold and makes the final call. This person should have the authority to buy or stop.
Billing lead. Times the manual baseline, pulls samples and checks extracted values against the page. Usually the person who knows the documents best.
Reviewers. Two or three of the people who key documents today. They work the review queue and report what slows them down. Their view of the review screen matters as much as any number.
IT lead. Sets up accounts, moves files safely, reviews the vendor's security answers and logs support requests.
Compliance or privacy lead. Decides what may be shared at each stage, approves the de-identification method and keeps live data out until the contract is signed.
Vendor contact. One named person on the vendor side who runs the documents and answers questions within the pilot's timeline. Before choosing a vendor to pilot with, ask the medical OCR vendor questions about review, confidence and audit trails.
Schema Care works this way too: it starts on de-identified samples and accepts live documents only after a BAA is signed. Its human review workflow shows what your reviewers would work with.
Pilot mistakes that make medical document automation look better than it is
Pilot mistakes that make medical document automation look better than it is usually come from the sample or the measurement. Cherry-picked documents, review time that nobody timed, tuning on the test set and a vendor running the pilot alone all inflate the result, and the gap shows up after you have signed.
- Cherry-picked samples. Clean, recent, typed pages from one payer will read well in any product. If the sample does not match your real mix, including faxes and handwriting, the result does not apply to you.
- Unmeasured review time. Counting the fields that went straight through and ignoring the time spent on the ones that went to review overstates the saving. Time the review queue every day of the pilot.
- Tuning on the test set. If the vendor sees the documents the decision rests on and adjusts for them, the result reflects that tuning, not your next month of documents. Keep a holdout set back until the last week.
- Checking only flagged fields. Errors can hide in fields the software was confident about. Check a sample of accepted fields too, or you will not know how many errors slip past the threshold.
- The vendor runs everything. If your reviewers never touch the review screen, you learn nothing about how it fits their day. Your own people should work the queue.
- No baseline. Without timed manual keying from before the pilot, there is nothing to compare against, and any time figure looks good.
Measuring errors properly is a topic of its own. How to run an accuracy test, including how to sample and what to count as an error, is covered in the human review category.