To build de-identified sample documents for a pilot, pick one page per layout you receive, such as each payer's EOB and each lab's report, then strip every identifier or rebuild the page with fictional values. Label each sample with its type, layout and edge cases, log the method used, and keep real patient pages out until a BAA is signed.
Sourcing de-identified sample documents: redacted real pages or fictional ones?
De-identified sample documents come from two sources: real pages with the identifiers removed, or fictional pages built to look like the real ones. Redacted real pages give the fairest test of layout and print quality. Fictional pages carry no patient risk and can be built to order, but they tend to be cleaner than anything that arrives in the mail.
Redacted real pages
A redacted real page keeps everything a document tool has to cope with: the payer's actual layout, the fax noise, the skewed scan, the handwritten note in the margin. That is what makes it a fair test. The cost is effort and care. Someone authorized to see the original has to remove each identifier, check the result and record what was done, and the work has to happen inside your own systems, because the original is still PHI.
The standard the redaction has to meet is set out in 45 CFR 164.514 on the eCFR. Paragraph (b)(2) lists the identifiers that must be removed under the Safe Harbor method, from names and dates to "any other unique identifying number, characteristic, or code", and adds that the covered entity must not have actual knowledge that what remains could identify the person. Paragraph (b)(1) describes the other route, a documented determination by a person with statistical expertise. The removal technique itself is covered in the guide to redacting PHI from PDFs step by step.
Fictional pages
A fictional page is built from a blank form or a real layout, filled in with invented names, member IDs, dates and amounts. It has no patient behind it, so there is nothing to de-identify. It suits forms with public templates, such as claim forms, and it is the only option when no real example of an edge case exists yet. How to generate fictional test documents in bulk belongs to the developer and integration guides.
Many teams mix the two: redacted real pages for the layouts that matter most, and fictional pages to fill gaps. The broader argument for keeping real patient data out of an evaluation altogether is made in the clinical records category, so this post assumes that choice and focuses on the build.
How many layouts should pilot sample documents cover?
Pilot sample documents should cover the layouts you receive, not a large volume of any one of them. One clean page from each payer, lab or form version you see every month tells you more than fifty copies of the same EOB. Start from a count of layouts, then add pages for the edge cases that cause errors today.
Count layouts by source. An EOB from one national payer and an EOB from a regional plan are two layouts even though both are EOBs. Two labs print reference ranges and abnormal flags in different places. Your own intake form may exist in a current and an older version that patients still fill in.
Weight the set toward the layouts that make up most of your volume, because those decide whether a tool saves time. Then add the ones that hurt: the payer whose remark codes run onto a second page, the lab that faxes reports at low resolution, the intake form that comes back handwritten.
A reasonable sample set for a small billing team might look like this. The counts are examples, not a rule.
| Document type | Layouts to include | Edge cases to add | De-identification method |
|---|---|---|---|
| EOB | Top 5 payers by volume, plus 1 low-volume payer | Multi-page EOB, claim split across pages, denied line | Redacted real pages, Safe Harbor list |
| Claim (CMS-1500 or UB-04) | Your own printed claims, 1 per form type | Corrected claim, handwritten claim, faxed copy | Fictional values on the blank form |
| Lab report | Top 3 labs by volume | Cumulative report, faxed page, abnormal flags | Redacted real pages, Safe Harbor list |
| Intake form | Current and previous version | Handwritten entries, skipped fields, consent checkboxes | Fictional values, filled in by hand |
A set built this way may hold only twenty or thirty pages. That is fine, because every row tests something specific, and you can say afterwards which layouts worked and which did not.
Preparing pilot sample documents step by step
Preparing pilot sample documents takes five steps: collect originals inside your own systems, strip or rebuild each one, label it, log what was done, and check it before it leaves. Each step is done by someone allowed to see the original, and nothing reaches the vendor until the last check passes.
- Collect. Pull candidate originals from your document system to a restricted folder. Choose them against the layout table above, not by date.
- Strip or rebuild. For a redacted page, remove every identifier on the Safe Harbor list, including the ones in fax headers, footers and margins. For a fictional page, fill a blank layout with invented values. Use obviously invented values, such as an example patient named Jordan Example with member ID ZZT000004417 and claim number SAMPLE-0042, so nobody mistakes them for real data.
- Label. Give each file a neutral name such as
eob-payer03-multipage-01.pdf. The name records the type, layout and edge case without any patient detail. - Log. Keep a sheet with one row per sample: filename, document type, layout, edge cases, method, who prepared it and who checked it.
- Check. Extract the text, search it for the identifiers you removed, read the metadata and look at every page.
Keep a key, or do not
Some teams want to link a sample back to its original so they can compare a tool's output with what they posted. Paragraph (c) of 45 CFR 164.514 allows a re-identification code only if it is not derived from information about the person and is not disclosed or used for any other purpose. The simplest approach is a random sample number on your own log, with the mapping kept inside your systems and never sent with the files.
Write down the expected answers
For each sample, record the values a correct extraction should return: the allowed amount on each service line, the result and unit for each lab test, the checkbox states on the intake form. Without those, a pilot turns into an impression. With them, you can score each field.
What should you test with de-identified pilot samples before a BAA?
De-identified pilot samples are good for testing layout handling, field coverage, confidence behaviour and the review workflow. They cannot show how a tool handles real volume, real image quality across a month, or your actual payer mix. Use samples to decide whether a tool deserves a contract, then confirm the rest on live documents under a BAA.
These results are meaningful on samples:
- Field coverage. Does the tool return every field you key today, for every layout in the set?
- Field accuracy against your expected answers. Score each field as right, wrong or missing.
- Confidence behaviour. When the tool is wrong, does it say it is unsure? A tool that is confidently wrong on a hard layout is a bigger problem than one that flags the field.
- The review step. Can a reviewer see where each value came from on the page and correct it quickly?
- Edge cases. Multi-page EOBs, handwritten intake fields and faxed lab pages, one at a time.
These results wait for live documents:
- Throughput and turnaround at your real monthly volume.
- How often each layout actually occurs, which decides where the time savings land.
- Integration with your billing system or EHR, which usually needs real accounts to test.
The reason real pages wait is contractual. Under 45 CFR 164.504(e) on the eCFR, a contract with a business associate must set the permitted uses of PHI and require safeguards, incident reporting and the return or destruction of the information when the contract ends. The questions to ask a vendor before that contract are covered in the guide to choosing a BAA OCR vendor, and the security review that often runs alongside it is in the healthcare vendor security questionnaire.
Schema Care follows the same order. The homepage puts it in one line: "You start on de-identified samples and move to live documents after a BAA." Its free plan for de-identified sample documents covers up to 500 pages a month, and every paid plan includes the BAA.
When de-identified pilot samples mislead the results
De-identified pilot samples mislead when the de-identification itself changes the page. Redaction boxes can hide the very fields a tool needs, missing values can trigger errors that real pages never cause, and fictional pages can be so tidy that any tool looks good. Each distortion can push the results either way.
Redaction marks over the wrong fields
A redaction over the patient name is fine. A redaction over the whole claim block also covers the claim number and service dates that a tool uses to group service lines. The tool then looks worse than it is. When you redact, remove the identifier and leave the label and the surrounding structure, so the tool still sees a claim block with a name field in it.
Blank fields that real pages never have
Removing a date of birth from an intake form leaves an empty field. A tool that flags empty required fields will flag it, and that flag tells you nothing about real use. Either replace removed values with fictional ones of the same format, or exclude those fields from scoring and note it in the log.
Fictional pages that are too clean
A fictional EOB typed into a crisp digital template has no skew, no fax lines, no stamps and no second page. A tool that reads it perfectly may still struggle with the same payer's real mail. Print fictional pages and scan or fax them back, add a handwritten note, and include at least one multi-page example, so the set looks like your inbox.
Samples chosen because they are easy
The pages that are quickest to de-identify are often the simplest ones. If every sample is a one-page EOB with three service lines, the pilot measures the easy case. Go back to the layout table and check that each hard row has at least one page.
Reading the results fairly
Score each sample against the expected answers you wrote down, by layout and by field. Mark any field you excluded because of de-identification. A result such as "every field right on the top five payers, two misses on handwritten intake fields, both flagged for review" is something you can act on. "It looked accurate" is not.