Testing with synthetic medical documents means running your extraction pipeline on fictional EOBs, claims, lab reports and intake forms that you built yourself, with invented patients and known correct values. You can share, repeat and score these tests freely, because no real patient appears on any page.
Why test with synthetic medical documents first?
Synthetic medical documents are pages built from scratch with made-up people and values, so they contain no protected health information (PHI). Testing on them first lets developers start before any contract is signed, share failing pages in a ticket, and rerun the same set after every change without asking who may see it.
The PHI question is the first reason. The eCFR definition section, 45 CFR 160.103, defines protected health information as individually identifiable health information held or sent in any form. A real EOB in a developer's test folder is that kind of information. A page about a patient who never existed is not.
Timing is the second. Under the eCFR text of 45 CFR 164.502, a covered entity may disclose PHI to a business associate only with satisfactory assurance, documented in a written contract or other agreement, that the business associate will safeguard it. That agreement is the business associate agreement (BAA), and negotiating one can take weeks. Synthetic documents let you test a vendor or your own code during those weeks.
Repeatability is the third. A test set is only useful if it stays the same between runs, so a change in score reflects a change in the pipeline. Real documents get deleted under retention rules or pulled when access changes. A synthetic set lives in version control and never has to leave.
Schema Care follows the same order. Its Free plan runs on de-identified sample documents, up to 500 pages a month, and real patient data waits for a paid plan with its BAA, as the Schema Care plans and pricing section sets out. Schema Care offers no API, SDK, webhooks, FHIR output, 835 or 837 output, so any pipeline code in a test like this is your own, and every request or file in it should point at fictional data.
Building a synthetic medical document test set
A synthetic medical document test set is a fixed collection of fictional pages, each paired with a file of the values it should produce. Build it from blank layouts, fill them with invented values from written rules, render some as clean PDFs and some as scans, and record the correct answer for every field before you run anything.
The steps below build a 50-document set across four document types. The counts are an example; change them to match what your team receives.
- Pick the mix. For example: 15 EOBs, 15 claims (10 CMS-1500 and 5 UB-04), 10 lab reports and 10 intake forms. Weight the mix toward the documents you key most.
- Make the layouts. Draw three or four layouts per type that copy the structure of what you receive: where the tables sit, how many columns, where totals appear. Do not copy a real payer's or lab's logo, name or address.
- Write a value sheet per document. A spreadsheet row per document, a column per field. Fill it with values that follow the fictional-value rules in the table further down.
- Fill and render. Merge each value sheet row into its layout and save a clean PDF. This gives you the easy version of every page.
- Degrade a share of them. Print about half, then scan or fax them back at different settings. The next section lists the variations worth adding.
- Label the ground truth. The value sheet already holds the right answers. Export it as one answer file per document, with the field names your pipeline returns.
- Run and score. Run the pipeline, compare each output field with its answer, and record exact matches, wrong values and missing fields per field type.
- Freeze a version. Tag the set and the answer files together. Add pages in a new version instead of editing old ones, so earlier scores stay comparable.
Steps 6 and 7 are where most of the thinking goes. Choosing which fields to label, how to label tables, and how to handle a field that is blank on purpose belongs to the guide on building a ground-truth test set for medical documents. Turning matches into scores you can track is covered in the guide to measuring field-level extraction accuracy.
Developers can run the set through a script like the one in the medical OCR API example in Python, which prints each field and lists those under a confidence threshold. That script calls a fictional endpoint, api.example.com, not a real service. If your pipeline also converts output to FHIR, the same answer files let you check the conversion, as the guide to turning a medical document into FHIR describes.
Synthetic document variations that stress an extraction pipeline
Synthetic document variations are deliberate defects added to clean fictional pages so the test set looks like a real inbox. A pipeline that scores well on clean PDFs can still fail on skewed scans, fax noise, stamps, handwriting and multi-page files, so each of these needs its own pages in the set.
Add each variation to a few documents of every type, and record which variation each page carries. That way a drop in score points straight at the cause.
- Skew and rotation. Scan some pages a few degrees off straight, and include at least one upside-down page and one sideways page.
- Fax noise. Send pages through a fax line or a fax-quality setting so text gets speckled, thin lines break and gray areas turn to dots.
- Low resolution. Scan a few pages at a low setting, so small print in remark code glossaries blurs.
- Stamps and notes. Add a "RECEIVED" date stamp over a table, a sticky note over a corner, and a highlighter mark across a total.
- Handwriting. Have colleagues fill a few intake forms by hand with fictional values, in different pens and styles, with one crossed-out and corrected entry.
- Checkboxes. Mark consent and history boxes with ticks, crosses and filled squares, and leave some blank on purpose.
- Multi-page files. Combine a two-page EOB with an unrelated lab report in one PDF, and put one page out of order.
- Blank and near-blank pages. Include a scanned back side with only a page number on it.
Vary one thing at a time where you can. A page that is skewed, faxed and stamped all at once tells you the pipeline failed but not why. Pair each heavily degraded page with a version that carries only one of its defects, and the comparison shows which defect did the damage.
Keep a few clean pages of each type as a control. If clean pages start failing too, the problem is in the pipeline, not in the scan quality.
How do I keep synthetic medical documents obviously fictional?
Keep synthetic medical documents obviously fictional by using values that cannot belong to a real person and that any reader would recognize as invented: test-style names, identifiers that fail their own check rules, domains reserved for examples and round amounts. A page that looks real is a page someone may mistake for PHI.
The table gives one set of rules. They are examples to adapt, not a standard.
| Field | Rule | Example value |
|---|---|---|
| Patient name | Use a word that reads as a test, never a plausible full name | Ada Testpatient |
| Date of birth | Use the first of a month in a year your team picks for test data | 01/01/1950 |
| Member ID | Start every ID with a fixed test prefix | ZZTEST000123 |
| NPI | Use 10 digits that fail the NPI check digit | 1234567890 |
| Claim number | Use a fixed test prefix and a sequence | TEST-CLM-0001 |
| Phone | Use all zeros | (000) 000-0000 |
| Use a reserved example domain | ada@example.com | |
| Address | Use a street and town that announce themselves | 1 Test Street, Sampleton |
| Amounts | Use round numbers that still add up across columns | $200.00 billed, $150.00 allowed |
Two of these rules rest on published standards. The CMS NPI check digit requirements say an NPI is nine digits plus a Luhn check digit, and their worked example gives 3 as the check digit for 123456789, so 1234567890 fails the check. The IETF's reserved example domain names, RFC 2606, set aside example.com, example.net and example.org for use as examples.
An invalid identifier has a cost. A pipeline that validates NPIs will flag every test NPI as an error. Decide whether your test set checks the validation, in which case most pages carry invalid NPIs on purpose, or the extraction, in which case you mark the test prefix as allowed in your test environment.
Synthetic documents are different from de-identified ones. De-identified documents start as real records and have identifiers removed under the method in the eCFR text of 45 CFR 164.514, whose safe harbor list runs from names and dates to any other unique identifying number. The steps for that process are in the guide to PHI de-identification.
Where do synthetic documents fall short of real ones?
Synthetic documents fall short of real ones because you can only build the problems you already know about. Real inboxes hold layouts you have never seen, payer quirks, unusual codes, and combinations of defects no one thought to fake. A pipeline that passes every synthetic test still needs a check on real or de-identified pages.
Layout variety is the biggest gap. A team can draw a handful of EOB layouts, while the documents a billing office receives come from many payers, each with its own column order and glossary format. Expect the first real batch to include layouts the synthetic set never covered, and add a fictional copy of each new layout to the next version of the set.
Value variety is the second. Invented values tend to be tidy. Real ones include long hyphenated names, a member ID with a letter O where a zero belongs, a lab result reported as a range or with a comparison sign, and a total that does not add up because the payer made an error.
Volume patterns are the third. Synthetic sets are balanced on purpose. A real month may be mostly one document type, with one payer's EOBs making up a large share. Scores weighted by the real mix can differ from scores on a balanced set.
The fix is a staged path. Run synthetic tests first and in every build. Then test on de-identified samples, which keep real layouts and quirks with the identifiers removed; choosing those pages is covered in the guide to picking de-identified sample documents for a vendor trial. Move to live documents only after the BAA is signed, and keep the synthetic set running afterwards as your regression check.