Batch Processing Medical Documents via API

Batch processing medical documents via API works best as many small, tracked jobs. Check each file's format and size first, upload every document with its own idempotency key, retry failures with backoff, record every job ID in a manifest, and reconcile results against that manifest so no document is lost or processed twice.

This guide is for developers building an uploader against a document API. Schema Care offers no API, SDK, webhooks, FHIR output, 835 output or 837 output, and nothing here describes a Schema Care endpoint. Every request, response and code sample is a fictional example written against https://api.example.com, with fictional documents and patients.

Supported input formats for medical document batches

The usual input formats for a medical document batch are PDF, TIFF and image files such as JPEG or PNG, and each API publishes its own list along with size and page limits. Check each file against that list before upload, by its content and not only its extension. A file rejected here costs nothing.

Check content with the file's first bytes. RFC 8118 on the application/pdf media type says all PDF files start with the characters %PDF- followed by a version number, such as %PDF-1.7. RFC 3302 on the image/tiff media type gives the TIFF magic numbers, 49 49 2A 00 for little-endian files and 4D 4D 00 2A for big-endian ones, and lists fax among the applications that use the format. Faxed referrals and scanned records can therefore arrive as TIFF files.

A pre-upload check per file catches the usual problems:

Encryption is worth checking early. RFC 8118 notes that PDF supports encryption and digital signatures, and a password-protected file usually cannot be read by an extraction service at all.

When a file is too large anyway, the API may refuse it. RFC 9110 on HTTP semantics defines 413 Content Too Large for a request whose content is larger than the server is willing or able to process. Treat 413 as a reason to split the file, never as a reason to retry the same request.

Combined files are a separate problem. A single PDF holding an EOB, a lab report and an intake form should usually be split into documents before upload, which the guide on how to split a multi document medical PDF covers. Scan quality decides how well any of these files can be read, and the guide on how to scan paper medical records for OCR covers resolution and settings.

Retries and idempotency keys for document uploads

An idempotency key lets a client retry a document upload without creating a duplicate job. The client generates a unique key per document and sends it with every attempt. If the server already processed that key, it returns the original result instead of starting a second job. Without a key, a timeout leaves the upload in doubt.

The problem comes from HTTP itself. RFC 9110 counts PUT, DELETE and safe methods as idempotent, but not POST, and says a client should not automatically retry a non-idempotent request unless it has some way to know the request is actually idempotent. Uploads are usually POST requests.

One answer is an Idempotency-Key request header, which is not an HTTP standard. It was described in an IETF working group draft, draft-ietf-httpapi-idempotency-key-header, whose latest version, 07, has since expired without becoming an RFC. The draft is still a useful reference for behavior:

• The key must be unique and must not be reused with a request that has a different payload. A UUID or a similar random value is recommended.
• If a request is retried after the first one completed, the server should return the result of the earlier operation.
• If a request is retried while the first is still running, the server should return 409 Conflict.
• If a key is reused with a different payload, the server should return 422.
• The server should publish how long it keeps keys.

Check the API's own documentation for which of these it follows.

One choice makes batch retries much safer: store the key with the file before the first attempt. A random UUID created in memory is lost if the uploader crashes, and the rerun creates new keys and duplicates. Writing the key to a manifest first means a rerun reuses it. Some teams derive the key from the batch ID and a hash of the file's bytes, which gives the same key on every rerun and a different key if the file changes, matching the draft's rule against reuse with a different payload.

An example batch uploader, step by step

The steps below upload a folder of fictional PDFs, such as eob_ex_0001.pdf to eob_ex_0400.pdf. The code is pseudocode against an example endpoint, POST https://api.example.com/v1/documents.

1. List the folder and run the pre-upload checks on each file.
2. Write a manifest row per file: path, byte hash, idempotency key, status pending.
3. Upload each pending file with its key, a few at a time.
4. On success, store the returned job ID and set status uploaded.
5. On 429 or 5xx, wait and retry with the same key. On 4xx other than 409 and 429, mark the file rejected with the reason.
6. After all uploads, poll or listen for job results and record each final state.
7. Write a reconciliation report comparing the manifest with the results.

# Example pseudocode. Endpoint and fields are illustrative.
batch_id = "batch_ex_2026_10_15"
manifest = load_or_create_manifest(batch_id)

for file in list_pdfs("incoming/eobs"):
    if file.path in manifest: continue          # rerun: keep the old key
    problem = precheck(file)                    # magic bytes, size, pages, password
    key = sha256(batch_id + sha256_file(file))  # same file, same key
    manifest.add(file.path, key, status = problem ? "skipped" : "pending", reason = problem)
manifest.save()

for row in manifest.rows(status = "pending"):
    for attempt in 1..6:
        resp = POST "https://api.example.com/v1/documents"
                    headers { "Idempotency-Key": quote(row.key) }
                    body    file_bytes(row.path)
        if resp.status in (200, 201, 202):
            row.job_id = resp.json.job_id; row.status = "uploaded"; break
        if resp.status == 409:                  # first attempt still running
            sleep(backoff(attempt)); continue
        if resp.status == 429 or resp.status >= 500:
            sleep(resp.retry_after or backoff(attempt)); continue
        row.status = "rejected"; row.reason = resp.status; break
    manifest.save()                             # save after every file

write_report(reconcile(manifest, fetch_job_states(manifest.job_ids())))

backoff(n) = min(60, 2 ** n) + random(0, 1)     # seconds, with jitter

How many documents should one batch request hold?

One document per upload request is usually the right size for a batch of medical documents. Each file gets its own job ID, its own idempotency key and its own failure, so one bad file cannot block the rest. Group them into a batch in your own manifest, and control speed with concurrency rather than with request size.

Some APIs also offer a batch endpoint that takes many files in one call and returns one batch ID. That saves requests, but it couples the files: a timeout on a 200-file request leaves all 200 in doubt, and a single rejected file may fail the whole call. If you use one, keep batches small, and still record a job ID per document from the response.

Rate limits set the pace either way. RFC 6585 on additional HTTP status codes defines 429 Too Many Requests for a client that "has sent too many requests in a given amount of time," and says the response may include a Retry-After header. RFC 9110 says Retry-After is either a date or a number of seconds. A sound uploader:

• Starts with a small number of parallel uploads, such as four.
• Honors Retry-After exactly when the server sends it.
• Backs off with jitter when it does not, so many workers do not retry at the same moment.
• Lowers concurrency after repeated 429 responses instead of only waiting.

Page counts can matter for cost as well as speed. Schema Care's own plans, for example, are priced by the pages processed each month, as the plans and page allowances show. Count pages before upload, so a 900-page record is an expected cost rather than a surprise. Whether files should go over an API at all, or by scheduled file drop, is the subject of the guide to REST API vs SFTP for healthcare document intake.

How do I track batch progress across many document jobs?

Track batch progress with a manifest you own, one row per document, holding the file path, idempotency key, job ID, current state and last update time. Results from webhooks or polling update the rows. Progress is then a count by state, and the batch is done when every row is in a final state.

A manifest for a fictional 400-file batch might summarize like this midway:

The total row is the check that matters. If the counts do not add up to the files in the folder, a document was lost between listing and upload. Run the comparison both ways: every file has a row, and every job the API reports for this batch has a row. A job with no row is a duplicate created outside the manifest.

Completion events are the fastest way to move rows forward. The job states those events report, and how a receiver should handle them, are covered in the section on job states in async document processing APIs. Keep a slow poll as well, for rows that sit in uploaded longer than expected.

When rows reach completed, the next step is writing values into the destination system, which needs its own matching rules. The guide to mapping extracted fields to an EHR covers that step. Write each value once, keyed by job ID, so a repeated result does not create a second entry.

Batch processing failure modes: partial batches and poison files

Batch processing fails in two common ways: partial batches, where some documents succeed and others do not, and poison files, which fail every time they are retried. Plan for both by treating each document as its own unit of success, capping retries, and moving repeat failures to a quarantine list a person reviews.

Partial batches. A batch of 400 that ends with 394 completed and 6 failed is normal, not a crisis. The mistake is to rerun the whole batch to fix six files. With idempotency keys and a manifest, a rerun touches only rows that are not final. Without them, rerunning creates 394 duplicate jobs and duplicate charges.

Poison files. A corrupt TIFF, a PDF whose pages are blank, or a file that crashes the reader will fail on every attempt. A retry loop without a cap spins on it forever and may starve good files of capacity. Cap attempts per file, record the last error, and stop.

Uploader crashes. If the process dies midway, the manifest shows where it stopped. Files marked pending with a stored key are retried with that same key, so any upload that actually reached the server returns its original job instead of a new one.

Expired keys. The idempotency draft lets servers expire keys on a published schedule. A rerun days later may hit an expired key and start a new job. Check the manifest for a job ID before retrying an old row, and fetch that job's state instead of re-uploading.

Duplicate files with different names. The same EOB scanned twice under two names has two paths and, if keys come from the path, two keys. Deriving keys from the file's bytes catches exact copies. Rescans of the same paper differ byte for byte, so near-duplicates still need a person or a matching rule later.

The reconciliation report closes the loop. For each batch it should list skipped, rejected, failed and quarantined files with reasons, totals by state, and any job not found in the manifest. A billing lead should be able to read it and know exactly which pieces of paper still need attention.