Clinical Notes Extraction From SOAP and Handwritten Notes

Clinical notes extraction works best when you pull a short list of dated, labelled facts from each note and leave the narrative alone. Use the note's sections to find those facts: header details, vital signs, medication and allergy lines, and the problems listed in the assessment. Every extracted value should point back to its line on the page.

What SOAP note structure gives clinical notes extraction

A SOAP note splits a visit into four parts: Subjective, Objective, Assessment and Plan. For clinical notes extraction, that structure works like a map. It tells software where a type of fact usually sits, so a blood pressure is searched for in one part and a follow-up date in another, instead of anywhere on the page.

The HL7 Consolidated CDA standard, which defines how many electronic clinical documents are built, gives each of these parts its own section definition. Those definitions are a fair guide to what each part of a paper or scanned note tends to hold too.

• Subjective. The HL7 C-CDA Subjective Section "describes in a narrative format the patient's current condition and/or interval changes as reported by the patient" or by a guardian or another informant. It is mostly free text in the patient's own account.
• Objective. The HL7 C-CDA Objective Section holds data "gathered through tests, measures, or observations that produce a quantified or categorized result." Vital signs, measurements and test values usually live here.
• Assessment. The HL7 C-CDA Assessment Section represents "the clinician's conclusions and working assumptions," and notes that it "may be a list of specific disease entities or a narrative block."
• Plan. The HL7 C-CDA Plan of Treatment Section captures "planned, pending, active, or completed orders, interventions, encounters, services, and procedures."

The same standard allows the assessment and plan to be written as one block. The HL7 C-CDA Progress Note requires either a combined Assessment and Plan Section or both separate sections, and lists Subjective, Objective, Vital Signs, Allergies and Medications among the sections a note may also carry. An extraction setup should expect both layouts and should not assume the four parts always appear in the same order.

Progress notes are one of several note types. The USCDI Clinical Notes data class from ONC lists Consultation Note, Discharge Summary Note, History & Physical, Procedure Note and Progress Note, among others, and describes the class as "narrative patient data relevant to the context identified by note types." Knowing which note type you hold tells you which sections to look for. The structure of a history and physical, which carries more sections than a progress note, is covered in the guide to how an H&P document is laid out.

A fictional SOAP note, line by line

The note below is invented for this post. The patient, the clinic and every value are examples only.

Read this way, eight lines produce about a dozen fields. Most of them come from the header, the objective part and the numbered assessment list. The subjective line produces nothing on purpose.

Clinical note fields worth extracting, and which to skip

Clinical note fields worth extracting are the ones a downstream job reads by name: dates, authors, vital signs, medications, allergies, listed problems and follow-up intervals. The narrative that explains the visit is worth keeping as text, linked to the note, but it is a poor source of fields. Pulling facts from it invites misreadings.

A useful test is to ask who will read each field and what they will do with it. A billing team needs the encounter date, the rendering provider and the problems listed in the assessment so it can match the note to a claim. A records team indexing charts needs the patient identifiers, the note type and the date. A quality or registry project needs whatever its data dictionary names, and nothing else. A field nobody reads is a field somebody has to check for no reason.

Fields that usually repay the effort:

• Header fields: patient name, date of birth, record number, encounter date, note type, author and signature date.
• Vital signs: each measurement with its unit and, where the note gives one, the time it was taken.
• Medication lines: drug name, strength and frequency exactly as written.
• Allergy lines: the substance and any reaction the note records.
• Problem entries: each numbered or bulleted item in the assessment, copied as written.
• Plan items: follow-up interval, referrals named, and documents issued.

Fields that usually do not repay it are the subjective account, the reasoning between the findings and the conclusions, and long review-of-systems paragraphs. These are better kept whole. Breaking them into fields makes it easy to lose the words that change their meaning, such as "denies" or "no history of." How software handles that kind of wording is covered in the guide to negation and context in unstructured clinical text.

Extraction records what the note says. It does not decide whether a value is normal, whether a problem is resolved, or what should happen next. Those are clinical judgments and stay with clinicians. When you need a readable overview of a long chart instead of a set of fields, the trade-offs are laid out in medical record summarization vs field extraction.

Can OCR read handwritten clinical notes?

OCR can read some handwritten clinical notes, but the results vary far more than with typed notes. Neat block capitals in a labelled box often read well. Cursive progress notes, abbreviations, crossed-out words and writing that runs into the margins often do not. Every handwritten field needs a confidence score and a path to a person.

Handwriting removes the clues that make typed notes easy. Section labels may be missing or abbreviated to a single letter. Line breaks may not match the logic of the note. Two numbers written close together, such as a pulse and a weight, can merge into one. A decimal point or a stray mark can change a medication strength. The software has less to anchor on, so it guesses more often.

The practical answer is to expect handwritten notes to send more fields to review, and to plan staffing around that. Set the review threshold by document type, so handwritten progress notes are held to a stricter bar than typed visit summaries. Track which fields reviewers correct most often. If one field, such as the medication strength, is corrected again and again, it may be cheaper to have staff key that field every time than to check it every time.

Scan quality matters as much as the writing. Faint pencil, carbon copies and faxed pages lose strokes before OCR ever sees them. The medical OCR basics category covers scanning and image quality. The detailed guide to what handwriting recognition gets right on medical forms covers checkboxes, signatures and the fields a person should always check.

Does copy-forward text in clinical notes mislead extraction?

Copy-forward text can mislead extraction badly. When a clinician copies a paragraph from an earlier note into a new one, the old facts arrive with a new date. Extraction software reads them as current. A medication stopped months ago, or a vital sign from a previous visit, can then appear as a field from this encounter.

CMS treats this as a documentation problem in its own right. The CMS Electronic Health Records Provider fact sheet describes cloning as "copying and pasting previously recorded information from a prior note into a new note" and says the record must show "the differences and the needs of the patient for each visit or encounter." The fact sheet is about provider documentation, but the effect on extraction follows from it: copied text looks exactly like new text.

There are signals extraction can use:

• Repeated blocks. Compare each note with the previous one for the same patient. Paragraphs that match word for word are likely carried forward.
• Dates inside the text. A line such as "BP 130/84 on 06/10" inside a note dated 09/02 points to an earlier reading. Keep the date in the text, not the note's date.
• Stale lists. A medication list that never changes across many visits may be copied rather than reviewed.

None of these signals proves a value is wrong. They are reasons to send a field to a person, and the person reads the note and decides. The reviewer should be able to see both notes side by side, so the comparison is quick.

Copied text also inflates counts. If you extract a problem list from every note in a chart and add them up, one problem copied into twenty notes looks like twenty mentions. De-duplicate by the problem and its first date, not by each appearance.

Clinical notes with no section headings

Clinical notes with no section headings, such as dictations, free-form letters and quick handwritten entries, have to be split before fields can be found. The software, or a person, first marks where the vital signs, medications and plan sit in the text. Fields are then extracted from those marked spans, each still linked to its place on the page.

Dictated notes often keep a spoken structure even without written headings. Phrases such as "on exam," "vitals today were" and "plan is to" act as informal labels. A setup that looks for these phrases can recover most of the structure. A free-form letter from a consultant may have no such cues and read as a single block.

For notes like these, narrow the job. Pick the few fields that matter most, such as the encounter date, the author and any medication or allergy line, and extract only those. Leave the rest as linked text. A short, reliable list is worth more than a long list that every reviewer has to rebuild.

Unlabelled notes are also where confidence scores earn their place. A value found next to a clear label, such as "BP," deserves more trust than a number found in the middle of a sentence. Scores should drop when the software has to infer what a number means, and those fields should wait for review.

Schema Care reads medical records into fields such as encounter dates, problem lists, allergies and vitals, links each value to its spot on the page, and sends low-confidence fields to a person, as the list of supported documents shows. For how those fields compare with a prose overview of the same chart, see the guide to medical record summarization with source links.