A lot of the work we do at Jellyfish Technologies is for clients whose business runs on documents. Many of them are in insurance and financial services. The documents arrive as PDFs, some clean, many scanned, and almost none of them share a layout.
On one of these projects we built a pipeline in three steps. EasyOCR and AWS Textract pulled the text off the page. An LLM extracted the entities from that text. Then we stored the entities and the relationships between them in a graph, using GraphRAG on Neo4j. This note is about why each step is there, and what I would change if I built it again today.
What breaks with OCR alone
OCR gives you characters. It does not tell you what they mean. On a form with a fixed layout that is often enough: you know the policy number sits in the top right corner, so you read the top right corner.
Our documents did not work like that. A field could sit in a different place on every document, depending on who produced it. Once the layout varies, a block of recognised text is just a long string. Plain text search finds words, but it cannot tell which value belongs to which entity when the two land in different spots on each page.
That is the gap the next two steps fill.
What the LLM step does
The LLM reads the OCR output and pulls out the things the business cares about: the entities, and the values attached to them. Its job is to turn a page of text into named, structured items that a program can work with.
This is where the layout problem mostly goes away. The model does not need the field to be in a fixed position. It needs enough text around the value to understand what the value is.
It does not remove the need to check. A model that reads well can still read the wrong thing with confidence, so the output of this step is something to verify, not something to trust because it came back in a clean format.
Why a graph helps when one fact spans many documents
A single document rarely tells the whole story. The same person, policy or account turns up across several files, and each file adds a piece. If you keep each document's extraction on its own, you end up with many small, disconnected records.
GraphRAG on Neo4j let us store the entities as nodes and the relationships between them as edges. With the relationships stored, we could find an item wherever it sat, and follow it from one document to the next. A question about one entity becomes a walk through the graph instead of a search through many pages.
The graph is the step I would keep if I built this again. It is also the step I would push further. Today I would use it to cross-check values across documents: if two files disagree about the same entity, that disagreement is worth surfacing, not averaging away.
Keeping answers traceable to a page
An answer from a document system is only useful if someone can check it. In insurance and finance that is not optional. A reviewer needs to see where a value came from before they act on it.
If I rebuilt the pipeline now, I would change two things to make that easier.
- Tie each entity to its spot on the page. Every extracted value should carry the document and the location it came from, so an answer can point straight back to the source.
- Read the text layer before running OCR. Many system-generated PDFs, such as carrier-generated declaration pages, already have a text layer with word positions. Use that plus the page image, and keep OCR for true scans.
I would also look again at the OCR step itself. Small models trained only to read pages have improved fast, and for scanned documents they are now a reasonable first thing to try.
Where people stay in the loop
I think of the pipeline as a way to put the right page in front of a person, not a way to remove the person. The places I would keep a human are the ones where a wrong value costs the most.
- When the LLM extraction and the source page disagree, or the value cannot be tied to a location.
- When the graph shows two documents giving different values for the same entity.
- Before any decision that goes back to a customer.
With the location stored on every entity, these checks are quick. The reviewer is not hunting through a long file. They are looking at one value and the exact spot it came from.
If you are scoping a document project, start with a pile of your messiest real samples and ask one question: can every value we extract be traced back to a place on a page? If the answer is no, fix that before you choose a model.