Document Extraction with Vision-Language Models: A Practical Guide

Document extraction with vision-language models means sending a page image directly to a multimodal model and asking it to return structured fields, instead of running OCR then pattern-matching the text. This matters because invoices, contracts, ID cards, and forms are laid out visually: a field's meaning often depends on its position, the table it sits in, or a checkbox next to it, and plain OCR throws that layout information away. A VLM sees the page the way a person does, so it can read a "Total Due" figure out of a table even when the OCR text stream has scrambled the column order.

This guide walks through why VLMs outperform classic OCR pipelines for this task, how to build a working extraction pipeline in Python, how to get reliable structured output, how to handle multi-page PDFs and tables, and how to measure accuracy so you know when the system is actually production-ready.

Why Vision-Language Models Beat OCR Pipelines for Document Extraction

A traditional document extraction pipeline looks like this: run an OCR engine (Tesseract, an OCR API, or a proprietary engine) to get text and bounding boxes, run a layout model to group text into fields, then run rules or a small classifier to map fields to a schema. Every stage can fail independently, and errors compound. A slightly rotated scan throws off OCR confidence, which throws off layout grouping, which throws off field mapping. Debugging is miserable because you have to figure out which of three or four stages introduced the error.

Vision-language models collapse this into one step. You send the image (or a PDF page rendered as an image) and a prompt describing the schema you want, and the model returns the extracted fields directly. The model reads pixels, not a flattened text stream, so it keeps the spatial relationships: it knows that the number under the "Qty" header in row three belongs to the item described in that row, even if the underlying OCR text order would have jumbled it.

The practical wins for document extraction with VLMs:

• Layout robustness. Multi-column layouts, tables with merged cells, and forms with checkboxes are handled natively because the model is reasoning over the visual layout, not a linearized text dump.
• Handwriting and low-quality scans. Modern VLMs handle messy handwriting and skewed or low-resolution scans noticeably better than classic OCR engines, which were tuned for clean printed text.
• Zero-shot schema changes. Swapping to a new document type (a new invoice template, a new government form) is a prompt change, not a retrain. A rules-based OCR pipeline usually needs new heuristics per template.
• Fewer moving parts. One API call replaces an OCR engine, a layout parser, and a rules engine, which means fewer places for silent failures to hide.

The tradeoffs are real too: VLM calls are slower and more expensive per page than a tuned OCR pipeline, and a VLM can hallucinate a plausible-looking value instead of admitting a field wasn't legible. The rest of this guide covers how to mitigate both.

How Document Extraction with VLMs Works

At a mechanical level, the pipeline is:

1. Convert each document page into an image (PDFs are rendered page-by-page; scanned images are used as-is).
2. Send the image to a multimodal model along with a prompt or tool schema that describes the fields to extract.
3. Force the model to respond in a structured format (JSON via a tool call or a strict schema), not free text.
4. Validate the returned structure against your schema and flag anything that fails validation or looks suspicious (empty required fields, values outside expected ranges).
5. Optionally reconcile results across pages for documents that span more than one page.

The key design decision is step 3: how you constrain the output. Free-text prompting ("extract the invoice number and total") works for demos but breaks in production because the model's response format drifts. Use structured output enforcement (tool use / function calling with a JSON schema) so the model is constrained to emit a shape you can parse without a second LLM call to "clean up" the response.

Setting Up: Rendering Documents as Images

Most source documents arrive as PDFs, so the first real step is rendering pages to images at a resolution the VLM can read clearly. Use PyMuPDF (the fitz module) because it is fast and does not require an external Poppler install like pdf2image does.

pip install pymupdf pillow anthropic

import fitz  # PyMuPDF
from PIL import Image
import io

def pdf_to_images(pdf_path, dpi=200):
    doc = fitz.open(pdf_path)
    images = []
    zoom = dpi / 72  # PDF points are 72 per inch
    matrix = fitz.Matrix(zoom, zoom)
    for page in doc:
        pix = page.get_pixmap(matrix=matrix)
        img_bytes = pix.tobytes("png")
        images.append(Image.open(io.BytesIO(img_bytes)))
    doc.close()
    return images

A DPI of 150 to 200 is usually enough for typed documents; push to 250-300 for small print or dense tables. Going much higher than that mostly adds latency and cost without improving accuracy, since the model downsamples internally anyway.

Building the Extraction Call with Structured Output

Define the schema you want as a JSON schema and pass it as a tool definition. This is the single most important step for making document extraction with VLMs reliable in production: never ask the model to "return JSON" in free text and parse it with a regex. Use the model provider's native structured output or tool-calling mechanism so the response is guaranteed to match your shape.

Here is a working example using the Anthropic Python SDK with Claude's vision input and tool use to extract invoice fields:

import base64
import io
import json
from anthropic import Anthropic

client = Anthropic()

INVOICE_SCHEMA = {
    "name": "record_invoice",
    "description": "Record the extracted fields from an invoice image.",
    "input_schema": {
        "type": "object",
        "properties": {
            "invoice_number": {"type": "string"},
            "invoice_date": {"type": "string", "description": "ISO 8601 date"},
            "vendor_name": {"type": "string"},
            "total_amount": {"type": "number"},
            "currency": {"type": "string"},
            "line_items": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "description": {"type": "string"},
                        "quantity": {"type": "number"},
                        "unit_price": {"type": "number"},
                        "line_total": {"type": "number"},
                    },
                    "required": ["description", "line_total"],
                },
            },
            "field_confidence": {
                "type": "object",
                "description": "Per-field confidence: high, medium, or low",
                "additionalProperties": {"type": "string", "enum": ["high", "medium", "low"]},
            },
        },
        "required": ["invoice_number", "total_amount", "line_items"],
    },
}

def image_to_b64(img):
    buf = io.BytesIO()
    img.save(buf, format="PNG")
    return base64.standard_b64encode(buf.getvalue()).decode("utf-8")

def extract_invoice(image):
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=2048,
        tools=[INVOICE_SCHEMA],
        tool_choice={"type": "tool", "name": "record_invoice"},
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/png",
                            "data": image_to_b64(image),
                        },
                    },
                    {
                        "type": "text",
                        "text": (
                            "Extract the invoice fields from this document image. "
                            "If a field is not present or illegible, omit it rather than guessing. "
                            "Mark your confidence for each extracted field."
                        ),
                    },
                ],
            }
        ],
    )
    for block in response.content:
        if block.type == "tool_use":
            return block.input
    raise ValueError("Model did not return a tool call")

Two details in this prompt matter more than they look. First, "omit it rather than guessing" pushes the model away from hallucinating a plausible-but-wrong value when a field is blurry or missing, which is the single biggest source of silent errors in document extraction with VLMs. Second, the field_confidence map gives you a signal to route low-confidence extractions to human review without needing a second model call.

Processing Multi-Page Documents

For documents longer than a page or two, extract each page independently and then merge, rather than stuffing all pages into one call. This keeps latency predictable and makes it easy to parallelize.

import concurrent.futures

def extract_document(pdf_path, extract_fn, max_workers=4):
    images = pdf_to_images(pdf_path)
    results = [None] * len(images)

    def worker(i, img):
        results[i] = extract_fn(img)

    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as pool:
        futures = [pool.submit(worker, i, img) for i, img in enumerate(images)]
        for f in futures:
            f.result()

    return results

For a form or contract where a single logical value spans two pages (a signature block referencing terms on page one, for example), a cheap fix is to always include the previous page's extracted summary as extra context text in the next call, rather than sending both images together:

def extract_with_context(image, previous_summary, extract_fn):
    context_note = f"Context from prior page: {previous_summary}" if previous_summary else ""
    # append context_note to the text block passed to extract_fn
    ...

This keeps each call cheap (one image) while still letting later pages resolve references to earlier ones.

Handling Tables and Forms

Tables are where OCR pipelines fail most often, and where VLMs earn their keep. The trick is to be explicit in the schema about row structure rather than asking for a flat list of numbers. In the invoice schema above, line_items is an array of structured objects, which forces the model to keep each row's fields together instead of returning parallel arrays that can misalign.

For forms with checkboxes or radio buttons, describe the expected states explicitly in the schema description rather than leaving it to free interpretation:

"marital_status": {
    "type": "string",
    "enum": ["single", "married", "divorced", "widowed", "not_specified"],
    "description": "Which checkbox is marked. Use not_specified if none are checked or the image is unclear.",
}

Constraining to an enum does two things: it prevents the model from inventing a value outside your accepted set, and it gives you a deterministic fallback (not_specified) instead of an empty string or null that you have to special-case downstream.

For dense tables spanning many columns, splitting the extraction into two calls, one for header/metadata fields and one dedicated purely to the table rows, tends to reduce cross-field confusion versus asking for everything in a single schema.

Evaluating Accuracy: Field-Level Metrics

Aggregate "accuracy" numbers hide where a document extraction with VLMs pipeline actually breaks. Build a small evaluation harness against a labeled sample (50 to 200 documents is enough to start) and score per field, not per document.

def score_extraction(predicted, ground_truth, numeric_tolerance=0.01):
    results = {}
    for field, true_value in ground_truth.items():
        pred_value = predicted.get(field)
        if pred_value is None:
            results[field] = "missing"
        elif isinstance(true_value, (int, float)):
            try:
                match = abs(float(pred_value) - float(true_value)) <= numeric_tolerance
            except (TypeError, ValueError):
                match = False
            results[field] = "correct" if match else "wrong"
        else:
            match = str(pred_value).strip().lower() == str(true_value).strip().lower()
            results[field] = "correct" if match else "wrong"
    return results

def summarize(all_results):
    from collections import defaultdict
    tally = defaultdict(lambda: {"correct": 0, "wrong": 0, "missing": 0})
    for doc_results in all_results:
        for field, status in doc_results.items():
            tally[field][status] += 1
    for field, counts in tally.items():
        total = sum(counts.values())
        print(f"{field}: {counts['correct']}/{total} correct, "
              f"{counts['wrong']} wrong, {counts['missing']} missing")

Run this after every prompt or schema change. In practice, a handful of fields (dates in ambiguous formats, currency symbols, line-item counts on dense tables) account for most of the errors, and once you can see that breakdown you can fix it directly, either by tightening the schema description for that field or by adding a targeted example in the prompt.

Common Failure Modes and How to Fix Them

Date format drift. Models will happily return 03/04/2026 without telling you if that's March 4th or April 3rd. Fix it by requiring ISO 8601 in the schema description and, where the source document's locale is known, stating it explicitly in the prompt ("dates are in DD/MM/YYYY format on this document type").

Silent hallucination on blurry fields. The single-page prompt instruction to omit rather than guess (shown above) helps, but the more reliable fix is the confidence map: route anything marked "low" to human review instead of trusting the raw value.

Currency and unit confusion. If a document mixes currencies or units, add a required currency or unit field next to every numeric amount rather than assuming one global currency for the document.

Table row misalignment on wide tables. If a table has more than eight or nine columns, consider cropping the image to just the table region (using the page's known layout, or a first-pass bounding-box call) before the extraction call. A tighter, larger view of just the table meaningfully improves row alignment versus asking the model to also parse header and footer content simultaneously.

Multi-page totals disagreeing with per-page sums. For financial documents, add a reconciliation step after extraction: sum the line items and compare against the extracted total. A mismatch is a strong, cheap signal to flag for review without needing another model call.

Production Considerations: Cost, Latency, Batching

Every page is one model call, so cost and latency scale linearly with page count unless you batch. Three practical levers:

• Concurrency, not sequential loops. The ThreadPoolExecutor pattern above keeps wall-clock time down for multi-page documents; tune max_workers to your provider's rate limits.
• Cache the rendered images, not just results. Re-running extraction after a prompt tweak is common during development; keep the rendered PNGs on disk keyed by document hash so you're not re-rendering PDFs on every iteration.
• Route by confidence, not by document type. Rather than sending every document through the most expensive model available, run a first pass with a lighter or faster model and only escalate documents whose confidence scores come back low to a stronger model. This keeps average cost down while still catching the hard cases.

Also budget for validation, not just extraction. A JSON schema check that rejects malformed tool calls, plus the reconciliation checks described above, catches a meaningful share of errors before they ever reach a downstream system, and it is far cheaper than debugging a bad record after it has already been written to a database.

FAQ

Do I still need OCR at all if I'm using a VLM? Not for the extraction itself, but OCR text can still be useful as a secondary signal, for example to cross-check a VLM-extracted field against an OCR'd value and flag disagreements for review. Some teams keep a lightweight OCR pass purely as a sanity check, not as the primary extraction path.

How do I handle documents with more pages than fit in one request? Extract page by page as shown in the multi-page section above, then merge results programmatically. Do not try to fit a 30-page contract into a single call; per-page extraction with a reconciliation step afterward is both cheaper and more reliable.

What's the best way to handle low-quality scans or photos of documents? Increase the render DPI if you control the source, and add an explicit instruction in the prompt to mark low-confidence fields rather than guess. If scans are consistently poor, a pre-processing deskew and contrast-normalization step (OpenCV's cv2.warpAffine for deskew, simple histogram equalization for contrast) before sending the image to the model measurably helps.

Can I extract free-form text sections, like clauses in a contract, the same way? Yes, but use a schema field with type: string and a generous max length rather than trying to force free text into rigid sub-fields. For clause-level extraction, it's often better to have the model return an array of {clause_title, clause_text} objects rather than trying to pre-define every possible clause type in the schema.

How much manual review should I budget for after launch? Start by routing every low-confidence field to review, then track how often reviewers actually change the value versus just confirming it. If confirmation rate for a given field type stays high over a few hundred documents, you can loosen the review threshold for that field; if a field keeps getting corrected, that's a signal to revisit the schema description or prompt for that field specifically, not to widen your tolerance.

Is this approach viable for structured forms with checkboxes and signatures, not just invoices? Yes. The enum-constrained schema pattern shown in the forms section handles checkboxes well. Signatures are harder to validate structurally; treat "is this field signed" as a boolean the model reports (based on whether it visually detects a mark in the signature area) rather than trying to verify signature authenticity, which is a different problem entirely.