RAG

RAG for PDFs: Handling Tables, Images and Multi-Column Layouts

Pramod Dutta · May 11, 2026 · 15 min read

Every RAG tutorial demos beautifully on a clean, single-column blog post converted to PDF. Then someone drops a 40-page insurance policy with three-column tables, a scanned invoice, and a research paper with two-column text wrapping around figures, and the whole pipeline falls apart. If you've shipped even one production RAG system, you already know: the retrieval math is the easy part. The hard part is getting clean, structured text out of a PDF in the first place. Building reliable rag for pdfs means treating PDF parsing as its own engineering problem, not an afterthought you bolt on before the "real" RAG work begins.

This article walks through the failure modes that show up specifically with PDFs — tables, images, multi-column layouts, headers/footers, and scanned pages — and the concrete techniques that fix them. No magic libraries, no fabricated benchmark numbers, just what actually works when you're debugging a retrieval pipeline that keeps returning garbage chunks.

Why PDFs Break Naive RAG Pipelines

A PDF is not a text file. It's a page-description format — a set of instructions for where to draw glyphs, lines, and images on a canvas. There is no inherent concept of "paragraph," "table," or "reading order" baked into the format. When you call a naive pdf.extract_text() function, you're asking the library to guess reading order from raw coordinate data, and that guess breaks constantly.

Here's what naive extraction typically produces from a two-column PDF:

import pypdf

reader = pypdf.PdfReader("research_paper.pdf")
page = reader.pages[0]
text = page.extract_text()
print(text)
# Output interleaves left and right columns line by line:
# "Abstract: This paper proposes a novel method Results show a 12%"
# "for extracting structured data from unstructured improvement over"
# "documents using layout-aware segmentation. baseline methods."

Notice what happened: the extractor read left-to-right across the page, not top-to-bottom within each column. The abstract text and the results text are now interleaved into a single incoherent paragraph. Feed that into a chunker and embed it, and you get a vector that represents neither the abstract nor the results — it represents noise. No amount of clever retrieval logic downstream fixes an input that's already scrambled.

This is the core insight for rag for pdfs: garbage layout extraction produces garbage embeddings, and garbage embeddings produce garbage retrieval, no matter how good your reranker or LLM is. Fix the input first.

Multi-Column Layouts: Reconstructing Reading Order

The fix for multi-column documents is to stop treating the page as a stream of text and start treating it as a set of positioned blocks that need to be reordered.

Most modern PDF libraries expose bounding boxes for text blocks. pdfplumber and PyMuPDF (imported as fitz) both give you (x0, y0, x1, y1) coordinates per word or line. You can use those coordinates to detect columns and sort blocks accordingly.

A practical approach:

Extract all text blocks with their bounding boxes.
Cluster blocks by their horizontal (x) position to detect column boundaries.
Within each column cluster, sort blocks top-to-bottom by y-coordinate.
Concatenate columns left-to-right.

import fitz  # PyMuPDF

def extract_columns(pdf_path, page_num):
    doc = fitz.open(pdf_path)
    page = doc[page_num]
    blocks = page.get_text("blocks")  # (x0, y0, x1, y1, text, block_no, block_type)

    # crude column split: anything left of page midpoint is column 1
    page_width = page.rect.width
    midpoint = page_width / 2

    left_col = [b for b in blocks if b[0] < midpoint]
    right_col = [b for b in blocks if b[0] >= midpoint]

    # sort each column top-to-bottom
    left_col.sort(key=lambda b: b[1])
    right_col.sort(key=lambda b: b[1])

    ordered_text = "\n".join(b[4] for b in left_col) + "\n" + \
                   "\n".join(b[4] for b in right_col)
    return ordered_text

This midpoint heuristic works for clean two-column layouts but breaks on anything with three columns, sidebars, or a figure that spans both columns. For messier documents, cluster on x-coordinate using something like k-means on block centroids, or fall back to a layout-detection model (more on this below) that classifies regions before you ever try to order them.

The lesson that generalizes: never trust the raw extraction order from a PDF library on anything but pure single-column text. Always inspect a sample of pages visually before you commit to a parsing strategy for a new document type.

Tables: The Hardest Problem in PDF RAG

Tables are where most RAG-for-PDF pipelines quietly fail. A table encodes meaning through 2D spatial relationships — this cell's value belongs to this row header and this column header — and flattening that into linear text destroys the relationship unless you're deliberate about it.

Consider a pricing table:

Plan       | Monthly | Annual  | Seats
Starter    | $29     | $290    | 5
Pro        | $99     | $990    | 20
Enterprise | Custom  | Custom  | Unlimited

Naive text extraction often produces something like Plan Monthly Annual Seats Starter $29 $290 5 Pro $99 $990 20 Enterprise Custom Custom Unlimited. If this whole blob gets chunked and split mid-table (which happens constantly with fixed-size chunking), a query like "what's the annual price for the Pro plan" retrieves a chunk with numbers in it but no way for the LLM to know which number maps to which plan.

There are three approaches that actually work, in increasing order of effort and reliability:

1. Table-aware extraction libraries. pdfplumber has a page.extract_tables() method that uses line detection to reconstruct table structure into lists of rows. camelot-py does something similar and works well on tables with visible ruling lines.

import pdfplumber

with pdfplumber.open("pricing.pdf") as pdf:
    page = pdf.pages[0]
    tables = page.extract_tables()
    for table in tables:
        for row in table:
            print(row)
    # ['Plan', 'Monthly', 'Annual', 'Seats']
    # ['Starter', '$29', '$290', '5']
    # ['Pro', '$99', '$990', '20']

2. Convert tables to Markdown before chunking, not after. Once you have structured rows, render them as a Markdown table and keep that table as one atomic chunk (or a small number of row-grouped chunks) instead of letting your text splitter cut through it.

def rows_to_markdown(headers, rows):
    md = "| " + " | ".join(headers) + " |\n"
    md += "|" + "|".join(["---"] * len(headers)) + "|\n"
    for row in rows:
        md += "| " + " | ".join(str(c) for c in row) + " |\n"
    return md

Markdown tables preserve row/column relationships in a format LLMs are heavily trained on, so retrieval and generation both benefit — the model can reason about "Pro plan, Annual column" even from a text chunk, because the structure is explicit rather than implied by whitespace.

3. Row-level chunking with repeated headers. For very large tables (hundreds of rows), don't embed the whole table as one chunk — you'll blow past context limits and dilute the embedding. Instead, chunk by row groups (e.g., 10-20 rows per chunk) and repeat the header row in every chunk. This means each chunk is self-contained and a query about a specific row can match it directly, without needing surrounding context to interpret column meaning.

def chunk_table_rows(headers, rows, rows_per_chunk=15):
    chunks = []
    for i in range(0, len(rows), rows_per_chunk):
        group = rows[i:i + rows_per_chunk]
        chunks.append(rows_to_markdown(headers, group))
    return chunks

Whichever approach you pick, the non-negotiable rule is: never let your recursive character/token splitter run over a table without table awareness. It will cut mid-row, and half your rows will lose their headers.

One more failure mode worth calling out: merged cells and multi-row headers. A lot of real-world tables (financial statements, lab reports, government forms) have a header that spans two rows, or a first column that's merged across several rows to group related items. Naive extraction tools flatten these into empty strings or None values in the wrong positions, which then produces a Markdown table with blank headers. When you hit this, it's usually faster to detect the pattern (a run of empty cells directly below a populated one) and forward-fill the merged value programmatically than to hunt for a library flag that handles it automatically:

def forward_fill_column(rows, col_index):
    last_value = None
    for row in rows:
        if row[col_index] in (None, ""):
            row[col_index] = last_value
        else:
            last_value = row[col_index]
    return rows

This single function fixes a surprising number of "why does the retrieval answer say None for the category" bugs in table-heavy documents.

Images and Scanned Pages: When You Need OCR

Some PDFs aren't really "documents" at all — they're scanned images wearing a PDF extension. Text extraction on these returns nothing, because there's no text layer, just a raster image per page. You'll recognize this immediately: extract_text() returns an empty string or garbled encoding artifacts.

The fix is OCR (Optical Character Recognition), and the two realistic options are:

Tesseract (open source, via pytesseract) — free, decent accuracy on clean scans, struggles with skewed pages, low-resolution scans, and unusual fonts.
Vision-capable LLMs (sending page images directly to a multimodal model) — much more robust to messy scans, handles handwriting better, and can be prompted to output structured Markdown directly, including tables, at the cost of higher latency and per-page API cost.

A basic Tesseract pipeline for a scanned page:

import fitz
import pytesseract
from PIL import Image
import io

def ocr_page(pdf_path, page_num):
    doc = fitz.open(pdf_path)
    page = doc[page_num]
    pix = page.get_pixmap(dpi=300)  # higher DPI improves OCR accuracy
    img = Image.open(io.BytesIO(pix.tobytes("png")))
    text = pytesseract.image_to_string(img)
    return text

Two details matter enormously here. First, DPI: rendering at the default 72-96 DPI produces blurry raster images and OCR accuracy tanks. Render at 300 DPI minimum. Second, always detect whether OCR is even needed before running it — check if extract_text() on a page returns a near-empty string, and only fall back to OCR for those pages. Running OCR on every page of a mixed document wastes time and can actually introduce errors on pages that already had a clean text layer.

For diagrams, charts, and photos embedded in a PDF (not full scanned pages, but figures within a text-based document), a growing practice is to extract the image, send it to a vision-capable LLM with a prompt like "describe this chart and any data trends visible," and store that description as searchable text tied to the surrounding page context. This means a query like "what does the Q3 revenue chart show" can actually match something, instead of retrieval treating every chart as an invisible blank spot in the document.

def extract_images(pdf_path, page_num):
    doc = fitz.open(pdf_path)
    page = doc[page_num]
    images = page.get_images(full=True)
    extracted = []
    for img in images:
        xref = img[0]
        base_image = doc.extract_image(xref)
        extracted.append(base_image["image"])  # raw bytes, feed to vision model
    return extracted

Headers, Footers, and Page-Level Noise

A subtler failure mode: every page of a 200-page compliance document repeats "CONFIDENTIAL — Internal Use Only" in the header and "Page X of 200" in the footer. If you don't strip these, they get embedded into nearly every chunk, adding noise that dilutes the semantic signal of the actual content and, worse, can cause chunks from completely unrelated sections to look artificially similar because they share the same boilerplate text.

Detection is straightforward: extract text with bounding boxes, and any line that repeats near-identically at the same y-coordinate range across a large fraction of pages is almost certainly a header or footer.

from collections import Counter

def detect_repeating_lines(pdf_path, y_threshold=50):
    doc = fitz.open(pdf_path)
    top_lines = Counter()
    bottom_lines = Counter()

    for page in doc:
        blocks = page.get_text("blocks")
        page_height = page.rect.height
        for b in blocks:
            y0, text = b[1], b[4].strip()
            if not text:
                continue
            if y0 < y_threshold:
                top_lines[text] += 1
            elif y0 > page_height - y_threshold:
                bottom_lines[text] += 1

    total_pages = len(doc)
    headers = [t for t, count in top_lines.items() if count > total_pages * 0.6]
    footers = [t for t, count in bottom_lines.items() if count > total_pages * 0.6]
    return headers, footers

Anything appearing in more than roughly 60% of pages in that top/bottom band is a strong candidate for boilerplate and should be stripped before chunking. Keep the threshold configurable — some legal documents legitimately repeat section titles in headers that you do want to keep as navigational context.

Chunking Strategy Once Text Is Clean

Once you've solved layout order, tables, and boilerplate, chunking still needs to respect document structure rather than blindly splitting every N tokens. A few practices that hold up in production:

Chunk by semantic unit first, size second. Split on headings, sections, and paragraph boundaries before falling back to a token-count splitter within an oversized section.
Keep tables as atomic units (or row-grouped units as covered above) — never let a generic splitter run through them.
Add page-number and section metadata to every chunk. When a user asks "what does page 14 say about liability," you want that filterable, not just semantically searchable.
Overlap sparingly. A 10-15% overlap between adjacent chunks helps preserve context across boundaries, but heavy overlap (50%+) mostly just triples your storage and retrieval noise for marginal benefit.

def chunk_with_metadata(sections, max_tokens=500):
    chunks = []
    for section in sections:
        text = section["text"]
        page = section["page"]
        heading = section["heading"]
        if len(text.split()) <= max_tokens:
            chunks.append({"text": text, "page": page, "heading": heading})
        else:
            # fall back to paragraph-level splitting within the section
            paragraphs = text.split("\n\n")
            buffer = ""
            for para in paragraphs:
                if len((buffer + para).split()) > max_tokens:
                    chunks.append({"text": buffer, "page": page, "heading": heading})
                    buffer = para
                else:
                    buffer += "\n\n" + para
            if buffer:
                chunks.append({"text": buffer, "page": page, "heading": heading})
    return chunks

Layout-Detection Models: When Heuristics Aren't Enough

For high-volume or highly variable document sets — think insurance claims, financial filings, or scientific papers from a dozen different publishers — coordinate heuristics eventually hit a wall. This is where layout-detection approaches earn their cost. Tools built on document layout models (trained to classify regions of a page as "title," "table," "figure," "paragraph," "footer," etc.) give you a structured map of the page before you extract a single character of text.

The workflow looks like this:

Render the PDF page to an image.
Run a layout-detection model to get bounding boxes labeled by region type.
For each region, run the appropriate extractor: text extraction for paragraphs, table extraction for tables, OCR or vision-model captioning for figures.
Reassemble regions in reading order based on their detected positions and types.

This is more infrastructure than a quick script, but if you're building rag for pdfs as a product feature rather than a one-off pipeline, it pays for itself the first time someone uploads a document your column-midpoint heuristic can't handle. A pragmatic middle ground many teams land on: use fast heuristics for the 80% of documents that are simple single or dual-column text, and route anything that fails a quick "does this look weird" check (unusual aspect ratios, low text-to-image ratio, table detection confidence below a threshold) to a heavier layout-model or vision-LLM path.

Evaluating Whether Your PDF Pipeline Actually Works

It's tempting to eyeball a few chunks and call it done, but PDF parsing bugs are often invisible until a specific query hits the exact broken section. Build a small, boring evaluation habit:

Keep a fixed set of 15-20 real documents representative of what your users actually upload — include at least one scanned PDF, one multi-column paper, one table-heavy report, and one document with lots of header/footer noise.
Write 5-10 questions per document where you know the ground-truth answer and which page/table it lives on.
After any change to your extraction or chunking code, re-run retrieval on this fixed set and check whether the correct chunk still surfaces in the top results.

This catches regressions early — the classic failure is "we improved chunking for long paragraphs and silently broke table extraction," which nobody notices until a user complains weeks later that the pricing bot gives wrong numbers.

It also helps to log which extraction path handled each page — native text, OCR, or vision-model fallback — alongside the confidence or heuristic score that triggered that path. When retrieval quality dips for a specific document, the first question is almost always "did this page get routed correctly," and without that log you're stuck re-deriving it from scratch every time. A simple table of page_number, extraction_method, confidence_score per document turns a vague "the bot is wrong" complaint into a two-minute lookup.

Practical Checklist for Production PDF RAG

Detect document type first: text-native, scanned, or mixed. Route accordingly.
Never trust raw extraction order on multi-column layouts; reconstruct reading order from bounding boxes.
Convert tables to Markdown (or structured row chunks) before they ever reach a generic text splitter.
Run OCR only where needed, at 300+ DPI, and consider a vision-LLM fallback for messy scans.
Caption embedded figures and charts with a vision model so they become searchable rather than invisible.
Strip repeating headers/footers using a frequency threshold across pages.
Attach page number and section metadata to every chunk for filtering and citation.
Maintain a small fixed evaluation set and re-check it after every pipeline change.

Closing Thoughts

None of this is glamorous work. It's coordinate math, frequency counting, and a lot of squinting at extracted text next to the original PDF to see where things went sideways. But this is exactly the work that separates a RAG demo from a RAG system people actually trust with real documents. The retrieval algorithm, the embedding model, the reranker — those are largely solved problems you can pick off a shelf. Clean extraction from messy, real-world PDFs is not, and it's usually where the actual engineering time goes on any serious document-QA project.

If you want to go deeper on how retrieval, chunking, and generation fit together once your documents are clean, our course Introduction to RAG covers the full pipeline end to end, including the parsing techniques discussed here applied to real datasets.