Document Preprocessing for RAG: A Practical Pipeline That Actually Works

RAG document preprocessing is the work of turning messy source files, PDFs, Word docs, HTML pages, scanned reports, into clean, chunked, metadata-rich text that an embedding model can actually use well. Most retrieval-augmented generation pipelines don't fail because of a bad embedding model or a weak LLM. They fail because the text that got embedded was garbled, badly chunked, or stripped of the context a retriever needs to rank it correctly. This guide walks through a complete preprocessing pipeline you can run today, with runnable code, no filler.

Why Document Preprocessing Is the Real Bottleneck in RAG Pipelines

Every RAG system follows the same rough shape: extract text, chunk it, embed it, store it, retrieve it, feed it to an LLM. Teams spend most of their tuning time on the last three steps, picking a vector database, comparing embedding models, tweaking prompts, while the first two steps quietly determine the ceiling on quality.

If a PDF extractor merges a footer into the middle of a paragraph, every chunk that touches that paragraph is now noisy. If a chunker splits a table in half, neither half means anything on its own. If a Word document's headings get flattened into plain text, you lose the structural signal that would have told your retriever "this paragraph is about pricing, not installation." None of this shows up as an error. It shows up three weeks later as "the chatbot keeps giving wrong answers" and nobody can tell why.

Good preprocessing is boring, mechanical, and non-negotiable. Treat it as a first-class pipeline stage with its own tests, not a throwaway script you ran once.

Extracting Text From PDFs, Word Docs, and HTML

Different formats need different extractors, and using the wrong one is the single most common cause of corrupted RAG input.

For PDFs, pypdf is fine for simple, text-based PDFs but struggles with multi-column layouts and embedded tables. PyMuPDF (imported as fitz) is faster and preserves layout better, which matters for reports and academic papers with columns.

import fitz  # PyMuPDF

def extract_pdf_text(path):
    doc = fitz.open(path)
    pages = []
    for page_num, page in enumerate(doc):
        text = page.get_text("text")
        pages.append({"page": page_num + 1, "text": text})
    doc.close()
    return pages

For scanned PDFs or image-based documents, you need OCR before any of this works. pytesseract wrapped around pdf2image handles most cases:

from pdf2image import convert_from_path
import pytesseract

def ocr_pdf(path):
    images = convert_from_path(path, dpi=300)
    return [pytesseract.image_to_string(img) for img in images]

For Word documents, python-docx gives you paragraph and heading structure directly, which is more useful than dumping raw text:

from docx import Document

def extract_docx(path):
    doc = Document(path)
    blocks = []
    for para in doc.paragraphs:
        if not para.text.strip():
            continue
        style = para.style.name if para.style else "Normal"
        blocks.append({"text": para.text, "style": style})
    return blocks

For HTML, resist the urge to strip tags with a regex. Use BeautifulSoup and explicitly drop navigation, scripts, and ads before extracting text, otherwise your chunks fill up with menu labels and cookie banners:

from bs4 import BeautifulSoup

def extract_html_text(html):
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "nav", "header", "footer", "aside"]):
        tag.decompose()
    return soup.get_text(separator="\n", strip=True)

If you're dealing with a mix of formats at scale, unstructured (the open source library, not a specific vendor) gives you a single interface across PDFs, DOCX, HTML, PPTX, and email formats, and it returns typed elements like Title, NarrativeText, and Table instead of one flat string. That typing is worth a lot downstream:

from unstructured.partition.auto import partition

elements = partition(filename="report.pdf")
for el in elements:
    print(type(el).__name__, el.text[:80])

Cleaning and Normalizing Text Before Chunking

Raw extracted text almost always carries junk: repeated headers and footers, broken hyphenation from PDF line wraps, mixed unicode encodings, and inconsistent whitespace. Clean this before chunking, not after, because chunking on dirty text bakes the mess into every chunk boundary.

Fix encoding issues with ftfy ("fixes text for you"), which repairs mojibake like curly quotes that got double-encoded:

import ftfy

def fix_encoding(text):
    return ftfy.fix_text(text)

Collapse repeated whitespace and de-hyphenate words broken across line wraps:

import re

def clean_text(text):
    # rejoin words split by a line-wrap hyphen
    text = re.sub(r"(\w+)-\n(\w+)", r"\1\2", text)
    # collapse newlines and spaces
    text = re.sub(r"\n{3,}", "\n\n", text)
    text = re.sub(r"[ \t]{2,}", " ", text)
    return text.strip()

Detect and strip repeated headers and footers by looking for lines that appear on nearly every page of a document. A simple frequency check across page-level text works well in practice:

from collections import Counter

def strip_repeated_lines(pages, threshold=0.6):
    line_counts = Counter()
    total_pages = len(pages)
    for page in pages:
        lines = set(line.strip() for line in page["text"].split("\n") if line.strip())
        line_counts.update(lines)

    boilerplate = {
        line for line, count in line_counts.items()
        if count / total_pages >= threshold
    }

    for page in pages:
        lines = page["text"].split("\n")
        page["text"] = "\n".join(l for l in lines if l.strip() not in boilerplate)
    return pages

Run language detection early if your corpus is multilingual. langdetect or fasttext's language identification model lets you route documents to language-specific chunking and embedding models instead of silently mangling non-English text:

from langdetect import detect

def tag_language(text):
    try:
        return detect(text[:1000])
    except Exception:
        return "unknown"

Chunking Strategies for RAG Document Preprocessing

Chunking is where most RAG document preprocessing pipelines quietly lose quality. Fixed-size character splitting is the easiest option and the worst one for anything beyond a quick prototype, because it cuts sentences and tables in half without regard for meaning.

Start with a recursive splitter that tries paragraph breaks first, then sentence breaks, then words, only falling back to hard character limits as a last resort. LangChain's RecursiveCharacterTextSplitter implements exactly this:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=120,
    separators=["\n\n", "\n", ". ", " ", ""],
)

chunks = splitter.split_text(cleaned_text)

Size chunks by tokens, not characters, since your embedding model and LLM context window both operate on tokens. tiktoken lets you measure and split against the actual tokenizer your model uses:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def token_length(text):
    return len(enc.encode(text))

Pass token_length as the length_function to the splitter so chunk_size=800 means 800 tokens, not 800 characters:

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=80,
    length_function=token_length,
)

For structured documents, semantic or structure-aware chunking beats fixed sizes. If you extracted typed elements with unstructured, chunk on section boundaries defined by Title elements so each chunk stays within one topic:

def chunk_by_sections(elements, max_tokens=500):
    chunks, current, current_title = [], [], "Untitled"
    for el in elements:
        if type(el).__name__ == "Title":
            if current:
                chunks.append({"title": current_title, "text": "\n".join(current)})
            current, current_title = [], el.text
        else:
            current.append(el.text)
    if current:
        chunks.append({"title": current_title, "text": "\n".join(current)})
    return chunks

Always keep a chunk overlap of roughly 10-15% of chunk size. It costs some duplication in your index but prevents a fact from getting sliced exactly at a chunk boundary and disappearing from both halves' retrieved context.

Preserving and Enriching Metadata

A chunk without metadata is a fact with no source. At minimum, attach document title, source path, page number, section heading, and ingestion date to every chunk before it goes into your vector store:

def build_chunk_record(chunk_text, doc_meta, page=None, section=None):
    return {
        "text": chunk_text,
        "source": doc_meta["filename"],
        "doc_title": doc_meta.get("title", doc_meta["filename"]),
        "page": page,
        "section": section,
        "token_count": token_length(chunk_text),
    }

This metadata does two jobs. First, it lets you cite sources back to the user, which matters for trust and for debugging bad answers. Second, many vector databases support metadata filtering, so you can restrict a search to a specific document type, date range, or section before the similarity search even runs, which improves both speed and precision.

If your source documents have version history, also record a content hash per chunk. When you re-ingest an updated document, you can diff hashes and only re-embed chunks that actually changed instead of re-embedding an entire corpus on every update.

Handling Tables, Code Blocks, and Structured Content

Tables and code blocks break under naive text chunking because their meaning depends on structure that plain text splitters don't respect. A table row split from its header row is unreadable to both a human and an embedding model.

Extract tables separately and serialize them into a format that keeps rows tied to headers, either markdown-style rows or a flattened sentence per row:

import pandas as pd

def table_to_chunks(df: pd.DataFrame, table_name="table"):
    chunks = []
    headers = list(df.columns)
    for _, row in df.iterrows():
        sentence = ", ".join(f"{h}: {row[h]}" for h in headers)
        chunks.append(f"{table_name} row: {sentence}")
    return chunks

This turns each row into a self-contained sentence that survives chunking and retrieval independently, at the cost of some redundancy in header names. For large tables, group a handful of rows per chunk instead of one row per chunk to keep chunk count reasonable.

For code blocks in technical documentation, keep the block intact rather than letting a generic splitter cut it mid-function. Detect fenced code blocks with a regex before running your main splitter, extract them as their own chunks tagged content_type: code, and splice the surrounding prose text around them:

import re

def extract_code_blocks(text):
    pattern = re.compile(r"```[\s\S]*?```")
    code_blocks = pattern.findall(text)
    prose = pattern.sub("[[CODE_BLOCK]]", text)
    return prose, code_blocks

Deduplication and Near-Duplicate Detection

Real document sets are full of duplicates: the same slide deck exported twice, a policy document with three near-identical revisions floating in a shared drive, boilerplate legal text repeated across every contract. Duplicate chunks waste index space and, worse, they can dominate retrieval results, pushing out genuinely different relevant chunks because five near-identical copies of the same paragraph all score high.

Exact duplicates are cheap to catch with a content hash before embedding:

import hashlib

def content_hash(text):
    return hashlib.sha256(text.strip().lower().encode()).hexdigest()

def dedupe_exact(chunks):
    seen, out = set(), []
    for c in chunks:
        h = content_hash(c["text"])
        if h not in seen:
            seen.add(h)
            out.append(c)
    return out

Near-duplicates need a similarity check. MinHash with locality-sensitive hashing (via the datasketch library) scales to large corpora without pairwise comparison cost:

from datasketch import MinHash, MinHashLSH

def build_lsh(chunks, threshold=0.85):
    lsh = MinHashLSH(threshold=threshold, num_perm=128)
    minhashes = {}
    for i, c in enumerate(chunks):
        m = MinHash(num_perm=128)
        for word in c["text"].lower().split():
            m.update(word.encode("utf8"))
        lsh.insert(str(i), m)
        minhashes[i] = m
    return lsh, minhashes

Run this as a filtering pass after chunking and before embedding. It's cheaper to drop near-duplicates here than to pay for redundant embedding calls and carry the noise into your vector index permanently.

Redacting PII Before It Reaches Your Vector Store

If your document set includes customer records, support tickets, contracts, or internal HR files, PII redaction belongs in the preprocessing pipeline, not as an afterthought bolted onto the retrieval layer. Once sensitive text is embedded and stored, scrubbing it later means re-processing your entire index.

Microsoft's presidio-analyzer and presidio-anonymizer detect and mask common PII types (names, emails, phone numbers, credit card numbers, government IDs) with configurable recognizers:

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

def redact_pii(text):
    results = analyzer.analyze(text=text, language="en")
    anonymized = anonymizer.anonymize(text=text, analyzer_results=results)
    return anonymized.text

Run redaction on the cleaned text before chunking, so a chunk boundary never accidentally splits a partially-masked entity in a way that leaks half of it. Log what got redacted (type and count, not the raw value) so you can audit coverage without storing the sensitive data twice.

A Full Preprocessing Pipeline You Can Run Today

Putting the pieces together, a minimal end-to-end pipeline looks like this:

def preprocess_document(path, doc_meta):
    # 1. extract
    if path.endswith(".pdf"):
        pages = extract_pdf_text(path)
        raw_text = "\n\n".join(p["text"] for p in pages)
    elif path.endswith(".docx"):
        blocks = extract_docx(path)
        raw_text = "\n\n".join(b["text"] for b in blocks)
    else:
        raise ValueError("unsupported format")

    # 2. clean
    text = fix_encoding(raw_text)
    text = clean_text(text)
    text = redact_pii(text)

    # 3. chunk
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=500,
        chunk_overlap=80,
        length_function=token_length,
    )
    raw_chunks = splitter.split_text(text)

    # 4. attach metadata
    records = [build_chunk_record(c, doc_meta) for c in raw_chunks]

    # 5. dedupe
    records = dedupe_exact(records)

    return records

Wire this into a batch job that walks your document store, runs each file through preprocess_document, and writes the resulting records to whatever queue feeds your embedding step. Keep the pipeline idempotent by keying on content hash, so re-running it on an unchanged corpus doesn't re-embed anything.

Before you point this at production data, run it against a deliberately messy sample: a scanned PDF, a document with an embedded table, a bilingual file, and a duplicate pair. If all four come out clean, chunked sensibly, and correctly tagged, the pipeline is ready for the rest of your corpus.

FAQ

What chunk size works best for RAG document preprocessing? There's no universal number, but 300-800 tokens with 10-15% overlap is a reasonable starting range for prose-heavy documents. Shorter chunks improve retrieval precision but lose surrounding context; longer chunks keep context but dilute the embedding's focus. Test against your own query set rather than trusting a default.

Should I chunk before or after cleaning the text? Always clean first. Chunking dirty text bakes header repetition, broken hyphenation, and encoding errors into every chunk boundary, and fixing it afterward means re-chunking anyway.

Do I need OCR if my PDFs already have a text layer? No. Check first with a quick text extraction call; if it returns substantial text, skip OCR entirely, since OCR is slower and introduces its own error rate. Reserve OCR for pages where text extraction returns empty or near-empty strings, which usually means the page is a scanned image.

How do I handle tables that span multiple pages? Extract tables as a distinct step from general text extraction, using a library that tracks table structure like unstructured or camelot, and stitch continuation rows together by matching header signatures across page boundaries before you serialize rows into chunks.

Is metadata filtering actually necessary, or just nice to have? It's close to necessary at any real scale. Without metadata filters, every query searches your entire corpus by embedding similarity alone, which gets noisier as the corpus grows. Filtering by document type, date, or section before the similarity search materially improves both precision and latency.

How often should I re-run preprocessing on an existing corpus? Re-run it whenever your extraction or chunking logic changes, and incrementally whenever source documents are added or updated. Use content hashing so incremental updates only reprocess changed documents instead of the full corpus every time.