RAG

Building a Multi-Modal RAG System: Text, Images and Tables Together

Ira Menon · May 22, 2026 · 14 min read

Why Your RAG Pipeline Breaks the Moment a PDF Has a Chart

Most RAG tutorials assume the world is made of clean paragraphs. You chunk some text, embed it, stuff it into a vector store, and call it a day. Then someone hands you a real document — a quarterly earnings report with a revenue table on page 4, an architecture diagram on page 9, and a scanned invoice with a logo in the corner — and the whole pipeline falls apart. The text extractor either skips the table entirely, mangles it into an unreadable wall of numbers, or ignores the image altogether because "images aren't text."

This is the gap that multi-modal RAG is built to close. Instead of treating a document as a stream of characters, a multi-modal RAG system treats it as what it actually is: a mixture of text, visual layout, tables, charts, and images, each of which carries information that plain text extraction throws away. A table's meaning depends on its rows and columns lining up correctly. A chart's meaning depends on you actually looking at it. If your retrieval system can't handle that, it can't answer the questions users actually ask.

In this article we'll build a multi-modal RAG system step by step — covering how to parse mixed documents, how to embed text, images, and tables into a retrievable format, how to fuse retrieval results from different modalities, and how to generate grounded answers that cite the right modality. We'll use concrete code, not hand-waving, so you can adapt this directly into a working prototype.

What "Multi-Modal" Actually Means in a RAG Context

Before writing code, it's worth being precise about what we're solving. In a multi-modal RAG system, there are typically three problems layered on top of standard RAG:

Extraction: pulling text, tables, and images out of source documents (PDFs, slide decks, scanned reports) without losing structure.
Representation: converting each modality into a form that can be embedded and compared — this might mean embedding image pixels directly with a vision-language model, or converting a table into a textual summary that a text embedding model can handle.
Fusion: combining retrieval results across modalities so that a single user query can surface the right paragraph, the right chart, and the right table row, ranked together in a sensible order.

There are two broad architectural approaches teams use today:

Route everything through a single multi-modal embedding space (like CLIP-style models or newer multi-modal embedding APIs) so text, images, and table summaries all live in one comparable vector space.
Keep separate embedding spaces per modality, retrieve independently from each, and merge results with a fusion step (like reciprocal rank fusion) before handing them to the generator.

Both work. The unified-embedding approach is elegant but can be lossy for tables, since compressing tabular structure into a single vector destroys a lot of relational information. The multi-store approach is more code to maintain but gives you finer control, and in practice tends to produce better answers on documents with dense tables. We'll build the multi-store version here because it maps more directly onto tools you likely already use, but I'll show you where the unified approach fits too.

Step 1: Parsing Mixed-Content Documents

The first real bottleneck in multi-modal RAG is document parsing. A naive pdf_to_text call will linearize a table into something like Q1 120 Q2 98 Q3 150, which is functionally useless — you've lost which number belongs to which column.

You need a layout-aware parser that classifies each region of a page as text, table, or image, and extracts each separately. Libraries like unstructured, PyMuPDF combined with a layout model, or commercial document AI APIs all do this to varying degrees. Here's a simplified extraction pipeline using PyMuPDF for layout detection and image extraction, paired with a table-specific extractor:

import fitz  # PyMuPDF
from dataclasses import dataclass
from typing import Literal

@dataclass
class DocChunk:
    chunk_id: str
    modality: Literal["text", "table", "image"]
    content: str          # raw text, markdown table, or image path
    page: int
    source_doc: str

def extract_document(pdf_path: str) -> list[DocChunk]:
    doc = fitz.open(pdf_path)
    chunks = []

    for page_num, page in enumerate(doc):
        # 1. Extract plain text blocks
        text = page.get_text("text").strip()
        if text:
            chunks.append(DocChunk(
                chunk_id=f"{pdf_path}-p{page_num}-text",
                modality="text",
                content=text,
                page=page_num,
                source_doc=pdf_path,
            ))

        # 2. Extract embedded images
        for img_index, img in enumerate(page.get_images(full=True)):
            xref = img[0]
            base_image = doc.extract_image(xref)
            image_path = f"extracted/{page_num}_{img_index}.png"
            with open(image_path, "wb") as f:
                f.write(base_image["image"])
            chunks.append(DocChunk(
                chunk_id=f"{pdf_path}-p{page_num}-img{img_index}",
                modality="image",
                content=image_path,
                page=page_num,
                source_doc=pdf_path,
            ))

    return chunks

Table extraction deserves its own pass. Rather than treating a table as text, extract it as structured data (a list of rows) and convert it to markdown, which both humans and LLMs parse reliably:

import camelot  # table-specific extraction

def extract_tables(pdf_path: str) -> list[DocChunk]:
    tables = camelot.read_pdf(pdf_path, pages="all", flavor="lattice")
    chunks = []
    for i, table in enumerate(tables):
        df = table.df
        markdown_table = df.to_markdown(index=False)
        chunks.append(DocChunk(
            chunk_id=f"{pdf_path}-table{i}",
            modality="table",
            content=markdown_table,
            page=table.page,
            source_doc=pdf_path,
        ))
    return chunks

Notice that we're keeping the table as markdown rather than flattening it into prose. This preserves row/column alignment, which matters enormously when the generator later needs to answer "what was Q3 revenue" — it can look at a well-formed row instead of guessing from a jumble of numbers.

Step 2: Representing Each Modality for Retrieval

Once you have text, table, and image chunks, each needs a representation that supports similarity search.

Text chunks are the easy case — chunk them semantically (by paragraph or heading boundary, not fixed character counts) and embed with any standard text embedding model.

Table chunks benefit from a dual representation: embed the markdown table directly using a text embedding model (tables read as text still carry semantic signal — column headers, entity names), but also generate a short natural-language caption describing what the table contains, and embed that caption too. Captions dramatically improve recall because a user query like "which region grew fastest" matches better against "This table shows quarterly revenue by region" than against raw numeric cells.

Image chunks are where you have a real choice. You can either:

Embed the image directly using a vision-capable embedding model, so image and text queries land in a shared space.
Generate a textual description of the image (via a vision-language model) and embed that description with your text embedding model, keeping everything in one text-based vector store.

The second approach is simpler to operate and integrates cleanly with existing text-only vector databases, so it's a good default for most teams starting out:

def caption_image(image_path: str, vlm_client) -> str:
    """Use a vision-language model to produce a retrievable text description."""
    response = vlm_client.describe(
        image_path=image_path,
        prompt=(
            "Describe this image in detail for search purposes. "
            "If it's a chart, state the chart type, axes, and key trend. "
            "If it's a diagram, describe the components and their relationships."
        ),
    )
    return response.text

def embed_chunks(chunks: list[DocChunk], embed_model, vlm_client) -> list[dict]:
    records = []
    for chunk in chunks:
        if chunk.modality == "text":
            text_for_embedding = chunk.content
        elif chunk.modality == "table":
            caption = f"Table data:\n{chunk.content}"
            text_for_embedding = caption
        elif chunk.modality == "image":
            text_for_embedding = caption_image(chunk.content, vlm_client)

        vector = embed_model.embed(text_for_embedding)
        records.append({
            "id": chunk.chunk_id,
            "vector": vector,
            "modality": chunk.modality,
            "raw_content": chunk.content,
            "searchable_text": text_for_embedding,
            "page": chunk.page,
        })
    return records

This keeps every modality in the same vector store schema, tagged with a modality field, which makes the next step — fusion and filtering — much simpler.

Step 3: Storing and Indexing Across Modalities

With everything embedded into a common vector space (or separate spaces, if you go that route), the storage layer needs to preserve enough metadata to reconstruct context at answer time. A single flat vector index isn't enough — you want to filter by modality, page, and source document, and you want to be able to pull the original image file back for display, not just its caption.

A practical schema in a vector database like Qdrant, Weaviate, or pgvector looks like this:

import psycopg2
from pgvector.psycopg2 import register_vector

def upsert_records(conn, records: list[dict]):
    register_vector(conn)
    cur = conn.cursor()
    for r in records:
        cur.execute(
            """
            INSERT INTO doc_chunks
                (id, modality, raw_content, searchable_text, page, embedding)
            VALUES (%s, %s, %s, %s, %s, %s)
            ON CONFLICT (id) DO UPDATE SET embedding = EXCLUDED.embedding
            """,
            (r["id"], r["modality"], r["raw_content"],
             r["searchable_text"], r["page"], r["vector"]),
        )
    conn.commit()

The key design decision here: raw_content stores the *original* representation (the image file path, the markdown table, the raw paragraph), while searchable_text stores what got embedded. At retrieval time you search against searchable_text embeddings but you hand the generator (and the user) the raw_content. This separation is what lets an image get retrieved via its caption but still show up in the final answer as an actual image, not a text description of one.

Step 4: Retrieval and Cross-Modal Fusion

When a query comes in, you typically want to search across all modalities and then merge results, rather than deciding up front "this is a text question" or "this is an image question." A finance question like "show me the trend in customer churn" might be best answered by a chart, a table, or a paragraph — you don't know which until you look.

Reciprocal rank fusion is a simple, robust way to combine ranked lists from different retrieval passes (or different modality-specific indexes) without needing to calibrate scores across incompatible embedding spaces:

def reciprocal_rank_fusion(ranked_lists: list[list[str]], k: int = 60) -> list[str]:
    """
    ranked_lists: multiple lists of chunk_ids, each already sorted by relevance
    Returns a single fused ranking.
    """
    scores = {}
    for ranked_list in ranked_lists:
        for rank, chunk_id in enumerate(ranked_list):
            scores[chunk_id] = scores.get(chunk_id, 0) + 1 / (k + rank + 1)

    fused = sorted(scores.items(), key=lambda x: x[1], reverse=True)
    return [chunk_id for chunk_id, _ in fused]


def multi_modal_retrieve(query: str, embed_model, conn, top_k: int = 5) -> list[dict]:
    query_vector = embed_model.embed(query)
    cur = conn.cursor()

    results_by_modality = {}
    for modality in ("text", "table", "image"):
        cur.execute(
            """
            SELECT id, raw_content, modality, page
            FROM doc_chunks
            WHERE modality = %s
            ORDER BY embedding <-> %s
            LIMIT %s
            """,
            (modality, query_vector, top_k),
        )
        results_by_modality[modality] = cur.fetchall()

    ranked_ids = [
        [row[0] for row in rows] for rows in results_by_modality.values()
    ]
    fused_order = reciprocal_rank_fusion(ranked_ids)

    all_rows = {row[0]: row for rows in results_by_modality.values() for row in rows}
    return [
        {"id": cid, "content": all_rows[cid][1], "modality": all_rows[cid][2], "page": all_rows[cid][3]}
        for cid in fused_order[:top_k]
        if cid in all_rows
    ]

Running separate top-k searches per modality and fusing afterward, rather than one blended search, avoids a common failure mode: text chunks are usually far more numerous than tables or images, so a single blended vector search tends to drown out the table and image hits purely on volume. Retrieving per modality first and fusing second keeps every modality in the running.

Step 5: Generation With Grounded, Multi-Modal Context

The generation step is where multi-modal RAG diverges most from text-only RAG. You're not just stuffing retrieved text into a prompt — you're assembling a context that may include actual images (passed to a vision-capable model), markdown tables (passed as structured text), and prose, all labeled so the model knows what it's looking at and can cite it correctly.

def build_generation_messages(query: str, retrieved: list[dict]) -> list[dict]:
    content_blocks = [{"type": "text", "text": f"User question: {query}\n\nUse the following retrieved context to answer. Cite the source type (text, table, or image) for each fact you use."}]

    for item in retrieved:
        if item["modality"] == "text":
            content_blocks.append({
                "type": "text",
                "text": f"[TEXT - page {item['page']}]\n{item['content']}",
            })
        elif item["modality"] == "table":
            content_blocks.append({
                "type": "text",
                "text": f"[TABLE - page {item['page']}]\n{item['content']}",
            })
        elif item["modality"] == "image":
            content_blocks.append({
                "type": "text",
                "text": f"[IMAGE - page {item['page']}]",
            })
            content_blocks.append({
                "type": "image",
                "source": {"type": "path", "path": item["content"]},
            })

    return [{"role": "user", "content": content_blocks}]

Passing the actual image bytes to a vision-capable model at generation time — rather than relying solely on the caption you generated during indexing — matters more than it looks. The caption gets you *retrieval*, but the raw pixels get you *accuracy*. A caption might say "bar chart showing revenue growth," but only the actual image tells the model that Q3 dipped slightly before recovering in Q4. Skipping this step and generating answers from captions alone is the single most common corner-cutting mistake teams make when they first bolt image support onto an existing text RAG pipeline — it looks like it works until someone asks a question that depends on a visual detail the caption didn't capture.

Handling Tables Well: The Details That Matter

Tables deserve extra attention because they're the modality most likely to be silently mangled. A few practical rules that make a measurable difference:

Preserve headers on every extracted row context. If a table spans a page break, make sure the header row is repeated or explicitly attached to both halves before chunking, otherwise the second half becomes meaningless.
Don't split a table across chunks by character count. Chunk tables as whole units (or logical sub-sections like "by region") — never mid-row.
Generate a caption chunk separately from the raw table chunk, as shown earlier, so retrieval can match on natural language while generation still gets the exact numbers.
Normalize numeric formatting during extraction (strip currency symbols and thousands separators into a consistent format) so the generator doesn't have to guess whether 1,200 and 1200 are the same value.
Validate row/column counts after extraction. Table extraction libraries occasionally merge or drop cells on complex layouts; a quick sanity check (row length consistency) catches this before it silently corrupts an answer.

Evaluating a Multi-Modal RAG System

Evaluation is where a lot of teams under-invest, and it's exactly where multi-modal systems need it most, because failures are quieter than in text-only RAG — the system doesn't error out, it just answers confidently from the wrong modality or misses a chart entirely.

A workable evaluation set should include:

Questions answerable only from tables (e.g., "what was the exact figure for X in Q2").
Questions answerable only from images (e.g., "does the trend line show acceleration or deceleration").
Questions answerable only from surrounding text (e.g., "what caused the dip mentioned in the report").
Questions that require combining modalities (e.g., "does the chart on page 9 match the numbers in the table on page 4").

For each, track two things separately: retrieval recall (did the right chunk, of the right modality, show up in the top-k results) and answer faithfulness (did the generated answer actually match what's in the retrieved content, rather than hallucinating a plausible-sounding number). It's entirely possible to retrieve the correct table and still generate a wrong figure if the generator misreads a markdown table with merged cells — so testing retrieval and generation as separate stages, rather than only checking final answer quality, will save you a lot of confused debugging later.

Common Pitfalls and How to Avoid Them

A few mistakes show up repeatedly when teams build their first multi-modal RAG system:

Treating images as decoration. Skipping image extraction because "most images are just logos" means missing the charts and diagrams that actually carry information. Filter out logos and decorative images at extraction time based on size or aspect ratio heuristics, rather than skipping image extraction altogether.
Flattening tables into paragraphs. This is the single biggest source of wrong numeric answers. Keep tables structured through the entire pipeline.
Relying on captions alone at generation time. As covered above, pass the actual image to the generator when a vision-capable model is available.
Using one blended vector index without modality tags. Without a modality field to filter or fuse on, text chunks (which vastly outnumber tables and images in most documents) dominate every search.
Ignoring page-level provenance. Multi-modal documents are exactly the case where users want to verify an answer against the source page — always propagate page numbers and source document IDs through every stage.

Where to Go From Here

Multi-modal RAG isn't a different discipline from text RAG — it's the same retrieval-then-generate loop, but with more honest handling of what documents actually look like. The core additions are a layout-aware extraction step, per-modality representations that don't destroy structure, a fusion strategy that keeps low-volume modalities from being drowned out, and a generation step that hands the model real images and real tables instead of lossy text summaries.

If you're still getting comfortable with the retrieval fundamentals — chunking strategy, embedding model selection, vector store tradeoffs, and how the retrieve-then-generate loop fits together — it's worth going back to the basics before layering on multi-modal complexity. Our Introduction to RAG course covers exactly that foundation, and it'll make everything in this article click faster once the core mental model is solid.