Multimodal RAG: Retrieving Images, Tables and Charts
Multimodal RAG is retrieval-augmented generation where the retrieval layer indexes and returns images, tables and charts, not just prose. You need it because most working corpora are PDFs, slide decks and wikis where the answer to a real question lives in a bar chart, a schematic or a pricing table, and a text-only pipeline drops all three without raising a single error. In 2026 there are three architectures that reliably work: caption everything into text, embed text and images into one shared vector space, or retrieve page images directly with a late-interaction model from the ColPali family.
This guide builds all three with runnable Python, covers the table and chart handling details that decide whether your numeric answers are right, and closes with an evaluation recipe and a decision guide. The examples use Docling for parsing, Chroma and Qdrant for storage, and current vision models for captioning and generation, but every pattern ports to whatever stack you already run.
Why text-only RAG fails on real documents
The failure mode is silent, which is what makes it dangerous. A PDF text extractor returns some string for every page, your embedding model happily embeds it, your retriever retrieves it, and every dashboard stays green. Meanwhile the system cannot answer a question about anything that was drawn rather than typed.
Three things break, in order of how often they bite:
- Tables get flattened into whitespace soup. The numbers survive, but their relationship to row and column headers does not, so the retriever can find the chunk and the generator still cannot tell which number belongs to which region.
- Charts vanish entirely. A chart is an image; a text extractor gives you the title and maybe an axis label. The data itself, the thing people actually ask about, is gone.
- Diagrams and screenshots carry the payload in runbooks, architecture docs and support macros. "See the screenshot below" followed by nothing is a common sight in extracted text.
Here is what a typical extractor gives you for a clean three-row churn table:
Region Q2 churn Q3 churn NA 4.1% 3.8% EMEA 5.0% 4.6% APAC 6.2% 5.1%Now run the question "What was enterprise churn in Q3?" against a corpus where the only occurrence of that number is a bar chart. The retriever returns the neighboring prose ("churn improved across all segments"), and the generator either hedges or invents a plausible figure. The document contained the answer the whole time, in pixels.
If your corpus is annual reports, clinical documents, datasheets, invoices or exported slide decks, a meaningful share of the facts users ask about is non-prose. Before building anything, sample 30 to 50 real user questions and mark where each answer physically lives: paragraph, table cell, chart, diagram. That number tells you how much a multimodal RAG pipeline is worth to you, and it becomes your evaluation set later.
The three multimodal RAG architectures
Every production multimodal RAG system I have seen is one of these three, or a hybrid of them.
- Caption everything, embed text. At ingest, a vision model writes a dense textual description of every image, chart and table. You embed those descriptions with your existing text embedding model and store a pointer to the original asset. At query time you retrieve descriptions, and optionally hand the original image back to the generator. Cheapest to adopt because your retrieval stack stays text-only.
- True multimodal embeddings. Models like voyage-multimodal-3, Cohere Embed v4 and the open-weight Jina CLIP v2 map text and images into one shared vector space. You embed figures directly, no captions needed, and a text query retrieves images by visual and semantic content.
- Page-image late interaction. The ColPali family (ColPali, ColQwen2) skips parsing entirely. Every PDF page becomes an image, a vision-language model turns it into a grid of patch embeddings, and retrieval scores query tokens against page patches ColBERT-style. Layout, tables, charts and stamps all become signal instead of noise.
These are not mutually exclusive. A common production shape is captions for tables plus page-image retrieval for scanned archives, merged at query time. Pick one as your backbone, then hybridize where the eval says you must.
Parsing PDFs into text, tables and images
Architectures 1 and 2 need a parser that preserves structure instead of flattening it. The current field: Docling (open source from IBM, strong table structure recovery), Unstructured, marker for fast PDF-to-markdown, PyMuPDF when you want raw speed and are willing to do layout work yourself, and pdfplumber or Camelot for stubborn table extraction.
Install the tools used in this article:
pip install docling chromadb anthropic voyageai byaldi qdrant-clientDocling gives you tables as structured objects and figures as image crops with page provenance:
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import PdfPipelineOptions
from docling.document_converter import DocumentConverter, PdfFormatOption
opts = PdfPipelineOptions()
opts.generate_picture_images = True # keep figure crops
opts.images_scale = 2.0 # render at 2x so axis labels stay legible
converter = DocumentConverter(
format_options={InputFormat.PDF: PdfFormatOption(pipeline_options=opts)}
)
doc = converter.convert("q3-report.pdf").document
tables = [
{"markdown": t.export_to_markdown(doc), "page": t.prov[0].page_no}
for t in doc.tables
]
figures = []
for i, pic in enumerate(doc.pictures):
img = pic.get_image(doc) # PIL image, or None if extraction failed
if img is None:
continue
path = f"assets/figure_{i:03d}.png"
img.save(path)
figures.append({"path": path, "page": pic.prov[0].page_no})Two details matter more than they look. First, images_scale: if the crop is too small for the captioning model to read the axis labels, the caption will be confidently wrong and it will poison your index. Second, keep the page number from prov on every element. Page-level citations are the difference between an answer users trust and one they re-verify by hand.
For scanned input, enable Docling's OCR options (it supports Tesseract, EasyOCR and RapidOCR backends), or skip OCR entirely with architecture 3.
Architecture 1: caption everything, embed text
The core pattern here is the multi-vector idea: embed a retrieval proxy, return the payload. The proxy is a text description of the figure or table; the payload is the original asset. LangChain ships this as MultiVectorRetriever, but it is ten lines by hand and worth understanding at that level.
The captioning prompt matters more than the captioning model. A generic "describe this image" produces captions like "a bar chart showing churn", which retrieves nothing useful. Demand the searchable facts:
import base64
import anthropic
client = anthropic.Anthropic()
CAPTION_PROMPT = """Describe this figure for a search index.
State the figure type (bar chart, line chart, table, diagram, screenshot).
List every axis label with units, every series name, and the exact values
of all labeled data points. Do not round numbers. End with one sentence
naming the main trend or takeaway."""
def describe_image(path: str) -> str:
with open(path, "rb") as f:
data = base64.standard_b64encode(f.read()).decode()
msg = client.messages.create(
model="claude-haiku-4-5", # small vision tier is enough at ingest
max_tokens=500,
messages=[{
"role": "user",
"content": [
{"type": "image",
"source": {"type": "base64", "media_type": "image/png", "data": data}},
{"type": "text", "text": CAPTION_PROMPT},
],
}],
)
return msg.content[0].text
def summarize_table(markdown: str) -> str:
msg = client.messages.create(
model="claude-haiku-4-5",
max_tokens=300,
messages=[{
"role": "user",
"content": "Summarize this table for a search index in two sentences: "
"what it measures, its row and column headers, units, and "
"the range of values.\n\n" + markdown,
}],
)
return msg.content[0].textIndex the proxies and keep the payloads in metadata:
import chromadb
chroma = chromadb.PersistentClient(path="./rag-index")
col = chroma.get_or_create_collection("q3_report")
for i, fig in enumerate(figures):
col.add(
ids=[f"figure-{i}"],
documents=[describe_image(fig["path"])],
metadatas=[{"kind": "figure", "asset": fig["path"], "page": fig["page"]}],
)
for i, t in enumerate(tables):
col.add(
ids=[f"table-{i}"],
documents=[summarize_table(t["markdown"])],
metadatas=[{"kind": "table", "raw": t["markdown"], "page": t["page"]}],
)Chroma embeds the documents with its default local model; in production you would plug in your usual text embedder. Text chunks from the same document go into the same collection the same way, so figures, tables and prose compete in one ranked list.
Why this architecture wins so often: it reuses your entire existing text RAG stack, and it is debuggable. When retrieval misses, you read the caption and immediately see why. The cost is one cheap vision call per figure at ingest, and the ceiling is caption quality: anything the captioner did not mention is unfindable forever. That is why the prompt above forbids rounding and demands axis units.
Architecture 2: true multimodal embeddings
CLIP started this line in 2021: train an image encoder and a text encoder so matching pairs land close together in one vector space. The current generation is trained heavily on documents, screenshots and figures rather than just natural photos, which is what makes it usable for multimodal RAG. Commercial options include voyage-multimodal-3 and Cohere Embed v4; open-weight options include Jina CLIP v2 and Nomic Embed Vision.
With Voyage, text and images go through one API and interleave freely:
import voyageai
from PIL import Image
vo = voyageai.Client() # reads VOYAGE_API_KEY
inputs = [[Image.open(f["path"])] for f in figures] + [[c] for c in text_chunks]
docs = vo.multimodal_embed(
inputs=inputs,
model="voyage-multimodal-3",
input_type="document",
)
q = vo.multimodal_embed(
inputs=[["What was enterprise churn by region in Q3?"]],
model="voyage-multimodal-3",
input_type="query",
).embeddings[0]Store docs.embeddings in any vector database as ordinary dense vectors; nothing downstream knows some of them came from pixels.
The appeal is that nothing is lost in translation: no captioner deciding what matters, the embedding sees the actual figure. The caveats are real, though. There is a modality gap: text-to-text similarity often scores systematically differently from text-to-image similarity, so a mixed collection can rank all prose above all figures for some queries. If you see that skew, index each modality in its own collection and merge the ranked lists with a fixed quota or reciprocal rank fusion. And for exact-number lookups in tables, a screenshot embedding loses to a markdown rendering of the same table almost every time. Embed tables as text even in this architecture.
Architecture 3: ColPali and page-image retrieval
ColPali flipped the problem in 2024: instead of parsing the document into elements, treat every page as an image. A vision-language model encodes the page into roughly a thousand patch embeddings of 128 dimensions each, and a query is encoded into token embeddings. Scoring is late interaction, ColBERT-style: each query token finds its best-matching page patch (MaxSim) and the scores sum. The ViDoRe benchmark was built to measure exactly this task, and ColQwen2, built on Qwen2-VL, is the stronger successor to the original PaliGemma-based ColPali.
The practical consequence is enormous: no parser, no OCR, no chunking strategy, no captioning prompts. Layout, fonts, stamps, handwriting and chart geometry all become retrieval signal. For scanned archives, forms and invoices, this family is usually the accuracy leader.
The byaldi library makes it a five-line proof of concept:
from byaldi import RAGMultiModalModel
rag = RAGMultiModalModel.from_pretrained("vidore/colqwen2-v1.0")
rag.index(
input_path="reports/", # a folder of PDFs, nothing preprocessed
index_name="finance-reports",
store_collection_with_index=True, # keep base64 page images for generation
overwrite=True,
)
hits = rag.search("What was enterprise churn by region in Q3?", k=3)
for h in hits:
print(h.doc_id, h.page_num, h.score)For production scale you want a vector database with native multivector support. Qdrant added a MaxSim comparator for exactly this:
from qdrant_client import QdrantClient, models
qdrant = QdrantClient(url="http://localhost:6333")
qdrant.create_collection(
collection_name="pages",
vectors_config=models.VectorParams(
size=128,
distance=models.Distance.COSINE,
multivector_config=models.MultiVectorConfig(
comparator=models.MultiVectorComparator.MAX_SIM
),
),
)Two costs to plan for. Storage: a thousand 128-dimensional vectors per page is two orders of magnitude more floats than one pooled page embedding; binary quantization and token pooling (both supported in the colpali-engine ecosystem and in Qdrant) claw most of it back with small accuracy loss. Latency: MaxSim over every page is expensive at scale, so the standard pattern is a cheap first stage (pooled vectors or BM25 over whatever text you have) followed by multivector rescoring of the top few hundred candidates.
One more consequence: retrieval returns pixels, not text, so your generator must be a vision model. That pairs naturally with the generation section below.
Getting tables right
Tables deserve their own rules regardless of architecture, because they are where wrong answers get expensive.
Extract structure, not strings. Markdown or HTML preserves the header-to-cell relationships that plain text destroys:
| Region | Q2 churn | Q3 churn |
|--------|----------|----------|
| NA | 4.1% | 3.8% |
| EMEA | 5.0% | 4.6% |Then apply the split that makes multimodal RAG work: the summary is for retrieval, the raw table is for generation. Summaries answer "which table is relevant"; the markdown answers "which cell is the number". Never make the generator work from the summary alone, and never embed the raw markdown of a 40-row table and hope the embedding captures it.
For long tables, chunk by row groups of 20 to 50 rows and repeat the header row (and any units row) in every chunk, so each chunk is independently interpretable. Keep a table_id in metadata so you can reassemble the full table when the generator needs totals.
For numeric integrity: pass tables to the generator as text, not screenshots, whenever you have clean extraction. Models still misread digits in dense table images more often than they misparse markdown, and in our experience the failure is worst exactly where it hurts, on long decimal-heavy financial tables. And never let a captioner round: "revenue around 4.2M" in the index becomes "4.2M" in an answer that should have said 4,183,000.
Getting charts right
A chart is a lossy rendering of an underlying data series, and your job at ingest is to recover as much of the series as possible. There are two levels.
Level one is the descriptive caption from architecture 1, which makes the chart findable. Level two is reconstruction: ask the vision model to emit the underlying data as JSON at ingest and index that as text. Reconstruction is what makes exact-value questions answerable without a vision call at query time. A prompt that works well:
Extract the underlying data from this chart as JSON with fields:
chart_type, title, x_axis {label, unit}, y_axis {label, unit, min, max},
series: [{name, points: [{x, y}]}], notes.
Read values from data labels where present. Where you must estimate a value
from bar or line positions, add "estimated": true to that point.
Do not round labeled values. If the y axis is truncated or log-scaled,
say so in notes.The estimated flag matters: downstream you can require that any number quoted in an answer comes from a non-estimated point, or that the original image gets attached for verification. The axis-range fields catch the two classic chart traps, truncated axes and log scales, that make naive visual reads wrong.
Whatever you extract, keep the chart image and attach it at generation time for high-stakes numeric answers. The JSON gets you retrieval and speed; the pixels get you a second opinion from a stronger model at the moment it counts.
Generation: pass the pixels back to the model
Retrieval gives you a mixed bag of text chunks, table markdown and image pointers. Assemble them into one multimodal prompt, with images inline and page numbers attached so the model can cite:
def answer(question: str) -> str:
hits = col.query(query_texts=[question], n_results=6)
content = []
for meta, summary in zip(hits["metadatas"][0], hits["documents"][0]):
if meta["kind"] == "figure":
with open(meta["asset"], "rb") as f:
data = base64.standard_b64encode(f.read()).decode()
content.append({"type": "image",
"source": {"type": "base64",
"media_type": "image/png", "data": data}})
content.append({"type": "text",
"text": f"(the figure above is from page {meta['page']})"})
elif meta["kind"] == "table":
content.append({"type": "text",
"text": f"Table from page {meta['page']}:\n{meta['raw']}"})
else:
content.append({"type": "text", "text": summary})
content.append({"type": "text", "text":
"Answer using only the context above. Cite the page number for every "
"number you state. If the answer is not in the context, say so.\n\n"
f"Question: {question}"})
msg = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=800,
messages=[{"role": "user", "content": content}],
)
return msg.content[0].textKeep the image count low: attach only the figures that actually ranked, not every image on every retrieved page, because image tokens are expensive and attention over many images dilutes. The same assembly works for architecture 3, except the images are whole page screenshots (byaldi returns them as base64 when you index with store_collection_with_index=True).
Evaluating a multimodal RAG pipeline
You cannot tune what you do not measure, and text-only eval sets are exactly how multimodal regressions hide. Build a set of 40 to 60 questions from real user traffic and label each with the gold page, the gold element type (prose, table cell, chart value, diagram) and the gold value where numeric.
Measure two layers separately:
- Retrieval: hit rate at k and NDCG at 5 (the metric the ViDoRe benchmark standardizes on) against the gold page or element. Slice the results by element type. A pipeline can score 90 percent on prose questions and 20 percent on chart questions, and the blended number will look fine. The slices are the whole point.
- Generation: exact numeric match against the gold value, with a small tolerance for legitimately estimated chart reads, plus a citation check that the stated page actually contains the value.
Use ViDoRe scores for shopping between retrieval models, and your own set for the final call, because your documents never look like the benchmark. Re-run the eval when you swap parsers, not just models: a Docling version change that alters table extraction is a retrieval change in disguise.
Choosing your multimodal RAG architecture
The decision usually falls out of the corpus and the team:
- Digital-born PDFs, an existing text RAG stack, and mostly table and figure questions: start with architecture 1. Lowest risk, best debuggability, one ingest-time vision call per element.
- Scanned documents, forms, invoices, handwriting, or layout-hostile archives: go straight to ColPali or ColQwen2. Parsing is the bottleneck you get to skip, and OCR errors stop existing as a category.
- Visual similarity search, "find the slide that looks like this", product image search: architecture 2 is the only one that natively does image-as-query.
- High-stakes numeric answers from tables in any architecture: extract tables as markdown and pass them as text to the generator, always.
- Hybrids are normal, not a smell: BM25 or pooled-vector first stage with ColPali rescoring, captions for tables plus page images for figures, JSON chart extraction plus attached pixels.
The through-line across all three: retrieval returns pointers, generation gets payloads. Keep original assets addressable by ID from day one and you can change retrieval strategies later without re-ingesting the world.
Start small this week: one document, architecture 1, twenty questions with labeled answer locations. That afternoon of work tells you whether captions are enough for your corpus or whether the eval is pointing you at page-image retrieval, and either way you will have the harness you need to prove it.
FAQ
Do I need a vision-capable LLM at query time? Only if you pass images to the generator. Architecture 1 can run with a text-only generator reading captions and table markdown, which is cheaper and often fine for prose-plus-tables corpora. The moment exact chart values matter, attach the image and use a vision model; captions alone plateau on numeric fidelity.
Is multimodal RAG still worth it now that context windows fit whole PDFs? For a handful of documents, stuffing the PDF into a long-context vision model is a legitimate baseline and you should measure against it. Retrieval wins on corpora: cost and latency scale with what you retrieve rather than what you own, and page-level citations come for free. Most teams land on retrieval for the corpus plus long context for the final assembled prompt.
How do I handle scanned PDFs? Either run OCR in the parser (Docling supports Tesseract, EasyOCR and RapidOCR) and continue with architecture 1, or skip OCR entirely with ColPali-family retrieval. On degraded scans the page-image approach usually wins, because OCR errors compound: a misread header corrupts every caption and chunk built on top of it.
Can I store ColPali embeddings in pgvector? Not idiomatically. Late interaction needs a MaxSim comparator over multivectors, which pgvector does not provide natively; you would be reimplementing scoring in SQL or application code. Qdrant and Vespa support late interaction natively, and byaldi is fine for prototypes. Check for a multivector or MaxSim feature before committing to a store.
Which model should do the captioning? The small vision tier of any current family is usually enough at ingest: Claude Haiku 4.5 in the examples above, or the equivalent small tier from other providers. Spend your quality budget on the generator and on the captioning prompt instead. The one exception is dense scientific figures, where a mid-tier captioner measurably reduces hallucinated axis values; let your eval slices make that call.
How large should figure crops be? Large enough that you can read the axis labels yourself when zoomed to 100 percent; render at 2x scale as in the Docling example. If the captioner cannot read a label it will guess, the guess goes into your index, and you will not find out until a user quotes it back to you.
BootcampA 30-day guided bootcamp: build, harden and ship a production autonomous agent from scratch.
AI AgentsUnderstand how AI agents really work: the loop, the tools, the memory, and why most agent projects fail.