Building Multimodal LLM Applications: A Practical Guide

A multimodal llm accepts more than one input type, usually text plus images, and increasingly documents, audio, and video, then reasons across all of them in a single context window. Building a production multimodal llm application means more than swapping a text prompt for an image: you have to think about encoding, token cost, grounding, and failure modes that text-only systems never hit. This guide walks through the architecture, code, and evaluation practices you need to ship one.

What makes a multimodal LLM different from a text-only pipeline

A text-only LLM application is a straight line: prompt in, tokens out. A multimodal llm application has more moving parts before the model ever sees a request. Images and PDFs need to be resized, compressed, or split into pages. Audio needs to be transcribed or passed as raw waveform chunks depending on the provider. Video needs to be sampled into frames or handled by a model with native video support.

The core difference is that visual and audio inputs consume tokens too, often a lot of them. A single high-resolution image can cost as many tokens as several paragraphs of text. If your application sends a dozen screenshots per request, your context budget disappears fast, and your latency climbs with it. This is the first design constraint every multimodal llm project runs into: token cost is not just a text-length problem anymore, it is a pixel-count and duration problem.

The second difference is grounding. When a model describes what's in an image or extracts a number from a chart, you need a way to check that it isn't hallucinating a detail that was never there. Text-only hallucination is bad enough; visual hallucination (a model confidently describing a bar in a chart that doesn't exist) is harder to catch because a human reviewer has to look at the image side by side with the output.

Choosing an architecture: native multimodal vs. pipeline

There are two broad ways to build a multimodal llm application today.

Native multimodal model. You send images, text, and sometimes documents directly to a single model call. The model itself was trained to understand pixels alongside tokens. This is the simplest architecture and the right default for most applications: fewer moving parts, no separate vision model to maintain, and the model can reason jointly across modalities (for example, "does the chart in this image match the number in this paragraph?").

Pipeline architecture. You run a specialized model first (OCR, an object detector, a speech-to-text model, a frame sampler) and feed structured output into a text-only LLM. This adds latency and moving parts, but it's still the right call in three situations:

• You need deterministic extraction, like exact bounding boxes or a fixed OCR confidence score, that a generative model won't reliably give you.
• Your documents are extremely long (hundreds of pages) and you need to pre-filter before spending multimodal tokens on the whole thing.
• You're on a strict latency budget and a lightweight specialized model is faster than a general-purpose multimodal llm call.

Most teams start native and only add pipeline steps where they hit a specific accuracy or cost wall. Don't build the pipeline architecture speculatively; it's harder to debug and harder to change later.

Sending images to a multimodal LLM

Almost every multimodal llm API accepts images as base64-encoded data or as a URL reference. Here's a minimal pattern using the Anthropic Python SDK, which generalizes to other providers with small syntax changes:

import base64
import anthropic

client = anthropic.Anthropic()

def encode_image(path):
    with open(path, "rb") as f:
        return base64.standard_b64encode(f.read()).decode("utf-8")

image_data = encode_image("invoice.png")

response = client.messages.create(
    model="claude-sonnet",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": image_data,
                    },
                },
                {
                    "type": "text",
                    "text": "Extract the vendor name, invoice number, and total due. Return JSON only.",
                },
            ],
        }
    ],
)

print(response.content[0].text)

A few practical rules that save you real money and latency:

• Resize before you encode. Most models cap useful resolution well below what a phone camera produces; sending a 12-megapixel photo when the model downsamples to a fraction of that just burns bandwidth and tokens.
• Crop when you can. If you only need the top third of a screenshot, crop it client-side instead of asking the model to "look at the top of the image."
• Batch multiple images in one call only when the task genuinely needs cross-image comparison. Otherwise, parallelize independent single-image calls; it's easier to retry a failed one without redoing the whole batch.
• Always specify the exact output format you want (JSON schema, a fixed set of fields) in the prompt. Multimodal llm outputs drift more than text-only outputs when the instructions are vague, because the model is also busy describing what it sees.

Document and PDF understanding

Documents are the most common real-world multimodal llm use case: invoices, contracts, resumes, scanned forms. Two approaches dominate.

Direct document input. Send the PDF (or page images) straight to the model and ask it to extract structured data. This works well for documents under roughly 50-100 pages and is the simplest path.

import anthropic

client = anthropic.Anthropic()

with open("contract.pdf", "rb") as f:
    pdf_data = f.read()

import base64
encoded = base64.standard_b64encode(pdf_data).decode("utf-8")

response = client.messages.create(
    model="claude-sonnet",
    max_tokens=2048,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "document",
                    "source": {
                        "type": "base64",
                        "media_type": "application/pdf",
                        "data": encoded,
                    },
                },
                {
                    "type": "text",
                    "text": "List every clause that mentions a termination date. Cite the page number for each.",
                },
            ],
        }
    ],
)

print(response.content[0].text)

Asking the model to cite a page number or a quoted snippet for every extracted fact is one of the highest-leverage prompt patterns in document-heavy multimodal llm work. It gives you a cheap grounding check: if the cited page doesn't actually contain the quoted text, you know the extraction is unreliable and can flag it for review instead of shipping it silently.

Chunk-and-retrieve for long documents. For anything past a few hundred pages, split the document into page ranges, run a cheap first pass to find which chunks are relevant to the question, then send only those chunks through the full multimodal llm call. This is the same retrieval-augmented pattern you'd use for text, just with page images or PDF chunks as the retrieved unit instead of text passages.

Audio and video inputs

Audio and video are handled less uniformly across providers than images. Some multimodal llm APIs accept audio natively; others expect you to transcribe first with a dedicated speech-to-text model and pass the transcript as text, optionally alongside a few sampled frames for video context.

A practical, provider-agnostic pattern for video:

import cv2

def sample_frames(video_path, every_n_seconds=5):
    cap = cv2.VideoCapture(video_path)
    fps = cap.get(cv2.CAP_PROP_FPS)
    frame_interval = int(fps * every_n_seconds)
    frames = []
    count = 0
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break
        if count % frame_interval == 0:
            frames.append(frame)
        count += 1
    cap.release()
    return frames

Sample frames at a fixed interval, encode each as an image, and send them as an ordered sequence alongside a transcript if you have one. Tell the model explicitly that the images are sequential frames from a video and roughly what time each one corresponds to; without that context, the model treats them as unrelated stills and loses the temporal thread.

For audio-heavy applications (call transcription, meeting summarization), transcribe first with a dedicated speech model, then let the multimodal llm (or even a text-only model at that point) do the reasoning over the transcript. Reserve native audio input for cases where tone, emphasis, or non-speech sound genuinely matters to the task, since it costs more than a transcript-based approach.

Combining multimodal input with tool use

The most useful multimodal llm applications aren't single-shot extractors, they're agents that look at something, decide what to do, and call a tool. A common pattern: a support agent that receives a screenshot from a user, diagnoses the issue, and looks up the relevant help article.

tools = [
    {
        "name": "search_help_docs",
        "description": "Search the internal help documentation for a topic",
        "input_schema": {
            "type": "object",
            "properties": {
                "query": {"type": "string"}
            },
            "required": ["query"],
        },
    }
]

response = client.messages.create(
    model="claude-sonnet",
    max_tokens=1024,
    tools=tools,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": image_data,
                    },
                },
                {
                    "type": "text",
                    "text": "The user sent this screenshot of an error. Diagnose it and search the docs for a fix.",
                },
            ],
        }
    ],
)

Nothing about the tool-calling loop changes because the input includes an image; the model still emits a tool-use block, you run the tool, and you feed the result back in the next turn. The only thing to watch is context growth: if your agent loop keeps every prior image in the conversation history, you'll blow through your context window fast. Drop images from history once the model has extracted what it needs from them, and keep only the text summary going forward.

Evaluating multimodal LLM output

Text-only evaluation (exact match, embeddings similarity, LLM-as-judge) mostly still applies, but you need two additions specific to multimodal work.

Grounding checks. For any extracted claim, verify it against the source image or document programmatically where possible. If the model claims a total of a specific amount, regex-check that the number actually appears somewhere in the OCR text of the document. If it doesn't, that's a strong hallucination signal worth flagging.

Golden sets with visual diversity. Your evaluation set needs to cover the actual visual variance your users will send: different resolutions, rotated scans, low-light photos, screenshots with different UI themes. A model that scores well on clean, high-resolution test images can fail badly on the blurry phone photo a real user actually sends. Build the eval set from real production inputs as early as possible, not synthetic clean examples.

A simple harness pattern:

import json

def run_eval(cases, run_fn):
    results = []
    for case in cases:
        output = run_fn(case["image_path"], case["prompt"])
        correct = case["expected"].lower() in output.lower()
        results.append({
            "id": case["id"],
            "correct": correct,
            "output": output,
        })
    accuracy = sum(r["correct"] for r in results) / len(results)
    print(f"Accuracy: {accuracy:.2%}")
    return results

Keep this harness simple at first. The value is in the diversity of the cases list, not the sophistication of the scoring function.

Cost and latency, and how to control them

Multimodal llm calls are more expensive per request than text-only calls because of image and document token overhead, and they're often slower because larger payloads take longer to upload and process. Three levers control this in practice:

1. Downsample aggressively. Test the smallest image resolution that still gives you acceptable accuracy for your task. For most document extraction and screenshot-diagnosis tasks, this is much lower than the original capture resolution.
2. Route by complexity. Use a smaller or faster model for simple classification tasks ("does this image contain a signature?") and reserve a larger model for tasks that need deeper reasoning ("does the signature match the name on the form?"). Not every multimodal call needs your top-tier model.
3. Cache repeated context. If the same reference image or document is sent across many requests (a product catalog image reused for every customer question about that product), use prompt caching where your provider supports it instead of re-uploading and re-processing the same bytes every call.

Common pitfalls

• Sending full-resolution images by default. This is the single most common source of runaway multimodal llm costs. Set a max dimension and enforce it in your upload pipeline, not as an afterthought.
• Trusting extracted numbers without a grounding check. Vision models are good at reading text but can still transpose digits or round numbers under ambiguity. Always validate extracted numeric fields against the source when the stakes are financial or legal.
• Treating video as "just more images." Without explicit framing about sequence and timing, models will describe frames independently instead of reasoning about what changed between them.
• Skipping a fallback path. Image quality varies wildly in production (blur, glare, cropped edges). Build a low-confidence branch that routes to human review instead of silently accepting a shaky extraction.
• Forgetting to prune images from long-running conversations. Agent loops that never drop old image content will hit context limits and slow down well before text-only agents would.

FAQ

What is a multimodal LLM? A multimodal llm is a model trained to process and reason over more than one input type in the same context, most commonly text and images, and increasingly documents, audio, and video, without needing a separate model for each modality.

Do I need a separate vision model, or can one multimodal LLM handle everything? For most applications, a single native multimodal model is sufficient and simpler to maintain. Add a specialized pipeline step (OCR, object detection, speech-to-text) only when you hit a specific accuracy, determinism, or latency requirement the general model can't meet.

How do I reduce the cost of image-heavy multimodal LLM calls? Downsample images to the smallest resolution that preserves task accuracy, crop to the relevant region before sending, cache repeated reference content, and route simple classification tasks to smaller or faster models instead of your top-tier one.

How should I handle very long PDFs with a multimodal LLM? Split the document into page ranges, run a cheap relevance pass to identify which chunks matter for the question, and send only those chunks through the full multimodal call, similar to retrieval-augmented generation for text.

How do I check if a multimodal LLM is hallucinating details from an image? Ask the model to cite a specific region, page number, or quoted snippet for every claim, then programmatically verify that citation against the source (for example, checking that an extracted number actually appears in the document's OCR text). Build evaluation sets from real, messy production images rather than clean synthetic ones.

Can I use a multimodal LLM inside an agent that calls tools? Yes. The tool-calling loop is unchanged by the presence of image or document input; the model still emits tool-use blocks you execute and feed back. The main thing to manage is context growth: drop images from conversation history once their information has been extracted, keeping only text summaries for later turns.