AI Agentsmultimodaltool usevisionagent architecture

Handling Multimodal Inputs in AI Agents

Pramod Dutta · Jul 9, 2026 · 10 min read

Multimodal agents are AI agents that can read and reason over images, audio, documents, and screenshots alongside text, then decide what to do next. If you have only ever built agents that read and write plain text, adding multimodal inputs touches almost every part of your stack: message formatting, context budgeting, tool schemas, and error handling. This article walks through the practical parts of that transition: how to structure multimodal messages, how to keep an agent loop stable when a tool call returns an image instead of a string, and how to avoid the token and cost traps that catch people the first time they wire a screenshot into a loop.

Why multimodal agents are different from multimodal chat

A chat app that accepts an image is a single request-response pair: user uploads a photo, model describes it, done. An agent is a loop. It calls tools, gets results, decides whether to call more tools, and eventually produces a final answer. The moment one of those tool results is an image (a screenshot from a browser automation tool, a chart rendered by a code execution tool, a page from a scanned PDF), you have to answer questions a single-turn chat app never has to answer:

Does this image go back into the model's context on the next turn, or do you summarize it into text and drop the pixels?
How many images can accumulate in a long-running agent session before you blow the context window or the per-request size limit?
What happens when a tool that used to return a string now sometimes returns an image, and your loop's serialization code assumes strings?

None of this is exotic engineering, but it is easy to skip during a text-only prototype and then hit all three problems at once when you add a "take a screenshot" or "read this PDF" tool.

Structuring multimodal messages

Most current model APIs, including Claude's Messages API, represent multimodal content as a list of content blocks inside a single message, rather than a separate field for "the image." A user or tool-result message can mix text blocks and image blocks in one array:

{
  "role": "user",
  "content": [
    { "type": "text", "text": "What's wrong with this chart?" },
    { "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": "<...>" } }
  ]
}

The important design decision for an agent is where this content block gets created. There are two common patterns:

User-supplied multimodal input. The user attaches an image or PDF up front. This is straightforward: you build the content block list once, before the agent loop starts.
Tool-generated multimodal input. A tool call (browser screenshot, chart renderer, camera capture, document scanner) produces an image mid-loop. This is the harder case, because the image has to be inserted as a tool_result content block, and your loop code needs to treat tool results as potentially multimodal from the start rather than bolting it on later.

Design your tool_result handling to accept a list of content blocks, not a single string, even for tools you know today will only ever return text. That one decision saves a rewrite later.

def make_tool_result(tool_use_id, blocks):
    # blocks is always a list of content blocks: text, image, or a mix
    return {
        "type": "tool_result",
        "tool_use_id": tool_use_id,
        "content": blocks,
    }

# text-only tool, wrapped consistently
def wrap_text(s: str):
    return [{"type": "text", "text": s}]

# screenshot tool
def wrap_screenshot(png_bytes: bytes):
    import base64
    b64 = base64.b64encode(png_bytes).decode("utf-8")
    return [{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}}]

Handling PDFs and documents

PDFs are a special case worth calling out separately from images, because a PDF is not one image, it is a stack of pages, and some of those pages may be scanned images while others are native text. Two approaches work in practice:

Native PDF support, where the API accepts a PDF document block directly and the model handles page extraction and layout internally. This is the simplest path when your provider supports it, and it preserves layout information (tables, columns) that a naive text extraction would lose.
Pre-processing into images or text, where you render each page to an image (for scanned or layout-heavy documents) or extract text with a library like pypdf or pdfplumber (for text-native documents), then feed the result as ordinary content blocks.

Pick pre-processing when you need fine control, such as sending only the three relevant pages of a fifty-page contract instead of the whole document, which matters a lot for cost and context budget on long documents.

import fitz  # PyMuPDF

def pdf_pages_to_images(path: str, page_numbers: list[int], zoom: float = 2.0):
    doc = fitz.open(path)
    blocks = []
    for n in page_numbers:
        page = doc[n]
        pix = page.get_pixmap(matrix=fitz.Matrix(zoom, zoom))
        png_bytes = pix.tobytes("png")
        blocks.extend(wrap_screenshot(png_bytes))
    return blocks

Handling audio

Audio input in an agent typically enters the loop in one of two ways: transcribed before the agent sees it, or passed natively if the model supports audio input directly. Transcription-first is still the more common and more debuggable pattern as of 2026, because it lets you log, search, and diff what the agent actually "heard" as plain text, and it decouples your agent's model choice from whether that specific model supports audio.

def transcribe_and_wrap(audio_path: str, transcribe_fn):
    text = transcribe_fn(audio_path)  # e.g. a speech-to-text API call
    return wrap_text(f"[transcribed audio]\n{text}")

If you do use native audio input, treat it like an image for context budgeting purposes: it counts against your token or size limits, and long recordings should be chunked or summarized rather than replayed in full on every turn of a multi-turn agent session.

Managing context and cost with images in the loop

This is where multimodal agents quietly get expensive or quietly break. Images consume meaningfully more tokens than an equivalent amount of text, and an agent that takes a screenshot every step (think a browser-automation or computer-use agent) will accumulate images fast.

Three techniques keep this under control:

Cap the image history. Keep only the last N images in context and drop or summarize older ones into text descriptions. A browser agent rarely needs to see the screenshot from ten steps ago; a one-line text summary ("logged in, on dashboard page") is enough.
Downscale before sending. Resize screenshots and photos to the smallest resolution that still lets the model read the relevant text or detail. A 4K screenshot of a web page rarely needs to stay 4K; a 1280px-wide version usually reads fine and costs a fraction of the tokens.
Summarize instead of replay. When a tool call produces an image purely to inform the next decision (e.g., "did the form submit successfully?"), have the agent extract a short text verdict from the image on that turn, store the text, and discard the image from the running context rather than keeping the raw image around for every subsequent turn.

MAX_IMAGES_IN_CONTEXT = 3

def prune_old_images(messages):
    image_positions = [
        i for i, m in enumerate(messages)
        if isinstance(m.get("content"), list)
        and any(b.get("type") == "image" for b in m["content"])
    ]
    to_drop = image_positions[:-MAX_IMAGES_IN_CONTEXT] if len(image_positions) > MAX_IMAGES_IN_CONTEXT else []
    for i in to_drop:
        messages[i]["content"] = [
            b if b.get("type") != "image" else {"type": "text", "text": "[older image omitted]"}
            for b in messages[i]["content"]
        ]
    return messages

Tool schemas for multimodal agents

Multimodal agents also need tool definitions that describe what kind of media a tool consumes and returns, so the model can decide correctly when to call it. If you're using function-calling style tool definitions, keep the schema honest about input types:

{
  "name": "read_pdf_page",
  "description": "Extract text and layout from a specific page of a PDF document. Returns the page as an image if it is scanned, or as text if it is native.",
  "input_schema": {
    "type": "object",
    "properties": {
      "file_path": { "type": "string" },
      "page_number": { "type": "integer" }
    },
    "required": ["file_path", "page_number"]
  }
}

Describing the return shape in plain language in the description field matters more than it looks. Models plan their next steps partly from tool descriptions, and an agent that expects a tool to "return text" will sometimes mishandle the case where it gets an image back, especially in loops that log or diff tool outputs as strings.

Error handling for multimodal tool calls

Multimodal tools fail in ways text tools don't: unsupported file formats, corrupted images, PDFs with no extractable text, screenshots taken of a blank or loading page. Build these into your tool result contract from day one rather than treating them as edge cases:

def safe_screenshot(capture_fn):
    try:
        png_bytes = capture_fn()
        if not png_bytes or len(png_bytes) < 100:
            return wrap_text("Screenshot capture returned no usable image; page may still be loading.")
        return wrap_screenshot(png_bytes)
    except Exception as e:
        return wrap_text(f"Screenshot failed: {e}")

Returning a text explanation instead of silently failing lets the agent reason about the failure ("the page hasn't loaded, I should wait and retry") instead of getting an empty or malformed content block that breaks the next API call.

A minimal multimodal agent loop

Putting the pieces together, here is the shape of a loop that stays agnostic to whether a given tool result is text or an image, prunes old images, and keeps the message list valid across turns:

def run_agent(client, model, system_prompt, tools, tool_fns, user_content_blocks, max_turns=15):
    messages = [{"role": "user", "content": user_content_blocks}]

    for _ in range(max_turns):
        messages = prune_old_images(messages)
        response = client.messages.create(
            model=model,
            system=system_prompt,
            tools=tools,
            messages=messages,
            max_tokens=2048,
        )
        messages.append({"role": "assistant", "content": response.content})

        tool_calls = [b for b in response.content if b.type == "tool_use"]
        if not tool_calls:
            return response  # final answer, no more tool calls

        tool_results = []
        for call in tool_calls:
            blocks = tool_fns[call.name](**call.input)
            tool_results.append(make_tool_result(call.id, blocks))
        messages.append({"role": "user", "content": tool_results})

    raise RuntimeError("Agent did not finish within max_turns")

The key property here is that every tool result, image or text, goes through the same make_tool_result wrapper, and every image-producing helper (wrap_screenshot, pdf_pages_to_images) returns the same content-block shape. That consistency is what lets you add a new multimodal tool later without touching the loop itself.

FAQ

Do I need a vision-capable model for every agent that touches images? Only the calls that actually need to interpret an image need a vision-capable model. If a tool converts an image to text before it reaches the model (OCR, a caption, a transcription), the rest of your loop can run on a text-only model. Mixing models by task is a valid way to control cost.

How many images can I keep in an agent's context before it becomes a problem? There's no universal number since it depends on image resolution and your model's context window, but the practical answer is: fewer than you think. Cap it explicitly (see the pruning example above) rather than letting it grow until you hit a limit in production.

Should I resize images before sending them to the model? Yes, in almost all cases. Send the smallest resolution that preserves the detail the task needs. A screenshot used to check "is there an error banner visible" needs far less resolution than one used to read small print in a table.

What's the difference between sending a PDF directly versus converting it to images first? Native PDF support (where the API accepts the document directly) is simpler and preserves layout, but gives you less control over per-page cost. Converting to images first costs you more upfront engineering but lets you select specific pages and control resolution, which matters for long documents.

How do I test a multimodal agent's tool-handling logic without calling a real model on every run? Keep your wrap_* and make_tool_result helpers pure functions that take bytes or text and return content blocks, and unit test those directly. Save a handful of real API responses (with tool_use blocks) as fixtures, and replay them through your loop's message-building code without a live model call, only doing full end-to-end runs against the real API in a smaller, separate test suite.

Can I use the same agent loop for images, audio, and PDFs, or do I need separate loops? One loop, many tools. The loop itself should not know or care what kind of content a tool returns; it should just pass content blocks through. Keep the media-specific logic (transcription, PDF rendering, image resizing) inside the tool functions, not the loop.