Building Agents with LlamaIndex

LlamaIndex agents let you wrap tools, query engines, and memory into a single object that decides what to call and when, instead of you hardcoding the control flow yourself. If you already have a LlamaIndex index for retrieval-augmented generation, turning it into an agent that can search your documents, call external functions, and hold a conversation takes only a few extra lines of code. This guide walks through the full path: setting up the environment, building a basic function-calling agent, wiring in a RAG query engine as a tool, adding memory, and composing multiple agents into a workflow.

What a LlamaIndex agent actually is

A LlamaIndex agent is a loop around an LLM that can call tools. You give it a list of FunctionTool or QueryEngineTool objects, the agent's LLM reads the user's request, decides which tool (if any) it needs, calls it, reads the result, and either calls another tool or returns a final answer. This is the same "reason, act, observe" pattern you'll recognize from ReAct-style agents, but LlamaIndex's newer FunctionAgent and AgentWorkflow classes lean on native function calling in the underlying model rather than parsing free text, which makes tool selection far more reliable.

The three building blocks you'll use in almost every LlamaIndex agent project are:

• Tools: Python functions wrapped as FunctionTool, or a RAG query engine wrapped as QueryEngineTool.
• An LLM: any chat model LlamaIndex supports (OpenAI, Anthropic, local models through Ollama, etc).
• An agent runtime: FunctionAgent for a single agent, or AgentWorkflow when you want several agents handing off tasks to each other.

Setting up the environment

Install the core package plus whichever LLM integration you plan to use. This example uses Anthropic's Claude models, but the same code works with any provider LlamaIndex supports, you just swap the import and the class name.

pip install llama-index llama-index-llms-anthropic llama-index-embeddings-openai

Set your API key as an environment variable rather than pasting it into code:

export ANTHROPIC_API_KEY="your-key-here"

Configure the default LLM once at the top of your script so every agent and query engine you create picks it up automatically:

from llama_index.core import Settings
from llama_index.llms.anthropic import Anthropic

Settings.llm = Anthropic(model="claude-sonnet-4-5")

Building your first function-calling agent

Start with plain Python functions. LlamaIndex reads the function signature and docstring to figure out what the tool does and what arguments it expects, so write clear docstrings, they double as the tool description the LLM sees.

from llama_index.core.tools import FunctionTool
from llama_index.core.agent.workflow import FunctionAgent

def get_weather(city: str) -> str:
    """Get the current weather for a given city name."""
    # In a real app, call a weather API here
    fake_data = {"london": "14C, light rain", "delhi": "34C, clear"}
    return fake_data.get(city.lower(), "No data for that city")

def convert_currency(amount: float, from_currency: str, to_currency: str) -> str:
    """Convert an amount from one currency code to another, e.g. USD to INR."""
    rates = {"usd_inr": 83.2, "usd_gbp": 0.79}
    key = f"{from_currency.lower()}_{to_currency.lower()}"
    if key not in rates:
        return "Conversion rate not available"
    return f"{amount} {from_currency.upper()} = {amount * rates[key]:.2f} {to_currency.upper()}"

weather_tool = FunctionTool.from_defaults(fn=get_weather)
currency_tool = FunctionTool.from_defaults(fn=convert_currency)

agent = FunctionAgent(
    tools=[weather_tool, currency_tool],
    llm=Settings.llm,
    system_prompt="You are a helpful travel assistant. Use tools when you need live data.",
)

FunctionAgent is async-first, so you run it with await inside an async function. If you're working in a plain script, wrap the call with asyncio.run:

import asyncio

async def main():
    response = await agent.run("What's the weather in Delhi, and what's 100 USD in INR?")
    print(response)

asyncio.run(main())

Run this and the agent will call both get_weather and convert_currency in the same turn, then merge the results into one natural-language answer. You didn't write any branching logic, the agent inferred from the docstrings and the user's question which tools to invoke.

Adding a RAG query engine as a tool

Most real agent projects need to answer questions from your own documents, not just call functions. LlamaIndex makes this easy because a query engine can be wrapped as a tool exactly like a Python function.

First, build a standard vector index over a folder of documents:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.tools import QueryEngineTool

documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(similarity_top_k=5)

docs_tool = QueryEngineTool.from_defaults(
    query_engine=query_engine,
    name="company_docs",
    description="Answers questions about internal company policy and product documentation.",
)

Now give the agent both the RAG tool and a plain function tool. This is the pattern most production LlamaIndex agents use: retrieval for grounded facts, function calls for live actions.

def create_support_ticket(summary: str, priority: str) -> str:
    """Create a support ticket with a short summary and a priority of low, medium, or high."""
    return f"Ticket created: '{summary}' with priority {priority}"

ticket_tool = FunctionTool.from_defaults(fn=create_support_ticket)

support_agent = FunctionAgent(
    tools=[docs_tool, ticket_tool],
    llm=Settings.llm,
    system_prompt=(
        "You are a support assistant. Answer questions using company_docs when relevant. "
        "If the user reports a bug or issue you cannot resolve, create a support ticket."
    ),
)

Ask it something like "What's our refund policy, and also log a ticket because a customer says refunds aren't processing", and the agent will query the index for the policy text, then separately call create_support_ticket, both from a single user message.

Giving the agent memory

By default each agent.run() call is stateless. For a multi-turn conversation, attach a Context object and reuse it across calls so the agent remembers earlier turns.

from llama_index.core.workflow import Context

ctx = Context(agent)

async def chat_loop():
    while True:
        user_input = input("You: ")
        if user_input.lower() in ("exit", "quit"):
            break
        response = await agent.run(user_input, ctx=ctx)
        print("Agent:", response)

asyncio.run(chat_loop())

Passing the same ctx on every call means the agent's internal chat history, and any state it wrote during tool calls, carries forward. This is the difference between a one-shot tool call and an actual assistant that remembers what city you asked about three messages ago.

Streaming responses

For a chat UI you almost always want to stream tokens rather than wait for the full response. FunctionAgent.run() returns a handler you can iterate over for events, including partial text deltas:

from llama_index.core.agent.workflow import AgentStream

async def stream_chat(question: str):
    handler = agent.run(question, ctx=ctx)
    async for event in handler.stream_events():
        if isinstance(event, AgentStream):
            print(event.delta, end="", flush=True)
    final_response = await handler
    return final_response

asyncio.run(stream_chat("Summarize the last conversation"))

Streaming events also let you show the user which tool is currently running, which matters a lot for trust when an agent is doing multi-step reasoning that takes several seconds.

Composing multiple agents with AgentWorkflow

Once you have more than one specialized agent, AgentWorkflow lets you chain them with explicit handoff rules instead of cramming every tool into one giant system prompt. This keeps each agent's tool list small, which improves tool-selection accuracy.

from llama_index.core.agent.workflow import AgentWorkflow

research_agent = FunctionAgent(
    name="researcher",
    description="Searches company docs and summarizes findings.",
    tools=[docs_tool],
    llm=Settings.llm,
    system_prompt="You research questions using company_docs and hand off to the writer once you have enough context.",
    can_handoff_to=["writer"],
)

writer_agent = FunctionAgent(
    name="writer",
    description="Writes the final answer for the user based on research notes.",
    tools=[],
    llm=Settings.llm,
    system_prompt="You write a clear, concise final answer based on what the researcher found.",
)

workflow = AgentWorkflow(
    agents=[research_agent, writer_agent],
    root_agent="researcher",
)

async def run_workflow():
    result = await workflow.run(user_msg="What's our data retention policy, explained simply?")
    print(result)

asyncio.run(run_workflow())

The researcher agent pulls facts from the query engine, then hands off to writer, which turns those notes into a clean answer. can_handoff_to is what makes the routing explicit and inspectable, rather than relying on one enormous prompt to juggle every responsibility.

Debugging tool calls

When an agent picks the wrong tool or hallucinates arguments, the fastest fix is almost always a better docstring or a stricter type hint, not a bigger model. Turn on LlamaIndex's callback handler to see the raw tool calls and arguments the LLM is generating:

from llama_index.core import set_global_handler

set_global_handler("simple")

This prints each LLM call, tool invocation, and tool output to the console. Two things to check when a tool call goes wrong:

• Argument types: if your function expects city: str but the LLM passes a list or a badly formatted string, tighten the docstring with an example, e.g. "e.g. 'Delhi' or 'New York'".
• Overlapping tool descriptions: if two tools sound similar, the agent may call the wrong one. Make descriptions mutually exclusive rather than both saying "answers questions about X".

Handling errors from tool calls gracefully

A tool that raises an exception will crash the agent run unless you catch it inside the function. Always return a string describing the failure instead of letting an exception propagate, so the agent can explain the problem to the user or try a different approach.

def lookup_order(order_id: str) -> str:
    """Look up the status of an order by its order ID."""
    try:
        # pretend this calls a real database
        if not order_id.isdigit():
            raise ValueError("Order ID must be numeric")
        return f"Order {order_id} is out for delivery"
    except Exception as e:
        return f"Could not look up order: {e}"

Because the agent sees the tool's return value as an observation, a clear error string lets it recover, for example by asking the user to double check the order ID, rather than failing the whole conversation.

Production considerations

A few things worth deciding before you ship a LlamaIndex agent:

• Timeouts: wrap agent.run() calls with asyncio.wait_for so a stuck tool call (a slow API, a hung database query) doesn't block the whole request indefinitely.
• Tool allowlists per user: if different users should have access to different tools (e.g. only admins can call a delete function), build separate agent instances rather than trying to gate access inside a single shared agent.
• Cost and latency: every tool call round-trip is another LLM call. Keep tool lists focused, and prefer AgentWorkflow with specialized agents over one agent holding fifteen tools, both for accuracy and for keeping each individual call's prompt smaller.
• Observability: log the full sequence of tool calls and arguments for each conversation. When something goes wrong in production, this trace is what tells you whether the retrieval was bad, the tool arguments were wrong, or the final synthesis step dropped information.

FAQ

What's the difference between a LlamaIndex agent and a plain RAG query engine? A query engine answers a question by retrieving relevant chunks and generating a response in one pass. An agent can call a query engine as one of several tools, decide whether retrieval is even necessary, call other functions, and combine multiple results before answering, so it handles multi-step and multi-source requests a plain query engine can't.

Do I need `AgentWorkflow`, or is a single `FunctionAgent` enough? A single FunctionAgent is enough for most projects, especially if you have fewer than about ten tools. Reach for AgentWorkflow once you have distinct responsibilities (research versus writing, or support versus billing) that benefit from separate system prompts and smaller, focused tool lists.

Can I use LlamaIndex agents with open-source or local models? Yes. LlamaIndex has LLM integrations for Ollama and other local runtimes. Function calling reliability varies by model though, smaller local models sometimes need a ReAct-style agent (ReActAgent in LlamaIndex) instead of the native function-calling FunctionAgent, since they don't support structured tool calls as consistently as larger hosted models.

How do I stop an agent from calling a tool it doesn't need? Tighten the tool's description to state exactly when it should and shouldn't be used, and keep the tool list short. Agents choose tools by matching the user's intent against tool descriptions, so vague descriptions like "gets data" invite unnecessary calls, while specific ones like "only use this to look up an existing order by numeric ID" cut down on misuse.

Is `FunctionAgent` the same as the older `OpenAIAgent` or `ReActAgent` classes? FunctionAgent is the current recommended class and works across providers that support native function calling, not just OpenAI. ReActAgent is still available for models without reliable function calling and uses a text-based reasoning loop instead. If you're starting a new project, default to FunctionAgent or AgentWorkflow unless you have a specific reason to use ReAct.

How do I test an agent before shipping it? Write a small set of representative prompts covering each tool, run them against the agent, and assert on which tools were called (not just the final text), since the final answer can look right even when the wrong tool produced it. The set_global_handler("simple") trace from the debugging section above gives you exactly the call sequence to assert against.