LLM App Frameworks Compared

Picking from the growing pile of llm app frameworks is one of the first real decisions any team building on top of language models has to make, and it is a decision that is easy to get wrong because most of these tools look interchangeable from the outside. They are not. Some are thin wrappers around an API call, some are full orchestration engines with graphs and state machines, and some exist mainly to make retrieval-augmented generation less painful. This article walks through the major categories, shows working code for the frameworks worth knowing, and gives you a checklist for choosing one without wasting a sprint on the wrong abstraction.

The short version: if you need a single call with tools and structured output, reach for a provider SDK directly (the Claude Agent SDK, the OpenAI Agents SDK, or the Vercel AI SDK if you are in JavaScript). If you need retrieval over a document corpus, use LlamaIndex or a lighter RAG-only library. If you need long-running, stateful, branching agent workflows, use LangGraph or a graph-based orchestrator. If you need many agents cooperating with defined roles, look at CrewAI or AutoGen. Everything below explains why.

What People Actually Mean by "LLM App Framework"

The term gets used for three genuinely different jobs, and conflating them is the number one reason teams pick the wrong tool.

The first job is orchestration: chaining prompts, tools, and model calls into a pipeline with retries, memory, and branching logic. LangChain and LangGraph live here.

The second job is retrieval: getting the right chunks of your data in front of the model at the right time. LlamaIndex and Haystack specialize in this, though most orchestration frameworks bolt on retrieval features too.

The third job is agency: letting the model decide, in a loop, which tool to call next until a task is done. This is where the Claude Agent SDK, the OpenAI Agents SDK, CrewAI, and AutoGen sit, each with a different opinion on how much structure to impose on that loop.

A single production app usually needs pieces of all three, which is why "framework fatigue" is real. The fix is not finding one framework that does everything. It is picking the smallest tool for each job and wiring them together yourself, which is usually less code than people expect.

Orchestration Frameworks: LangChain and LangGraph

LangChain was the framework that popularized the "chain" abstraction: compose prompt templates, models, and parsers into a pipeline you can call like a function. It is still the most widely adopted general-purpose framework, with the largest set of prebuilt integrations for vector stores, document loaders, and third-party tools.

from langchain_anthropic import ChatAnthropic
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

model = ChatAnthropic(model="claude-sonnet-4-5")
prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a concise technical writer."),
    ("human", "Summarize this in three bullet points: {text}"),
])

chain = prompt | model | StrOutputParser()
result = chain.invoke({"text": "Long article content goes here..."})
print(result)

That pipe syntax (LangChain Expression Language) is the framework's biggest strength and its biggest source of confusion for newcomers. Once you understand it, chains compose cleanly. Before that, debugging a five-stage pipeline feels like debugging someone else's regex.

LangGraph is LangChain's answer to the fact that real agent workflows are not linear chains, they are graphs with loops, conditionals, and shared state. You define nodes (functions or model calls) and edges (the logic that decides what runs next), and LangGraph handles the state machine underneath.

from langgraph.graph import StateGraph, END
from typing import TypedDict

class AgentState(TypedDict):
    query: str
    result: str
    retries: int

def call_model(state: AgentState) -> AgentState:
    state["result"] = run_model(state["query"])
    return state

def needs_retry(state: AgentState) -> str:
    if "error" in state["result"] and state["retries"] < 3:
        return "retry"
    return "done"

graph = StateGraph(AgentState)
graph.add_node("model", call_model)
graph.set_entry_point("model")
graph.add_conditional_edges("model", needs_retry, {"retry": "model", "done": END})

app = graph.compile()
output = app.invoke({"query": "Explain vector databases", "retries": 0})

Use LangGraph when your workflow genuinely branches or loops, for example an agent that plans, executes, checks its own work, and replans on failure. Do not reach for it for a straight-line summarize-then-format pipeline; that is LangChain Expression Language or, honestly, three lines of plain Python.

RAG-Focused Frameworks: LlamaIndex and Haystack

If your app's core problem is "answer questions grounded in our documents," a RAG-specialized framework will save you weeks compared to hand-rolling chunking, embedding, and retrieval logic.

LlamaIndex is built around the index as the central object: point it at a folder of documents, pick an index type, and query it.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

response = query_engine.query("What is our refund policy for annual plans?")
print(response)

Under the hood this handles chunking, embedding, storage, and retrieval, and it exposes knobs for every stage when the defaults are not good enough: custom node parsers, hybrid search, re-ranking, and query transformation. LlamaIndex has also grown agent and workflow features over the years, but its core strength and reason to exist is still retrieval quality.

Haystack takes a similar "pipeline of components" approach but leans more toward production search infrastructure, with first-class support for hybrid retrieval (keyword plus vector), evaluation pipelines, and deployment patterns that predate the current wave of LLM tooling. Teams with an existing search or information-retrieval background often find Haystack's mental model more familiar than LlamaIndex's.

The practical rule: if retrieval quality (not agent behavior) is your bottleneck, measure it directly, do not just swap frameworks. Bad chunking strategy or a mismatched embedding model will sink LlamaIndex, LangChain, or a hand-rolled pipeline equally.

Comparing the Major LLM App Frameworks for Agents

This is where the field is most crowded, and where picking the wrong llm app framework costs the most time, because agent loops are harder to rip out once tools and memory are wired through them.

The Claude Agent SDK and the OpenAI Agents SDK sit at the lightweight end. They give you a tool-calling loop, structured output, and session management, without imposing a graph or crew abstraction on top. If you are building on one provider and want the thinnest possible layer between your code and the model's tool-use loop, start here.

from anthropic import Anthropic

client = Anthropic()

def get_weather(city: str) -> str:
    return f"It is sunny in {city}."

tools = [{
    "name": "get_weather",
    "description": "Get current weather for a city",
    "input_schema": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"],
    },
}]

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    tools=tools,
    messages=[{"role": "user", "content": "What's the weather in Lisbon?"}],
)

for block in response.content:
    if block.type == "tool_use" and block.name == "get_weather":
        result = get_weather(**block.input)
        print(result)

CrewAI and AutoGen sit at the "multiple agents with roles" end. CrewAI's model is a crew of agents, each with a role, a goal, and a backstory, working through a defined process (sequential or hierarchical).

from crewai import Agent, Task, Crew

researcher = Agent(
    role="Researcher",
    goal="Find accurate technical facts about the given topic",
    backstory="A meticulous analyst who never states an unverified claim.",
)

writer = Agent(
    role="Writer",
    goal="Turn research notes into a clear explanation",
    backstory="A technical writer who values clarity over jargon.",
)

research_task = Task(
    description="Research how vector databases handle approximate nearest neighbor search",
    agent=researcher,
    expected_output="A bullet list of key facts",
)

write_task = Task(
    description="Write a short explanation based on the research notes",
    agent=writer,
    expected_output="A three paragraph explanation",
)

crew = Crew(agents=[researcher, writer], tasks=[research_task, write_task])
result = crew.kickoff()

AutoGen (from Microsoft Research) takes a conversational approach: agents talk to each other in a group chat, and a manager agent decides turn order. It is a good fit when the problem genuinely looks like a discussion, for example a coder agent and a critic agent iterating on a solution together, but it can produce meandering conversations if the roles and stop conditions are not tightly specified.

The tradeoff across all of these is the same one you see in every abstraction layer: more structure means less code for the common case and more fighting the framework for the uncommon case. A two-agent CrewAI setup is genuinely less code than the equivalent hand-rolled loop. A twelve-agent CrewAI setup with custom routing logic is often more code and harder to debug than just writing the loop yourself.

Lightweight and Language-Specific Options

Not every project needs Python. The Vercel AI SDK is the default choice for JavaScript and TypeScript teams, particularly anyone building with Next.js, because it handles streaming responses to the browser, tool calling, and structured generation with a small, well-typed API.

import { generateText, tool } from "ai";
import { anthropic } from "@ai-sdk/anthropic";
import { z } from "zod";

const result = await generateText({
  model: anthropic("claude-sonnet-4-5"),
  tools: {
    getWeather: tool({
      description: "Get current weather for a city",
      parameters: z.object({ city: z.string() }),
      execute: async ({ city }) => `It is sunny in ${city}.`,
    }),
  },
  prompt: "What's the weather in Lisbon?",
});

console.log(result.text);

Pydantic AI is worth knowing if your team already lives in Pydantic for data validation. It treats the model's output as a typed Pydantic model from the start, which removes an entire category of "parse the JSON the model returned and hope it validates" bugs.

DSPy takes a fundamentally different angle: instead of hand-writing prompts, you declare the input/output signature you want and let DSPy's optimizers search for effective prompts and few-shot examples against your own evaluation metric. It is less of an app framework and more of a prompt-optimization compiler, and it is worth adopting once you have a working pipeline and want to systematically improve accuracy rather than hand-tune wording.

Semantic Kernel, Microsoft's framework, targets .NET and enterprise Python shops that need plugin-style extensibility and tight integration with the Microsoft stack (Azure AI Search, Azure OpenAI). If your organization already standardizes on .NET, it removes a lot of glue code you would otherwise write yourself.

How to Choose an LLM App Framework for Your Project

Work through these questions in order, because later ones only matter once the earlier ones are settled.

1. What is the actual task shape? A single request-response with tools is not the same problem as a multi-step research agent, and a multi-step agent is not the same problem as a retrieval question-answering system. Name the shape before naming a framework.
2. How many models and providers do you need? If you are committed to one provider, that provider's own SDK (Claude Agent SDK, OpenAI Agents SDK) is usually leaner and better maintained for that provider's features than a cross-provider abstraction. If you need to swap providers per environment or per customer, a cross-provider layer like LangChain or the Vercel AI SDK earns its keep.
3. Do you need retrieval, and how good does it need to be? If retrieval accuracy is core to the product, budget time to evaluate chunking and embedding choices directly, using LlamaIndex or Haystack as the harness rather than treating either as a black box.
4. How many independent agents does the task really need? Most tasks that look like they need three agents actually need one agent with three tools. Multi-agent frameworks add real value when the sub-tasks require genuinely different context, memory, or permissions, not just different prompts.
5. What is your team's debugging tolerance? Heavier frameworks (LangChain, CrewAI, AutoGen) trade transparency for convenience. When something goes wrong three layers deep in a chain, you need to be comfortable reading the framework's source, not just its docs. If that sounds unpleasant, lean toward the lightest tool that solves the problem.
6. What does the exit cost look like? Ask before you commit: if this framework turns out to be wrong in six months, how much of the code is portable? Code built directly against a provider's tool-calling API tends to port easily. Code built against a framework's proprietary chain or crew abstractions tends to require a rewrite.

A reasonable default for a new project in 2026: start with the provider SDK directly for the core loop, add LlamaIndex only if retrieval quality is a measured bottleneck, and reach for LangGraph or CrewAI only once you have a concrete workflow that the plain SDK cannot express cleanly. Adding structure early, before you know what your workflow actually looks like, is the most common way teams end up fighting their own framework.

Combining Frameworks Instead of Choosing Just One

Production systems rarely use exactly one framework end to end, and that is fine. A common, effective stack looks like this: LlamaIndex or a custom pipeline handles document ingestion and retrieval, the retrieved context gets passed into a Claude Agent SDK or OpenAI Agents SDK loop that handles tool calls and reasoning, and the whole thing gets wrapped in a thin API layer (FastAPI, Express, or the Vercel AI SDK's route handlers if you are in Next.js) that your frontend talks to.

The mistake to avoid is letting one framework's opinions leak into every layer of the app. If LangChain's chain abstraction is handling your prompt composition, you do not also need CrewAI managing agent roles for the same task, and you do not need LlamaIndex's full query engine when a plain vector similarity search would answer the question. Each framework should own exactly the layer it is good at, and the seams between them should be plain function calls and typed data, not framework-specific objects passed across boundaries.

FAQ

Do I need a framework at all, or can I just call the API directly? For many apps, no framework is needed. A single provider SDK call with tool definitions, wrapped in your own retry and logging logic, covers a large share of real products. Reach for a framework only when you hit a specific problem it solves better than fifty lines of your own code would, such as managing retrieval over thousands of documents or coordinating several agents with distinct memory.

Is LangChain still worth learning given how much criticism it gets online? Yes, mainly because of its integration coverage and because LangGraph (built by the same team) has become a genuinely strong choice for stateful agent workflows. The criticism usually targets early versions of the chain abstraction being over-engineered for simple tasks, which is a real concern, not a reason to skip the ecosystem entirely.

What's the difference between an orchestration framework and an agent framework? Orchestration frameworks (LangChain, Haystack pipelines) are mostly about you defining the control flow: step one runs, then step two, with conditional branches you write. Agent frameworks (Claude Agent SDK, CrewAI, AutoGen) hand more of that control flow to the model itself, which decides which tool to call next inside a loop you configure but do not fully script.

Should I use CrewAI or AutoGen for a multi-agent system? CrewAI tends to be a better fit when you can define clear roles and a mostly sequential or hierarchical process up front, because its abstractions match that shape directly. AutoGen tends to be a better fit when the task benefits from open-ended back-and-forth between agents, such as a coder and a reviewer iterating until tests pass, because its group-chat model handles that kind of exchange more naturally.

Can I switch frameworks later without rewriting everything? It depends on how much of your business logic lives inside the framework's abstractions versus in plain functions the framework calls. Keep your tool implementations, retrieval logic, and data models as plain, framework-agnostic code, and treat the framework as a thin layer that wires them together. Done that way, swapping the orchestration layer later is a days-long task, not a weeks-long rewrite.

How do I evaluate whether a framework is actually helping or just adding overhead? Track two things over a few weeks of real use: how much boilerplate it removed compared to hand-rolling the same feature, and how much time you lost debugging inside the framework's internals when something broke. If the second number starts approaching the first, that is a strong signal you have outgrown the framework's sweet spot for your use case, or picked one with a mismatched abstraction.