A Guide to the DSPy Framework

Most people building with large language models spend their time editing prompt strings by hand, running a few examples, tweaking a sentence, and hoping the change generalizes. The dspy framework takes a different approach: instead of writing prompts, you write typed Python functions that describe what a step of your pipeline should do, and DSPy compiles those descriptions into optimized prompts (or fine-tunes) automatically, based on your own data and metric. This guide covers the core building blocks of DSPy, walks through a complete retrieval-augmented pipeline, and shows how the optimizers actually improve a program instead of just running it.

What DSPy Actually Is

DSPy stands for "Declarative Self-improving Python." It was built at Stanford NLP and is now widely used for building LLM pipelines that need to be reliable across model swaps, not just a single provider's quirks. The core idea is a separation of concerns:

• Signatures describe the input/output contract of a task ("given a question and context, produce an answer").
• Modules are strategies for executing a signature, like Predict, ChainOfThought, or ReAct.
• Optimizers (formerly called "teleprompters") search over prompt instructions, few-shot examples, or even weight updates to make a module perform better on a metric you define.

The practical effect: you stop hand-crafting prompt text. You describe the task, write a metric that scores correctness, and let an optimizer run dozens or hundreds of trials to find phrasing and examples that actually move the metric. When you switch from one model to another, you re-run the optimizer instead of rewriting every prompt in your codebase.

Installing DSPy and Setting Up a Language Model

Install the package and point it at a model provider. DSPy supports OpenAI-compatible endpoints, Anthropic, local models served through Ollama, and anything reachable via LiteLLM.

pip install dspy

import dspy

lm = dspy.LM("anthropic/claude-sonnet-4-5", api_key="your-api-key")
dspy.configure(lm=lm)

dspy.configure sets a global default language model for every module in the program, though you can override it per call by passing lm= to any module invocation. This matters when you want a cheap model for a retrieval-filtering step and a stronger model for final synthesis.

Signatures: Describing Tasks Instead of Writing Prompts

A signature is the smallest unit in DSPy. You can write one inline as a string:

qa = dspy.Predict("question -> answer")
result = qa(question="What year was the transistor invented?")
print(result.answer)

That string form is convenient for prototyping, but the real power shows up when you define a signature as a class, because you can attach docstrings, field descriptions, and types that the optimizer will use when searching for better instructions.

class AnswerQuestion(dspy.Signature):
    """Answer the question using only the provided context. If the context does not contain the answer, say 'not found'."""

    context: str = dspy.InputField(desc="passages retrieved from the knowledge base")
    question: str = dspy.InputField()
    answer: str = dspy.OutputField(desc="a concise, factual answer, one or two sentences")

Note what is missing here: no "You are a helpful assistant," no formatting instructions, no few-shot examples baked into a string. Those are exactly the pieces DSPy generates and tunes for you later. Field types can be more than str too. DSPy signatures support Python type hints including list[str], dict, Literal, and Pydantic models, which lets you constrain outputs to a fixed set of categories or a structured object without writing manual parsing code.

from typing import Literal

class ClassifyTicket(dspy.Signature):
    """Classify a support ticket into one of the given categories."""

    ticket_text: str = dspy.InputField()
    category: Literal["billing", "technical", "account", "other"] = dspy.OutputField()

Modules: Strategies for Executing a Signature

A signature says *what* to do; a module says *how*. DSPy ships several built-in modules:

• `dspy.Predict`: a direct call, one prompt in, one structured output out.
• `dspy.ChainOfThought`: automatically adds a reasoning field before the output field, so the model "thinks" before answering. This alone often improves accuracy on multi-step questions without you writing a single "think step by step" instruction.
• `dspy.ReAct`: interleaves reasoning with tool calls, useful for agents that need to look things up or run functions before responding.
• `dspy.ProgramOfThought`: has the model generate and execute code to solve a problem, useful for math or data-manipulation tasks.

Swapping strategies is a one-line change:

predict_answer = dspy.Predict(AnswerQuestion)
cot_answer = dspy.ChainOfThought(AnswerQuestion)

Both objects expose the exact same input/output interface. This is the point: your pipeline code depends on the signature's field names, not on how the underlying call is structured. You can experiment with Predict versus ChainOfThought versus a custom module without touching any downstream code that consumes result.answer.

Composing Modules Into a Program

Real applications chain multiple steps together. DSPy programs are just Python classes that subclass dspy.Module, store sub-modules in __init__, and wire them together in forward.

class RAGPipeline(dspy.Module):
    def __init__(self, retriever, num_passages=3):
        super().__init__()
        self.retrieve = retriever
        self.num_passages = num_passages
        self.generate_answer = dspy.ChainOfThought(AnswerQuestion)

    def forward(self, question):
        passages = self.retrieve(question, k=self.num_passages)
        context = "\n\n".join(passages)
        prediction = self.generate_answer(context=context, question=question)
        return dspy.Prediction(answer=prediction.answer, context=context)

Here retriever is any callable that returns a list of text passages for a query, for example a wrapper around a vector database client. DSPy does not force a specific retrieval backend; it just expects something with a compatible calling convention. A minimal retriever wrapper around a vector store might look like this:

class SimpleRetriever:
    def __init__(self, vector_store):
        self.vector_store = vector_store

    def __call__(self, query, k=3):
        results = self.vector_store.search(query, top_k=k)
        return [r.text for r in results]

retriever = SimpleRetriever(vector_store=my_vector_db_client)
rag = RAGPipeline(retriever=retriever)

response = rag(question="How does a transformer's attention mechanism scale with sequence length?")
print(response.answer)

Running this program right now, unoptimized, already works: DSPy generates a reasonable default prompt from the signature and docstring. The optimization step below is what pushes accuracy up on your actual data.

Writing a Metric

Optimizers need something to optimize against. A metric in DSPy is a plain Python function that takes an example and a prediction and returns a score, usually a boolean or a float.

def answer_correctness(example, prediction, trace=None):
    predicted = prediction.answer.strip().lower()
    gold = example.answer.strip().lower()
    return gold in predicted or predicted in gold

For tasks where a simple string match is too strict, you can use another LLM call as the judge, which DSPy also treats as an ordinary module:

class JudgeCorrectness(dspy.Signature):
    """Judge whether the predicted answer is factually consistent with the gold answer."""

    question: str = dspy.InputField()
    gold_answer: str = dspy.InputField()
    predicted_answer: str = dspy.InputField()
    is_correct: bool = dspy.OutputField()

judge = dspy.Predict(JudgeCorrectness)

def llm_judged_metric(example, prediction, trace=None):
    result = judge(
        question=example.question,
        gold_answer=example.answer,
        predicted_answer=prediction.answer,
    )
    return result.is_correct

Metrics can be as strict or as lenient as your application demands. For classification tasks, exact match is usually fine. For open-ended generation, an LLM judge or a rubric-based score tends to correlate better with what users actually care about.

Building a Training Set

DSPy optimizers need labeled examples, but the bar is lower than it sounds. Ten to fifty good examples is often enough to see a meaningful jump, and you do not need to hand-write full reasoning chains, just the final input/output pairs.

trainset = [
    dspy.Example(
        question="What is the time complexity of binary search?",
        answer="O(log n)",
    ).with_inputs("question"),
    dspy.Example(
        question="Who wrote the original TCP/IP specification?",
        answer="Vint Cerf and Bob Kahn",
    ).with_inputs("question"),
    # ... more examples
]

.with_inputs("question") tells DSPy which fields are inputs versus labels, since the same Example object is reused both for scoring against the gold label and for constructing few-shot demonstrations.

Optimizing the Pipeline

This is where DSPy earns its name. An optimizer takes your program, your metric, and your training set, and searches for a better configuration: better instructions, better few-shot examples, or both.

BootstrapFewShot is the simplest starting point. It runs your program on the training set, keeps the examples where the metric passed, and uses those as few-shot demonstrations inside the compiled program.

from dspy.teleprompt import BootstrapFewShot

optimizer = BootstrapFewShot(metric=answer_correctness, max_bootstrapped_demos=4)
compiled_rag = optimizer.compile(rag, trainset=trainset)

For a bigger accuracy jump, MIPROv2 also searches over the instruction text itself, generating and testing candidate phrasings of the signature's docstring in addition to selecting demonstrations.

from dspy.teleprompt import MIPROv2

optimizer = MIPROv2(metric=answer_correctness, auto="medium")
compiled_rag = optimizer.compile(rag, trainset=trainset)

After compiling, compiled_rag behaves like the original program from the outside; same forward signature, same call interface, but internally it now uses the optimized instructions and demonstrations DSPy found. Save it so you don't have to recompile every run:

compiled_rag.save("compiled_rag.json")

loaded_rag = RAGPipeline(retriever=retriever)
loaded_rag.load("compiled_rag.json")

Evaluating Before and After

Always measure the delta, because compilation is not guaranteed to help on every dataset, and you want a number to justify the extra latency of a more complex prompt.

from dspy.evaluate import Evaluate

evaluator = Evaluate(devset=devset, metric=answer_correctness, num_threads=8, display_progress=True)

baseline_score = evaluator(rag)
optimized_score = evaluator(compiled_rag)

print(f"baseline: {baseline_score}, optimized: {optimized_score}")

Keep a held-out devset separate from trainset. It is easy to overfit the few-shot selection to your training examples, and a separate dev split catches that before it reaches production.

Adding Tools With ReAct

For agentic tasks, dspy.ReAct wires a signature to a set of Python functions the model can call mid-reasoning. Each tool is just a regular function with a docstring; DSPy handles the calling convention.

def search_docs(query: str) -> str:
    """Search internal documentation and return the top matching passage."""
    results = retriever(query, k=1)
    return results[0] if results else "no results found"

def calculate(expression: str) -> str:
    """Evaluate a basic arithmetic expression and return the result."""
    try:
        return str(eval(expression, {"__builtins__": {}}))
    except Exception as e:
        return f"error: {e}"

agent = dspy.ReAct(AnswerQuestion, tools=[search_docs, calculate])
result = agent(context="", question="What is 15% of the average of 240 and 360?")
print(result.answer)

The agent decides when to call search_docs or calculate based on the question, and DSPy manages the trace of tool calls and intermediate reasoning behind the scenes. You can compile a ReAct module the same way as any other module, using BootstrapFewShot or MIPROv2, and the optimizer will improve both the reasoning trace and the tool-use pattern.

Debugging: Inspecting What Actually Got Sent

When output quality is off, the fastest fix is looking at the literal prompt DSPy generated, not guessing.

dspy.inspect_history(n=1)

This prints the last prompt sent to the language model, including the compiled instructions and any few-shot demonstrations that were attached. It is the single most useful debugging command in DSPy, because the abstraction can otherwise feel like a black box. Run it right after a call that produced a bad result and you will usually spot the issue immediately, whether it's a missing constraint in the signature's docstring or a bad demonstration that snuck into the bootstrapped set.

Common Pitfalls

Skipping the metric. Some teams treat BootstrapFewShot as a magic "make it better" button and pass a metric that always returns True. That produces a compiled program with random demonstrations attached, no better than the original. Write a metric that actually discriminates good from bad outputs before compiling anything.

Training set too small or too homogeneous. Ten examples that are all variations of the same question type won't teach the optimizer much about edge cases. Vary difficulty and format in your training set the way you'd vary a test suite.

Not versioning compiled artifacts. A compiled program is a JSON file plus your source code. If you change the signature's field names after compiling, the saved demonstrations can silently mismatch on load. Recompile after any signature change, and store the compiled JSON next to the pipeline version it belongs to, in source control.

Assuming one model's compiled prompt transfers to another model. A program compiled against one provider's model does not automatically transfer its quality gains to a different model family. Recompile when you swap models; the whole point of DSPy is that this recompilation step is cheap and automated compared to hand-rewriting prompts.

FAQ

Is DSPy a replacement for LangChain or LlamaIndex? Not exactly. LangChain and LlamaIndex are largely oriented around orchestration, chaining calls, managing memory, and connecting to data sources. DSPy focuses specifically on the prompt itself: describing what a step should do and then automatically optimizing how that description becomes a prompt. Many teams use DSPy for the reasoning and generation modules and a separate library or a custom client for retrieval and storage.

Do I need hundreds of examples to see a benefit? No. Meaningful gains often show up with as few as ten to thirty labeled examples, especially with MIPROv2, which searches instructions in addition to demonstrations. More examples generally help, but the framework is designed to work with the kind of small, hand-curated datasets most teams can realistically produce.

Can DSPy fine-tune model weights, not just prompts? Yes, for supported open models DSPy includes optimizers that produce fine-tuning data from your bootstrapped traces and can drive a fine-tuning job, in addition to the prompt-only optimizers. This is a more advanced path and typically comes after you've already validated a prompt-based compiled program works well.

How does DSPy handle structured outputs like JSON? Through typed output fields on a signature. Declaring an OutputField with a Pydantic model or a Literal type tells DSPy to constrain and parse the model's response into that structure, so you get a validated Python object back instead of raw text you have to parse yourself.

What happens if the optimizer makes things worse? Always compare compiled_rag against the original rag on a held-out dev set using Evaluate, as shown above. If the optimized version scores lower, keep the baseline. Optimization is a search process, not a guarantee, and comparing before and after is a required step, not an optional one.

Does DSPy lock me into a single model provider? No. dspy.LM is a thin wrapper that works with any OpenAI-compatible endpoint, Anthropic's API, or local models through Ollama and similar servers. Switching providers is a one-line change to the dspy.LM constructor, followed by a recompile if you want the optimizer to re-tune the prompt for the new model's behavior.