LangChain

LangChain Few-Shot Prompting: Dynamic Example Selection

Ira Menon · Jun 29, 2026 · 13 min read

Why Your Few-Shot Prompts Stop Working At Scale

Here's a pattern almost every AI engineer hits within the first month of building with LLMs. You write a prompt, add three or four hand-picked examples to show the model the format you want, ship it, and it works beautifully in the demo. Then real traffic arrives. Users ask questions that look nothing like your examples. The model starts drifting into formats you never approved, hallucinating fields that don't exist in your schema, or just ignoring the pattern entirely. You add a fifth example to patch the gap. Then a sixth. Eventually your prompt is a wall of text, half of it irrelevant to any given query, burning tokens and latency on examples that don't even apply to the current input.

The problem isn't few-shot prompting itself — it's static few-shot prompting. A fixed set of examples baked into a template assumes every incoming request is similar enough to your original test cases. In production, that assumption breaks fast. The fix is to stop hardcoding examples and start *selecting* them dynamically, per request, based on what the current input actually looks like. That's exactly what LangChain's ExampleSelector abstraction is built for, and it's one of the more underused tools in the framework.

This article walks through what few-shot prompting actually buys you, why static examples fail, and how to build a dynamic example selection pipeline in LangChain using semantic similarity, length-based constraints, and custom selection logic — with real, runnable code.

What Few-Shot Prompting Actually Does For an LLM

Before diving into selectors, it's worth being precise about why few-shot examples work at all. A large language model doesn't "understand" your task specification the way a junior engineer reads a ticket. It predicts the next token based on patterns in the input. When you show it two or three input-output pairs before your actual question, you're not explaining the task — you're demonstrating the exact transformation you want, and the model pattern-matches against that demonstration.

This matters for three reasons in production systems:

Format enforcement. If you need JSON with specific keys, or a particular tone, or a fixed output length, examples communicate that far more reliably than prose instructions.
Domain calibration. A general-purpose model doesn't know your company's internal jargon, your specific classification categories, or the edge cases your support team cares about. Examples teach it in-context, without fine-tuning.
Token efficiency versus fine-tuning. Fine-tuning a model for a narrow task can cost real engineering time and money. Few-shot prompting gets you 70-90% of the benefit for a fraction of the effort, and you can iterate on examples in minutes instead of retraining.

The catch is that examples are only useful if they're *relevant* to the current input. An example about refund requests doesn't help the model when the user is asking about a shipping delay. This is where static prompts fall apart — and where LangChain's example selectors earn their keep.

The Static Few-Shot Baseline (And Where It Breaks)

Let's start with the naive approach, because you need to see the failure mode before the fix makes sense. LangChain's FewShotPromptTemplate lets you hardcode a list of examples directly:

from langchain_core.prompts import FewShotPromptTemplate, PromptTemplate

examples = [
    {"input": "The app crashes when I upload a photo", "output": "Bug Report: Crash on media upload"},
    {"input": "How do I cancel my subscription?", "output": "Billing Question: Subscription cancellation"},
    {"input": "Can you add dark mode?", "output": "Feature Request: Dark mode support"},
]

example_prompt = PromptTemplate(
    input_variables=["input", "output"],
    template="Input: {input}\nOutput: {output}",
)

static_prompt = FewShotPromptTemplate(
    examples=examples,
    example_prompt=example_prompt,
    prefix="Classify the following customer message into a category.",
    suffix="Input: {input}\nOutput:",
    input_variables=["input"],
)

print(static_prompt.format(input="My payment failed twice today"))

This works fine for a demo with three categories. But now imagine your support system actually handles fifteen categories: refunds, shipping, account security, API errors, billing disputes, feature requests, bug reports, onboarding questions, and so on. You have two bad options:

Keep only 3-4 examples, which means most categories have zero representative examples in the prompt, so the model has to guess based on categories it *has* seen, biasing it toward whatever's in the static set.
Add examples for every category, which bloats the prompt to 20+ examples on every single call, even when the user's question is obviously a shipping issue and none of the billing or security examples are remotely relevant. You pay for those tokens on every request, and irrelevant examples can actively confuse the model by diluting the pattern it should be following.

Neither option scales. What you actually want is: for *this specific input*, pull the 3-4 examples from your library that are most similar to it, and only show those. That's dynamic example selection, and LangChain ships a purpose-built abstraction for exactly this.

The ExampleSelector Abstraction

LangChain defines a base class, BaseExampleSelector, with two methods you need to implement: add_example (to grow your example bank) and select_examples (to pick relevant ones given the current input). Everything else — semantic similarity search, length budgeting, n-gram overlap — is built on top of that interface.

The three selectors you'll actually use in practice are:

`SemanticSimilarityExampleSelector` — embeds your examples and the incoming query, then retrieves the nearest neighbors using a vector store. This is the workhorse for most real applications.
`LengthBasedExampleSelector` — picks as many examples as fit within a token/length budget, dropping examples as the input grows longer. Useful when you're context-constrained.
`MaxMarginalRelevanceExampleSelector` — like semantic similarity, but explicitly optimizes for *diversity* among the selected examples so you don't get four near-duplicate examples that all teach the same narrow pattern.

You can also write a fully custom selector when your selection logic doesn't fit any of these — for example, selecting examples based on a business rule like "always include one example from the user's own account history."

Let's build the semantic similarity version first, since it's the one you'll reach for most often.

Building a Semantic Similarity Selector

The idea is straightforward: convert every example into a vector embedding, store those vectors, and when a new query comes in, embed it too and retrieve the nearest examples by cosine similarity. LangChain wires this up for you through SemanticSimilarityExampleSelector.from_examples, which handles embedding and indexing in one call.

from langchain_core.example_selectors import SemanticSimilarityExampleSelector
from langchain_core.prompts import FewShotPromptTemplate, PromptTemplate
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

support_examples = [
    {"input": "The app crashes when I upload a photo", "output": "Bug Report"},
    {"input": "Video freezes on the checkout screen", "output": "Bug Report"},
    {"input": "How do I cancel my subscription?", "output": "Billing Question"},
    {"input": "Why was I charged twice this month?", "output": "Billing Question"},
    {"input": "Can you add dark mode?", "output": "Feature Request"},
    {"input": "It would be great to export data as CSV", "output": "Feature Request"},
    {"input": "Someone logged into my account from another device", "output": "Security Concern"},
    {"input": "I think my password was leaked", "output": "Security Concern"},
]

example_selector = SemanticSimilarityExampleSelector.from_examples(
    support_examples,
    OpenAIEmbeddings(),
    Chroma,
    k=3,
    input_keys=["input"],
)

example_prompt = PromptTemplate(
    input_variables=["input", "output"],
    template="Input: {input}\nCategory: {output}",
)

dynamic_prompt = FewShotPromptTemplate(
    example_selector=example_selector,
    example_prompt=example_prompt,
    prefix="Classify the customer message into a category based on the examples below.",
    suffix="Input: {input}\nCategory:",
    input_variables=["input"],
)

result = dynamic_prompt.format(input="I noticed a login I didn't make from an unknown location")
print(result)

Run this and the selector will surface the two security-related examples plus one more near-neighbor, ignoring the billing and feature-request examples entirely. That's the core win: the model sees only the examples that are actually instructive for *this* input, regardless of how many total examples live in your library. You can scale your example bank to hundreds of entries without bloating any individual prompt, because retrieval only ever pulls the top k.

A few details worth calling out:

`input_keys=["input"]` tells the selector which field of the example dict to embed and compare against — useful when your examples have multiple fields (input, output, metadata) but you only want similarity computed on the input text.
`k=3` controls how many examples get pulled per call. Start small (3-5) and increase only if you see the model missing patterns — more examples means more tokens and, past a point, diminishing returns.
The vector store (Chroma here) persists the embedded examples, so if you're selecting from a large library repeatedly, you want to build the index once and reuse it rather than re-embedding on every request.

Adding Examples at Runtime

One advantage of the selector abstraction over a hardcoded list is that you can grow your example bank as you go, without touching your prompt template. This matters in production because your best examples are often the ones you discover from real user traffic and mistakes the model made — not the ones you dreamed up while writing the initial prompt.

# A new pattern emerges from production logs: users describing slow performance
example_selector.add_example(
    {"input": "The dashboard takes forever to load after I filter by date", "output": "Bug Report"}
)

# Verify it gets picked up for a similar query
matches = example_selector.select_examples({"input": "Everything is really slow since the last update"})
for m in matches:
    print(m)

In a real system, you'd wire this into a feedback loop: when a human reviewer corrects a misclassification, that corrected pair becomes a new example added to the store. Over weeks, your example bank organically covers the actual distribution of inputs you see, instead of the handful of cases you imagined up front.

Length-Based Selection for Context Budgets

Semantic similarity is great when you have room to spare, but sometimes your constraint isn't relevance — it's raw context length. If your prompt already includes a long system message, retrieved documents from a RAG pipeline, and conversation history, you might only have a small token budget left for examples. LengthBasedExampleSelector handles this by greedily adding examples until a length threshold is hit, then stopping.

from langchain_core.example_selectors import LengthBasedExampleSelector
from langchain_core.prompts import FewShotPromptTemplate, PromptTemplate

examples = [
    {"input": "happy", "output": "sad"},
    {"input": "tall", "output": "short"},
    {"input": "energetic", "output": "lethargic"},
    {"input": "sunny", "output": "gloomy"},
    {"input": "windy", "output": "calm"},
]

example_prompt = PromptTemplate(
    input_variables=["input", "output"],
    template="Input: {input}\nOutput: {output}",
)

length_selector = LengthBasedExampleSelector(
    examples=examples,
    example_prompt=example_prompt,
    max_length=25,
)

dynamic_prompt = FewShotPromptTemplate(
    example_selector=length_selector,
    example_prompt=example_prompt,
    prefix="Give the antonym of every input.",
    suffix="Input: {input}\nOutput:",
    input_variables=["input"],
)

# Short input -> more examples fit
print(dynamic_prompt.format(input="big"))

# Longer input -> fewer examples get selected to stay under budget
long_input = "surprisingly enthusiastic and full of unexpected optimism this morning"
print(dynamic_prompt.format(input=long_input))

Notice the trade-off: as the input itself grows, the selector automatically shrinks the number of examples to keep the total prompt length bounded. This is the kind of thing you'd otherwise implement by hand with token counting and truncation logic — the selector abstracts it away. The default length function uses whitespace-based word counts, but you can pass a custom get_text_length callable if you want it to count actual tokens using tiktoken or your model provider's tokenizer, which is more accurate for billing and context-window purposes.

Custom Example Selectors for Business Logic

Sometimes neither semantic similarity nor length budgeting captures your actual selection criteria. Maybe you need to guarantee that at least one example comes from the user's own historical data, or that examples are balanced across categories so the model doesn't over-index on whichever category has the most training examples. For this, implement BaseExampleSelector directly.

from langchain_core.example_selectors.base import BaseExampleSelector
import random

class BalancedCategoryExampleSelector(BaseExampleSelector):
    """Always returns one example per category, so the model
    never sees a lopsided set skewed toward one label."""

    def __init__(self, examples):
        self.examples_by_category = {}
        for ex in examples:
            self.examples_by_category.setdefault(ex["output"], []).append(ex)

    def add_example(self, example) -> None:
        category = example["output"]
        self.examples_by_category.setdefault(category, []).append(example)

    def select_examples(self, input_variables):
        selected = []
        for category, examples in self.examples_by_category.items():
            selected.append(random.choice(examples))
        return selected


balanced_selector = BalancedCategoryExampleSelector(support_examples)
picked = balanced_selector.select_examples({"input": "anything at all"})
for p in picked:
    print(p)

This selector ignores the input entirely and instead enforces a structural guarantee: one example per known category, every time. You'd use this alongside a classification task where category imbalance in your example bank could otherwise bias the model toward over-predicting whichever label has the most demonstrated examples. Custom selectors are also where you'd hook in things like recency weighting (favor examples added in the last week), user-specific personalization (favor examples from the same customer segment), or A/B testing different example sets against each other.

The key insight is that select_examples just needs to return a list of dicts matching your example_prompt's expected keys — how you decide *which* dicts to return is entirely up to your business logic. LangChain doesn't force you into any one retrieval strategy.

Combining Selectors With Chat Models

Everything above uses PromptTemplate and plain string formatting, but most production LLM calls today go through chat models with structured message roles. LangChain supports few-shot examples in chat format via FewShotChatMessagePromptTemplate, which pairs naturally with an example selector.

from langchain_core.prompts import (
    ChatPromptTemplate,
    FewShotChatMessagePromptTemplate,
)
from langchain_openai import ChatOpenAI

example_prompt = ChatPromptTemplate.from_messages(
    [("human", "{input}"), ("ai", "{output}")]
)

few_shot_prompt = FewShotChatMessagePromptTemplate(
    example_selector=example_selector,  # reusing the semantic selector from earlier
    example_prompt=example_prompt,
)

final_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a support ticket classifier. Respond with only the category name."),
        few_shot_prompt,
        ("human", "{input}"),
    ]
)

model = ChatOpenAI(model="gpt-4o-mini", temperature=0)
chain = final_prompt | model

response = chain.invoke({"input": "My card was billed twice for the same order"})
print(response.content)

Here the selected examples get rendered as alternating human/AI turns, matching the conversational structure chat models expect, rather than as raw text blocks. This tends to produce better instruction-following than the plain-text version, because it mirrors the actual format the model was fine-tuned on — a turn-by-turn dialogue rather than a flat document. The selector logic underneath is identical; only the rendering target changes.

Practical Guidance for Production Use

A few lessons that matter once this moves past a notebook and into a real service:

Cache your embeddings. Re-embedding your entire example bank on every process restart is wasteful. Persist the vector store (Chroma, FAISS, or a hosted vector DB) so cold starts don't pay the embedding cost repeatedly.
Keep `k` small and measure. It's tempting to set k=10 "to be safe." In practice, 3-5 well-matched examples usually outperform 10 loosely related ones, and you save meaningfully on tokens and latency.
Version your example bank separately from your code. Examples are data, not logic. Treat additions and edits to your example store like you'd treat a dataset — with review, not ad hoc edits buried in a deploy.
Watch for example leakage in evaluation. If you're evaluating your classifier against a test set, make sure none of your few-shot examples are drawn from that same test set, or you'll get inflated accuracy numbers that don't reflect real-world performance.
Combine selectors when needed. Nothing stops you from writing a custom selector that first filters by length budget, then ranks the survivors by semantic similarity. The base class is intentionally minimal so you can compose strategies.

Dynamic example selection is a small architectural change with an outsized payoff: your few-shot prompts stay lean and relevant no matter how large your example library grows, and you stop babysitting a hardcoded list every time a new edge case shows up in production. It's one of those LangChain features that looks like a minor utility class until you've actually hit the wall that static examples create — at which point it becomes indispensable.

If you want to go deeper into this pattern alongside retrieval-augmented generation, agent tool use, and the rest of the LangChain ecosystem, our LangChain Tutorial 2026 course on TeachYou.ai walks through building production-grade LLM pipelines from first principles, including full working projects that use example selectors, memory, and chains together in realistic applications.