Selecting Few-Shot Examples That Work

Few-shot example selection is the practice of choosing which input-output pairs to paste into a prompt so a model can infer a task's pattern without fine-tuning. Most people treat it as an afterthought, they grab the first two or three examples that come to mind and move on. That is why so many "few-shot" prompts perform worse than a well-written zero-shot instruction. The examples you choose, how many you include, and the order you put them in change model behavior more than almost any other lever in prompt design, and the effect is measurable if you test for it.

This matters because few-shot examples do two jobs at once: they specify the output format (structure, tone, length) and they implicitly define the input distribution the model should expect. If your examples are all easy cases, the model will guess that every input is easy. If your examples share a quirk, like all outputs starting with the same word, the model will copy the quirk instead of the underlying task. Selection is not a cosmetic step, it is where most of the "prompt engineering" actually happens.

Why few-shot example selection matters more than example count

A common mistake is assuming more examples are always better. In practice, three well-chosen examples routinely beat ten randomly grabbed ones, and past a certain point extra examples just eat context budget and slow down every call. What actually moves accuracy is coverage: do your examples span the range of inputs the model will see in production, including the awkward edge cases, or do they all look like the happy path?

Think of few-shot examples as a tiny, hand-labeled training set. The same principles that make a training set useful apply here:

• Representativeness: examples should mirror the real distribution of inputs, not just the easy 80%.
• Diversity: examples should differ from each other in structure, length, and difficulty, not just in surface wording.
• Label quality: every example's output must be exactly what you want the model to produce, including formatting. The model will copy mistakes as faithfully as it copies correct answers.
• Minimal redundancy: two examples that teach the same lesson waste a slot that could cover a different failure mode.

If you only remember one rule, make it this one: every example should teach the model something it could not infer from the others.

Start from failures, not from convenience

The fastest way to build a bad few-shot set is to write examples from memory before you have any real data. Instead, run your task zero-shot (or with a rough draft prompt) against a batch of real or realistic inputs, and look at where the model gets it wrong. Those failures are your example candidates. This flips the usual order: instead of guessing what examples might help, you let the model tell you what it needs help with.

A simple loop:

def collect_candidate_failures(model_call, dataset, grader):
    """
    model_call: function(input_text) -> output_text
    dataset: list of (input_text, expected_output)
    grader: function(output_text, expected_output) -> bool
    """
    failures = []
    for input_text, expected_output in dataset:
        output_text = model_call(input_text)
        if not grader(output_text, expected_output):
            failures.append({
                "input": input_text,
                "expected": expected_output,
                "got": output_text,
            })
    return failures

Cluster the failures by the kind of mistake, wrong format, wrong reasoning path, missed edge case, hallucinated field, and pick one or two representative examples from each cluster. That gives you a few-shot set that is explicitly aimed at your model's weak spots instead of your intuition about what's hard.

Cover the difficulty range, not just the errors

Failure-driven selection is a great starting point, but a set built entirely from hard cases can backfire: the model may start treating every input as a hard case and over-hedge on simple ones. Balance your set across three buckets:

• Trivial cases that establish the basic format and the "default" behavior.
• Medium cases that show a common but non-obvious variation (a missing field, an ambiguous category, a multi-part answer).
• Hard cases pulled from your failure analysis, showing exactly how to handle the trap that broke the model before.

A three-to-five example set with one trivial, two medium, and one or two hard examples usually generalizes better than five examples of uniform difficulty. If you're doing classification with a small label set, make sure every label appears at least once, an omitted label is an invitation for the model to never predict it.

Static sets versus retrieval-based selection

There are two broad strategies for choosing examples, and picking the wrong one for your situation is a common source of wasted effort.

Static few-shot means you hand-pick a fixed set of examples once and reuse them for every call. This works well when your task has low input variance, support ticket triage into five categories, invoice field extraction from a consistent template, sentiment labeling. Build the set carefully using the failure-driven process above, test it, and ship it. Static sets are simpler to version, cheaper (no retrieval step), and easier to audit for bias.

Dynamic (retrieval-based) few-shot means you keep a larger pool of labeled examples and, for each new input, retrieve the examples most similar to it, then insert only those into the prompt. This is the right choice when your input distribution is wide, freeform customer questions, code review comments across many languages, document types that vary a lot. The core idea: an example that is semantically close to the current input teaches the model more than a generic example ever could.

Here's a minimal retrieval-based selector using embeddings and cosine similarity, with no vector database required for pools under a few thousand examples:

import numpy as np

def cosine_similarity(a, b):
    a = np.array(a)
    b = np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def select_few_shot_examples(query_embedding, example_pool, k=4, min_similarity=0.2):
    """
    example_pool: list of dicts with keys "input", "output", "embedding"
    Returns the top-k most similar examples above a similarity floor,
    so the query never gets padded with irrelevant filler.
    """
    scored = [
        (cosine_similarity(query_embedding, ex["embedding"]), ex)
        for ex in example_pool
    ]
    scored = [pair for pair in scored if pair[0] >= min_similarity]
    scored.sort(key=lambda pair: pair[0], reverse=True)
    return [ex for _, ex in scored[:k]]

To generate embeddings, use whatever embedding model you already have in your stack (an OpenAI embeddings endpoint, a Cohere embed model, or a local sentence-transformers model all work fine for this). The important part is not which embedding model you pick, it's that you embed the *input* text of both the pool and the incoming query consistently, using the same model and the same preprocessing.

A practical addition worth making: enforce diversity within the retrieved set so you don't return four near-duplicate examples for a query that has several similar entries in the pool. A simple greedy approach works:

def select_diverse_few_shot(query_embedding, example_pool, k=4, diversity_penalty=0.3):
    remaining = list(example_pool)
    chosen = []
    while remaining and len(chosen) < k:
        best_score = -1
        best_example = None
        for ex in remaining:
            relevance = cosine_similarity(query_embedding, ex["embedding"])
            redundancy = max(
                (cosine_similarity(ex["embedding"], c["embedding"]) for c in chosen),
                default=0,
            )
            score = relevance - diversity_penalty * redundancy
            if score > best_score:
                best_score = score
                best_example = ex
        chosen.append(best_example)
        remaining.remove(best_example)
    return chosen

Ordering examples changes the answer

Language models are sensitive to example order, this is well documented and easy to reproduce yourself. Two patterns matter most in practice:

• Recency bias: the last example in the prompt has outsized influence on the output format and tone. If your final example happens to be the shortest or simplest one, expect the model to default toward short, simple answers even for complex queries.
• Majority-label bias in classification: if three out of four examples share the same label, the model skews toward predicting that label regardless of the actual input, even when each individual example is correct.

Two defenses that are cheap to apply:

1. Put your best, most representative example last, not your simplest one.
2. For classification tasks, balance labels within the visible example set, or explicitly shuffle order per call if you're using dynamic retrieval, so no fixed bias creeps in across a large volume of calls.

If you're building a system that logs a lot of few-shot calls, it's worth a quick experiment: hold the example pool fixed, vary only the order, and check whether output labels shift. If they do, your prompt is more order-sensitive than you'd like, and it's worth restructuring the instruction text to state the task rule explicitly rather than relying purely on pattern-matching from the examples.

How many examples is enough

There's no universal number, but a useful heuristic: increase the count until adding another example stops changing your eval score, then stop, because every extra example costs context tokens and latency with no return. In practice:

• Simple format-only tasks (extract this field, follow this template) often need only one or two examples.
• Classification with more than four or five labels usually needs at least one example per label.
• Multi-step reasoning tasks benefit from two to four worked examples that show the reasoning chain, not just the final answer, more than that tends to just pad the prompt without adding new patterns.
• Style transfer or tone matching benefits from three to six examples that show the range of the target style, since tone is harder to pin down from a single sample.

Run this as an actual experiment rather than a guess. Fix everything except the example count, run your eval set at 1, 2, 4, and 8 examples, and plot accuracy against count. The curve almost always flattens well before you expect it to.

Testing your example set like a mini training pipeline

Treat your few-shot set as a versioned artifact, not a paste-and-forget block of text. A workable process:

1. Hold out a test set of real inputs that never gets used to pick examples, this prevents you from unconsciously tuning examples to match your own eval set.
2. Score each candidate example set (static or retrieval config) against the held-out test set using an automated grader where possible, exact match, structural validation, or a rubric-based LLM judge for open-ended tasks.
3. Track scores over time as you swap examples in and out. If a new example set improves accuracy but shifts a specific failure mode, note it explicitly rather than only tracking the aggregate number.
4. Re-run the eval whenever you change the base model or the instruction text above the examples, both interact with example selection in ways that are hard to predict from first principles.

def evaluate_example_set(model_call_with_examples, test_set, grader):
    correct = 0
    for input_text, expected_output in test_set:
        output_text = model_call_with_examples(input_text)
        if grader(output_text, expected_output):
            correct += 1
    return correct / len(test_set)

This is a small amount of extra scaffolding, but it turns "I think these examples work better" into a number you can defend and iterate on, which matters a lot once a prompt is running in production against real traffic.

Common mistakes to avoid

• Copying examples from documentation instead of your own domain. Generic examples teach generic patterns. Always pull from real inputs your system will actually see.
• Letting examples leak unintended patterns. If every example happens to be under 50 words, the model will assume short answers are always correct, even when the task calls for a longer one.
• Never retiring stale examples. As your task or user base evolves, examples that were representative a year ago can become misleading. Revisit the set on a schedule, not just when something breaks.
• Skipping the "why" in reasoning examples. For tasks requiring multi-step reasoning, showing only the final answer teaches the model to skip steps too. Show the intermediate reasoning explicitly if you want the model to reproduce it.
• Mixing formats across examples. If one example ends the output with a period and another doesn't, or one uses a bulleted list and another prose, the model has to guess which convention to follow, and it often blends the two badly.
• Ignoring negative examples. Sometimes showing what *not* to do, paired with a short note on why, corrects a stubborn failure mode faster than another positive example would.

FAQ

How many few-shot examples should a prompt have? There is no fixed number, but most tasks plateau between two and six examples. Run an eval sweep at increasing counts and stop adding examples once the accuracy curve flattens, since extra examples past that point mostly add cost and latency without improving results.

Should few-shot examples be static or retrieved dynamically per query? Use a static set when your input distribution is narrow and consistent, it's simpler to maintain and audit. Switch to retrieval-based selection, picking the most similar examples from a larger pool per query, when your inputs vary widely in topic, length, or difficulty.

Does the order of few-shot examples actually affect model output? Yes. Models show measurable sensitivity to example order, especially recency bias toward the last example and skew toward majority labels in classification sets. Put your strongest, most representative example last, and balance label distribution across the visible examples.

Where should few-shot examples come from? Pull them from real failures. Run your draft prompt against a realistic input set, collect the cases where the model gets it wrong, cluster those by failure type, and turn a representative case from each cluster into an example. This targets your example set at the model's actual weak spots instead of guesswork.

Can too many few-shot examples hurt performance? Yes, in two ways. First, once the pattern is well covered, extra examples add token cost and latency without improving accuracy. Second, if the extra examples are redundant or skewed toward one type of case, they can reinforce an unwanted bias, like always predicting the majority label or always producing short answers.

How do I know if my example set is actually good? Score it against a held-out test set that was never used to select the examples, using an automated grader (exact match, schema validation, or a rubric-based judge for open-ended output). Track that score over time as you version your example set, and re-run it whenever the base model or instruction text changes.