Meta-Prompting: Using LLMs to Write Better Prompts

Meta-prompting is the practice of using an LLM to generate, critique, or rewrite prompts for another LLM call, instead of hand-editing wording yourself until it works. You give a model your task, your failure cases, and maybe a rubric, and it produces (or repairs) the prompt that will actually run in production. It sounds like a shortcut, but in practice it's a more disciplined version of what good prompt engineers already do: treat the prompt as an artifact with inputs, outputs, and a feedback loop, and let the thing that's good at language do the language work.

This matters because most teams write prompts the way they write regex: by trial and error, staring at outputs, tweaking a sentence, rerunning, repeating. That works for a five-line prompt. It falls apart once you have a system prompt with twelve constraints, three few-shot examples, and a JSON schema, and someone on the team "just adds a line" that breaks something else. Meta-prompting turns prompt writing into a task you can delegate, version, and test, the same way you'd delegate a code review or a first draft of a function.

What meta-prompting actually is

At the core, meta-prompting is a two-model (or two-call) pattern:

1. A generator prompt describes the task you want solved: what the target LLM should do, its inputs, its constraints, and what "good" looks like.
2. A meta call takes that description, plus optionally some failing examples or a scoring rubric, and produces or revises the actual target prompt.
3. You run the target prompt against test cases, collect failures, and feed them back into the meta call.

This is different from just asking a chatbot "write me a prompt for X" once and copy-pasting the result. That's meta-prompting's laziest form, and it's still better than nothing, but the real value shows up when you close the loop: generate, test, diagnose, revise, retest. Anthropic's own documentation on prompt engineering explicitly recommends this pattern for Claude: draft a prompt, run it against a representative eval set, and use a second model call to analyze the failures and propose a rewrite.

There's a useful three-way split worth keeping in your head:

• Prompt generation: cold-starting a prompt from a task description. Useful when you're facing a blank page and don't know the right structure (system vs. user split, XML tags vs. markdown, where to put examples).
• Prompt critique/repair: you have a working-ish prompt and some bad outputs, and you want a model to diagnose why and rewrite the offending section.
• Prompt compression/adaptation: you have a long, over-specified prompt (often written by a human who kept bolting on edge cases) and you want it distilled to the minimum instructions that still pass your eval set, or ported to a different model family with different conventions.

All three use the same mechanism, an LLM reasoning about a prompt as text, but they're solving different problems and you should be explicit with yourself about which one you're doing.

Why this works better than manual tuning

Three reasons, none of them magic.

LLMs are good at pattern-matching prompt failure modes. If you paste in a system prompt and five transcripts where the model ignored an instruction, a capable model can usually spot that the instruction was buried on line 40 after eight other instructions, or that it conflicts with an earlier line, or that the desired output format was described in prose instead of shown as an example. Humans staring at their own prompt for the tenth time develop blind spots. A fresh model call doesn't have that context poisoning.

It forces you to externalize your rubric. You can't ask a model to improve a prompt against "make it better." You have to say what better means: fewer refusals, stricter JSON validity, shorter outputs, no hedging language. Writing that rubric down is valuable on its own, independent of whether an LLM ever touches it, because it turns "this feels off" into something you can grep for.

It's cheap to iterate. A meta-prompting loop is a handful of API calls. Manually rewriting a prompt, deploying it, and waiting for user feedback is a day or a week. If you have even a small eval set (20-50 representative cases), you can run generate -> test -> critique -> revise dozens of times in an afternoon, which is simply not something a human editing by hand can match.

None of this replaces judgment. A meta-prompt can happily "fix" a prompt by making it pass your flawed eval set while making the actual behavior worse. The loop is only as good as what you're measuring.

The core loop, concretely

Here's the pattern in its most useful shape, with an actual worked example. Say you're building a support-ticket classifier: given a customer message, output one of billing, bug, feature_request, other, as strict JSON.

Step 1: Write a task spec, not a prompt.

Don't start by writing the prompt yourself. Write down what the prompt needs to accomplish:

Task: classify a support ticket into one of four categories.
Input: raw customer message (1-500 words, may include profanity, may be
  in a non-English language, may reference a prior ticket).
Output: strict JSON, one key "category", value is exactly one of
  billing | bug | feature_request | other.
Constraints:
  - Never output anything other than the JSON object.
  - If ambiguous, prefer "other" over guessing.
  - Multi-issue tickets should be classified by the primary issue only.
Known failure modes we've seen from a naive first draft:
  - Model adds explanation text before the JSON.
  - Model invents a fifth category like "complaint".
  - Non-English tickets get misclassified as "other" even when clear.

Step 2: Have a model generate the first draft prompt.

Feed that spec to Claude (or whichever model you're targeting) with an instruction like:

You are an expert prompt engineer. Given the task specification below,
write a system prompt for an LLM that will perform this classification
task in production. Use clear structure (XML tags for sections is fine).
Include 2-3 few-shot examples that cover the known failure modes listed.
Output only the prompt text, nothing else.

<task_spec>
...paste the spec from step 1...
</task_spec>

You now have a first-draft prompt you didn't hand-write.

Step 3: Build a small eval set and run it.

You need real or realistic examples, 20 is a reasonable floor, 50-100 is better. For each, you know the correct category. Run the generated prompt against all of them programmatically:

import anthropic
import json

client = anthropic.Anthropic()

def classify(system_prompt, ticket_text):
    resp = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=100,
        system=system_prompt,
        messages=[{"role": "user", "content": ticket_text}],
    )
    return resp.content[0].text

results = []
for ticket, expected in eval_set:
    raw = classify(current_system_prompt, ticket)
    try:
        got = json.loads(raw)["category"]
    except (json.JSONDecodeError, KeyError):
        got = "PARSE_ERROR"
    results.append({
        "ticket": ticket,
        "expected": expected,
        "got": got,
        "correct": got == expected,
    })

accuracy = sum(r["correct"] for r in results) / len(results)
print(f"accuracy: {accuracy:.2%}")
failures = [r for r in results if not r["correct"]]

Step 4: Feed the failures back into a critique call.

This is the actual meta-prompting step. Don't rewrite the prompt yourself, hand the failures to a model:

You wrote the system prompt below for a ticket classification task.
It scored 78% accuracy on our eval set. Here are the cases it got wrong,
with the ticket text, the expected category, and what it actually
output. Diagnose the likely cause of each failure category (not just
each individual failure) and produce a revised system prompt that fixes
these failure modes without breaking the cases it already gets right.

<current_prompt>
...
</current_prompt>

<failures>
...paste the `failures` list, formatted...
</failures>

Output the full revised prompt, then a short bullet list explaining
what you changed and why.

That "explain what you changed and why" instruction matters. It turns the meta-prompt into something reviewable, not a black box swap. You read the bullets, decide if the reasoning is sound, and only then adopt the new prompt.

Step 5: Rerun the eval, diff the failures, repeat.

If accuracy went up and the failure set shrank, keep the new prompt and loop again on the remaining failures. If accuracy went up but a previously-passing case now fails, that's a regression worth looking at by hand, sometimes the fix for one failure mode overcorrects into causing a different one (classic "fixed the false negative, introduced a false positive" pattern).

Two or three iterations of this loop typically gets you further than a full day of manual tweaking, mostly because the model doesn't get tired and doesn't anchor on its own previous phrasing the way a human editor does.

Patterns worth knowing

Prompt-to-prompt distillation. If you have a long, battle-tested prompt (say, 2000 tokens of accumulated edge-case handling) and you want a shorter version for a cheaper or faster model, ask a model to compress it while explicitly preserving behavior on your eval set. Give it the eval set as ground truth, not just the prompt text, otherwise compression tends to silently drop the instructions that look redundant but were actually load-bearing for a specific edge case.

Cross-model porting. A prompt tuned for one model family often underperforms on another because of stylistic conventions (heavy XML-tag structuring, positive vs. negative instructions, how examples are formatted). Rather than manually rewriting, ask the target model itself: "here is a prompt written for [model A], rewrite it to follow best practices for [model B] while preserving the same task and constraints." Models are generally decent at self-reporting the conventions they respond best to.

Adversarial self-critique. Before you even have failure data, you can ask a model to red-team its own draft prompt: "list five ways a user could misuse or an edge case could break this prompt, then propose fixes for each." This catches obvious gaps (no handling for empty input, no handling for a fifth unlisted category, no instruction for what to do if the input is in a different language) before you burn eval budget discovering them the slow way.

Rubric-driven revision for open-ended tasks. Classification has a clean right answer. Summarization, tone, and creative-writing prompts don't. For those, write a scoring rubric (5-8 named criteria, each 1-5) and have a separate model call score outputs against it, then feed low-scoring examples plus the specific criteria they failed into the meta-prompt. This is essentially building a lightweight LLM-as-judge and using its verdicts as the feedback signal.

Keep a prompt changelog. Every time a meta-prompting pass changes the production prompt, save the old version, the new version, the eval score delta, and the bullet-point rationale the model gave you. This is the single highest-leverage habit in this whole workflow: six months from now when something regresses, you want to be able to git blame your prompt the same way you blame code, not reconstruct history from Slack messages.

Where it breaks down

Meta-prompting is not free lunch, and it fails in specific, recognizable ways.

Garbage eval set, garbage prompt. If your 20 test cases don't represent real production traffic, the loop will happily optimize a prompt that's great at your eval set and mediocre in the wild. Pull real examples, including the ugly ones, users pasting HTML, messages in three languages in one ticket, sarcasm.

Overfitting to specific phrasing. A meta-prompt revision that adds a few-shot example matching your exact failure case sometimes just teaches the model to pattern-match that literal example rather than generalizing the underlying rule. Watch for a fixed failure that was really just memorized, not understood, by checking a fresh, never-seen batch after each revision, not just the same failure set you fed in.

Prompt bloat. Left unchecked, the critique-and-revise loop tends to grow prompts, each fix adds another sentence, another constraint, another example. Long prompts cost more tokens, add latency, and past a certain point actually reduce instruction-following reliability because critical constraints get diluted among restated ones. Periodically run the compression pattern above to prune back to essentials.

It won't fix a task that's underspecified. If you genuinely don't know what "good" output looks like for a task, no amount of meta-prompting will discover it for you. The model can only optimize against the rubric or eval set you give it. Garbage-in-garbage-out applies to the meta layer just as much as the base layer.

Latency and cost of the loop itself. Running full eval passes on every iteration costs real tokens, especially with larger eval sets and slower models. For high-stakes prompts (anything customer-facing, anything feeding a decision with consequences) this cost is trivial next to the cost of shipping a bad prompt. For quick internal scripts, it's overkill, just edit the three lines by hand.

A minimal template you can reuse

If you want a starting point rather than building this from scratch, this is the shape that holds up across tasks:

ROLE: You are an expert prompt engineer improving a production prompt.

CURRENT PROMPT:
<paste current system/task prompt>

TASK CONTEXT:
<one paragraph: what this prompt is for, who calls it, what "success"
means for the business, not just the model output>

EVAL RESULTS:
<paste failing cases: input, expected output, actual output>
<optionally paste a sample of PASSING cases too, so the model knows
what not to break>

INSTRUCTIONS:
1. Diagnose the failure modes (group similar failures, don't list
   each one as unrelated).
2. Propose a revised prompt that addresses the diagnosed causes.
3. Explicitly note anything in the current prompt you preserved on
   purpose because it's handling a passing case.
4. Output the full revised prompt in a code block, followed by a
   bullet list of changes and rationale.

Wire that into whatever eval harness you already have, even a bare Python loop like the one above, and you have a repeatable prompt-improvement pipeline instead of a folder of prompt_v7_final_FINAL.txt files.

FAQ

Does meta-prompting need a different model than the one running the actual task? No, but it often helps to use your most capable available model for the meta layer even if the target task will run on a smaller, cheaper model. The meta call only runs during development, not in production traffic, so its cost is a rounding error, and a stronger model tends to give sharper diagnosis of failure modes.

How is this different from just using few-shot examples in my prompt? Few-shot examples are one ingredient a prompt can contain. Meta-prompting is the process of deciding what should go in the prompt, including whether few-shot examples are needed at all, by having a model reason about your task and failures rather than you guessing.

Can I automate the whole loop end-to-end without a human in it? You can automate generate-test-critique-revise as a script, and some teams do run it unattended for several iterations. But someone should review the final prompt before it ships, models optimizing against an eval set can find degenerate solutions (like a prompt that nudges outputs toward whatever's statistically likely in the eval set rather than genuinely solving the task) that a five-minute human read catches immediately.

What's a reasonable eval set size to start with? 20-30 cases is enough to start iterating and catch obvious problems. Treat it as a floor, not a target, keep adding real production failures to it over time so it stays representative instead of ossifying around your first guesses.

Does this replace prompt engineering skill, or does it require it? It requires it. You still need to know what a well-structured prompt looks like to judge whether the meta-prompt's output is actually good, and you still need to write a clear task spec and rubric, that's the hard part. Meta-prompting removes the tedious hand-editing labor, not the judgment.

Is this the same as "self-improving prompts" or prompt optimization frameworks like DSPy? Related but not identical. Automated prompt-optimization frameworks typically search over prompt variants against a metric using algorithmic methods (sometimes including LLM calls as one component). Meta-prompting as described here is the simpler, manual-in-the-loop version: you decide when to iterate, you review each revision, and you keep a human in the review step. You can absolutely combine the two, use an optimization framework for the search and a meta-prompting critique pass as a final human-reviewable polish step.