OpenAI Codex for Test Generation

Why test generation is the perfect Codex use case

Most engineers who try an AI coding agent for the first time reach for it to write features. That's the wrong starting point. Feature code carries product ambiguity, changing requirements, and design decisions that a model can't fully own. Tests are different. A test has one job: prove that a specific behavior holds under specific conditions. That narrow, verifiable scope is exactly where OpenAI Codex — the CLI agent that reads your repo, plans, edits files, and runs commands in a loop — earns its keep.

When you ask Codex to write a feature, you're trusting it to make judgment calls. When you ask it to write tests for code you already trust, you're asking it to do something closer to mechanical translation: read the function, enumerate the paths through it, and encode each path as an assertion. That's a task where an agent that can read your entire codebase, run your test suite, and iterate on failures has a real structural advantage over pasting a function into a chat window.

This article is a working guide to using Codex for test generation — unit tests, integration tests, edge cases, and regression tests for bugs you've just fixed. We'll go through setup, prompting patterns, example sessions, how to handle flaky or wrong tests, and how to fold this into a CI-friendly workflow. If you want a structured, hands-on walkthrough beyond what a blog post can cover, our OpenAI Codex CLI Tutorial course goes deeper into the agent loop, sandboxing, and multi-file workflows.

Getting Codex CLI ready for a test-writing session

Before Codex can write useful tests, it needs three things: access to your repo, awareness of your test framework conventions, and permission to actually run commands. Install and authenticate first.

npm install -g @openai/codex
codex login

Once installed, run codex from inside your project root, not from a scratch folder. Codex builds context by reading files in your working directory, and it works best when it can see your actual source tree, existing tests, and config files like package.json, pytest.ini, or jest.config.js.

A detail that trips people up: Codex operates in different approval modes. In interactive mode it will ask before running commands or editing files. For a test-generation session, you generally want it to be able to run your test suite without asking every single time, so you can watch it iterate. You can scope this per-session:

codex --approval-mode auto-edit

This mode lets Codex edit files freely but still prompts before running arbitrary shell commands outside the test runner. If you're comfortable letting it run your test command in a loop unattended (recommended once you trust the workflow), you can go further:

codex --full-auto

Full-auto is best used inside a container or a disposable branch — not because Codex is reckless, but because any agent that can execute shell commands should be sandboxed as a matter of hygiene, the same way you wouldn't let a CI job run with root access to your laptop.

The other setup step people skip: give Codex a project-level instructions file. Codex reads a file named AGENTS.md (in the same spirit as CLAUDE.md for Claude Code) if present in your repo root. This is where you tell it your testing conventions before it ever writes a line.

# AGENTS.md

## Testing conventions
- Test framework: pytest
- Test files live in `tests/`, mirroring the `src/` structure
- Use `pytest.mark.parametrize` for multiple input cases instead of separate test functions
- Mock external HTTP calls with `responses`, never real network calls
- Every new test must run with `pytest -q` and pass before you consider the task done
- Do not modify source files while writing tests unless explicitly asked

That last line matters more than it looks. Left unconstrained, an agent asked to "make the tests pass" will sometimes take the shortcut of changing the source code instead of writing a correct test. Explicitly separating "write tests" from "fix code" keeps the agent honest about which side of that line it's on.

The basic loop: generate, run, fix, repeat

The core value of Codex over a plain chat-based assistant is that it doesn't just write a test file and stop — it runs the tests, reads the failure output, and revises. This loop is worth understanding explicitly because it changes how you prompt.

A typical session looks like this:

1. You point Codex at a file or function and ask for tests.
2. Codex reads the target code plus any related files (types, imports, existing tests for similar modules).
3. It writes a test file.
4. It runs the test suite (or just the new file).
5. If tests fail because of a bug in its own test (wrong expected value, missing fixture, import error), it reads the traceback and fixes the test.
6. If tests fail because of an actual bug in your source code, a well-instructed Codex will flag this rather than silently "fixing" your code to match a wrong assumption.

Here's a realistic prompt for step 1, using a small Python module as the target:

# src/pricing.py
def calculate_discount(price: float, quantity: int, member: bool = False) -> float:
    if price < 0 or quantity < 0:
        raise ValueError("price and quantity must be non-negative")
    discount = 0.0
    if quantity >= 10:
        discount += 0.10
    if quantity >= 50:
        discount += 0.05
    if member:
        discount += 0.05
    discount = min(discount, 0.30)
    return round(price * quantity * (1 - discount), 2)

A prompt that gets good results:

Write pytest tests for src/pricing.py in tests/test_pricing.py.
Cover: no discount tier, each discount tier boundary (9 vs 10, 49 vs 50 units),
member discount stacking with quantity discounts, the 30% discount cap,
and the ValueError path for negative inputs. Use parametrize where the
cases share structure. Run the tests after writing them and fix any failures.

Notice this prompt does the enumeration work partially for the model — it names the boundaries (9 vs 10, 49 vs 50) instead of just saying "test edge cases." You'll get better coverage from Codex, as from any model, when you spell out the boundary conditions you already know matter, and let it fill in the ones you didn't think of.

Writing prompts that produce real edge cases, not filler

The most common complaint about AI-generated tests is that they're padding — five tests that all exercise the happy path with slightly different numbers, and nothing that would actually catch a regression. This isn't a Codex-specific problem, it's a prompting problem, and it's fixable with structure.

Instead of "write tests for this function," decompose the ask into categories:

• Boundary values: the exact edges of any conditional (0, -1, off-by-one on loop bounds, the largest/smallest allowed input)
• Type and null handling: empty strings, empty lists, None/null, unexpected types if the language doesn't enforce them
• Error paths: every raise, throw, or early return that signals failure
• State interactions: for stateful code, sequences of calls that matter (create-then-delete, double-submit, calling a method twice)
• Concurrency-adjacent behavior: if the code touches shared state, async operations, or retries, ask specifically for tests around race conditions or idempotency

Here's a prompt shaped around that decomposition, this time for a TypeScript function with more surface area:

You are writing Jest tests for the function below. Structure your test
file into four describe blocks: "happy path", "boundary values",
"error handling", and "idempotency". Do not put more than one assertion's
worth of new behavior into a single test — one concept per test block.

async function reserveSeat(eventId: string, userId: string): Promise<Reservation> {
  const event = await db.events.findById(eventId);
  if (!event) throw new NotFoundError(`event ${eventId} not found`);
  if (event.seatsRemaining <= 0) throw new SoldOutError(eventId);

  const existing = await db.reservations.findByUserAndEvent(userId, eventId);
  if (existing) return existing;

  const reservation = await db.reservations.create({ eventId, userId });
  await db.events.decrementSeats(eventId);
  return reservation;
}

Codex, given this, typically produces something like the structure below (trimmed for length, but this is the real shape you should expect):

describe("reserveSeat", () => {
  describe("happy path", () => {
    it("creates a reservation and decrements seats when a seat is available", async () => {
      // arrange: mock event with seatsRemaining = 5
      // act, assert reservation returned and decrementSeats called once
    });
  });

  describe("boundary values", () => {
    it("throws SoldOutError when seatsRemaining is exactly 0", async () => {
      // ...
    });
    it("still allows reservation when seatsRemaining is exactly 1", async () => {
      // ...
    });
  });

  describe("error handling", () => {
    it("throws NotFoundError when the event does not exist", async () => {
      // ...
    });
  });

  describe("idempotency", () => {
    it("returns the existing reservation without decrementing seats again on a repeat call", async () => {
      // ...
    });
  });
});

That idempotency test is the one a rushed human writer skips and a rushed prompt to any LLM also skips — it only shows up because the prompt explicitly asked for a describe block dedicated to it. This is the pattern worth internalizing: Codex is good at filling in tests within a category you name, and noticeably weaker at inventing categories you didn't ask for. Do the taxonomy work yourself, delegate the instance-generation work to the agent.

Using Codex to write regression tests from a bug report

One of the highest-leverage uses of Codex isn't greenfield test writing — it's turning a bug report into a permanent regression test before or while you fix the bug. The workflow:

1. Reproduce the bug and paste the reproduction steps or stack trace to Codex.
2. Ask it to write a failing test that captures the bug, without touching the source code.
3. Confirm the test fails for the right reason.
4. Only then ask Codex (or yourself) to fix the underlying code, using that test as the pass/fail signal.

This order matters. If you let the agent fix the bug first and write the test afterward, there's a real risk it writes a test that's shaped around whatever fix it already made, rather than one that would have caught the original bug regardless of how it's fixed.

Bug report: calling normalize_email(" John.Doe+promo@GMAIL.com ") returns
"john.doe+promo@gmail.com" but our billing system treats "+promo" tags
as separate accounts, which is wrong for Gmail addresses (Gmail ignores
everything after "+" in the local part).

Write a failing test in tests/test_email.py that encodes the expected
behavior: Gmail addresses should have the +tag stripped before dedup
comparison, non-Gmail addresses should not. Do not modify normalize_email
itself yet — just get the failing test in place.

A reasonable Codex output:

import pytest
from src.email_utils import normalize_email

class TestNormalizeEmailGmailTagging:
    def test_gmail_plus_tag_is_stripped_for_dedup(self):
        result = normalize_email(" John.Doe+promo@GMAIL.com ")
        assert result == "johndoe@gmail.com"

    def test_non_gmail_plus_tag_is_preserved(self):
        result = normalize_email("Jane.Doe+promo@outlook.com")
        assert result == "jane.doe+promo@outlook.com"

    def test_gmail_dots_are_ignored_but_plus_tag_still_stripped(self):
        result = normalize_email("j.o.h.n+newsletter@gmail.com")
        assert result == "john@gmail.com"

Run it, confirm it fails against the current implementation, and now you have both a specification and a safety net in one artifact. This is a materially better workflow than fixing first and testing later, and it's one of the clearest cases where an agentic tool changes your process, not just your typing speed.

Integration tests: giving Codex enough context to not hallucinate fixtures

Unit tests are relatively contained — Codex can usually infer what to mock from the function signature and imports. Integration tests are where things go sideways if you don't set up context properly, because the agent needs to know about database schemas, existing fixtures, test containers, and how your app wires dependencies together.

The fix is to point Codex at your existing integration test setup before asking for new tests, rather than trusting it to reverse-engineer conventions from a single file.

Before writing anything, read tests/integration/conftest.py and
tests/integration/test_orders.py to understand our fixture patterns
(the test_db fixture, the client fixture, and how we seed data with
factory functions). Then write an integration test for the new
POST /api/refunds endpoint in src/api/refunds.py, following the same
fixture and factory conventions. Use the existing OrderFactory and
UserFactory rather than constructing raw dicts.

This single instruction — read the existing pattern first — eliminates the majority of "Codex invented a fixture that doesn't exist" failures. Agentic tools are only as good as the context they gather, and unlike a chat assistant with a fixed context window you paste into, Codex CLI can actually go read three or four files before writing anything, provided you tell it to.

A trimmed example of what comes back:

import pytest
from tests.factories import OrderFactory, UserFactory

class TestRefundEndpoint:
    def test_full_refund_creates_refund_record_and_restores_inventory(self, client, test_db):
        user = UserFactory()
        order = OrderFactory(user=user, status="completed", total=4999)

        response = client.post(f"/api/refunds", json={"order_id": order.id, "amount": 4999})

        assert response.status_code == 201
        refund = test_db.query(Refund).filter_by(order_id=order.id).first()
        assert refund is not None
        assert refund.amount == 4999

    def test_partial_refund_leaves_order_status_as_partially_refunded(self, client, test_db):
        user = UserFactory()
        order = OrderFactory(user=user, status="completed", total=10000)

        response = client.post("/api/refunds", json={"order_id": order.id, "amount": 3000})

        assert response.status_code == 201
        test_db.refresh(order)
        assert order.status == "partially_refunded"

    def test_refund_exceeding_order_total_is_rejected(self, client, test_db):
        order = OrderFactory(status="completed", total=5000)

        response = client.post("/api/refunds", json={"order_id": order.id, "amount": 9000})

        assert response.status_code == 422

Notice it reused OrderFactory and UserFactory instead of inventing new setup code — that's the payoff of the "read first" instruction.

Measuring whether the generated tests are actually good

Getting Codex to produce a green test suite is the easy part. The harder question: are these tests actually worth keeping? A test suite full of assertions that merely restate the implementation gives you false confidence, and false confidence is worse than no tests, because it changes how carefully humans review the code path later.

A few checks worth running on any batch of AI-generated tests before merging them:

• Mutation-adjacent sanity check: temporarily break the source code (flip a comparison operator, remove a branch) and confirm the new tests fail. If they don't, the test isn't testing what you think it's testing. Tools like mutmut (Python) or stryker (JS/TS) automate this, but even a manual "break it on purpose" pass catches a lot.
• Read the assertions, not just the test names: a test named test_handles_empty_input_correctly that asserts result is not None isn't actually verifying correctness — it's verifying the function didn't crash. Ask Codex directly if a test seems weak: "does this assertion actually verify the discount cap logic, or just that no exception was raised?" It's decent at self-critiquing when prompted specifically.
• Check for tautological mocks: a failure mode specific to AI-written integration tests is mocking so aggressively that the test only proves the mock was called, not that the real logic behaves correctly. If every dependency is mocked and the assertion is mock.assert_called_once(), ask for a rewrite with fewer mocks.
• Look for coverage of the thing that actually broke last time: if your team has a history of a specific bug class (off-by-one in pagination, timezone bugs, race conditions in webhook handlers), explicitly ask Codex whether the new tests cover that class of failure. It won't proactively know your team's history unless you tell it.

None of this is Codex-specific hygiene — it's the same discipline you'd want from a junior engineer's test PR. The difference is that Codex will produce a large volume of syntactically plausible tests very quickly, which means the review bottleneck shifts from "did they write tests" to "are these tests worth anything," and that review has to be deliberate.

A workflow for folding this into daily development

The pattern that works well in practice, after enough sessions to see what breaks, looks like this:

1. Write or accept the implementation code first (from a human, from Codex, doesn't matter).
2. Before opening a PR, run a dedicated Codex session scoped only to test generation for the diff: codex "write tests for the changes in this branch, following AGENTS.md conventions".
3. Review the generated tests using the checklist above — mutation sanity check, assertion quality, mock tautology check.
4. Commit the tests you keep, discard or rewrite the ones that are padding.
5. For any bug fix, always generate the regression test before the fix, not after, using the workflow from the bug-report section.

A minimal shell alias that makes step 2 a one-keystroke habit:

alias codex-test='codex --approval-mode auto-edit "Write tests for the uncommitted changes in this repo. Follow AGENTS.md conventions. Run the test suite after writing and fix any failures caused by the tests themselves, but do not modify application source code."'

Running this before every PR turns test generation from an occasional chore into a default step, the same way git diff before a commit becomes automatic once it's one command away.

Common pitfalls and how to avoid them

A few failure patterns show up often enough to call out directly:

• The agent "fixes" your source code to pass a wrong test. This happens when you don't explicitly forbid source edits during test generation. Always separate the "write tests" instruction from the "fix code" instruction into different sessions or explicit permission changes.
• Tests pass locally but the agent never actually ran them. If you're in a mode where Codex can't execute commands (some restricted sandboxes, or if your test runner isn't installed in its environment), it may write plausible-looking tests without ever confirming they run. Always verify in your own CI or terminal before trusting a session that claims "all tests pass."
• Over-mocking that defeats the purpose of integration tests. Covered above, but worth repeating: if you asked for an integration test and got something that mocks every external call, that's not an integration test, it's a unit test wearing a costume.
• Flaky tests from unseeded randomness or real timestamps. If your code uses datetime.now() or random IDs, tell Codex up front to use frozen time and seeded randomness (freezegun, fixed random.seed()), or you'll get intermittent CI failures that are annoying to trace back to AI-generated test code.
• Skipping the "read existing conventions first" step. This is the single highest-leverage instruction you can add to any test-generation prompt, and it's the one people forget under time pressure.

Closing thoughts

Test generation is one of the few areas where handing real autonomy to an AI agent is a low-risk, high-leverage trade. The blast radius of a bad test is small — worst case, you delete it — while the time saved on the repetitive parts of writing thorough coverage (boundary values, error paths, regression tests for bugs you've already diagnosed) is substantial. The quality bar isn't "did Codex write something" — it's whether you did the taxonomy work up front, gave it your project's real conventions through an AGENTS.md file, and reviewed the assertions rather than just the green checkmark.

If you want to go deeper into the agent loop itself — sandboxing modes, multi-file refactors, how Codex plans before editing, and how to structure larger autonomous sessions safely — our OpenAI Codex CLI Tutorial course walks through all of it hands-on, with real repositories and real failure cases, not just toy examples.