OpenAI Codex Benchmark: Build a Repeatable Test
Build a repeatable Codex benchmark with a fixed task, clear acceptance checks, preserved outputs and a record of retries. Understand what a pelican test measures.
Last updated: 2026-09-18
Codex Pulse preserves two different experiments: 12 static SVG attempts from September 15, 2026, and 3 HTML animation attempts from September 16. They answer different questions. Combining their outputs into one leaderboard would hide changes in the prompt, instructions and scoring.
A useful Codex benchmark starts with a question you can check against the saved output. For a drawing, that might be whether the bicycle renders. For a code change, it might be whether a failing test now passes without breaking another behavior. A nice drawing and a working patch require different evidence.
Choose the outcome before choosing the prompt
Start with a task that resembles something you actually need Codex to do. Keep it small enough to rerun from a known starting state.
| Question | Task | Evidence to preserve |
|---|---|---|
| Can it produce this visual artifact? | Generate one SVG or HTML animation | Original output and a consistent browser capture |
| Can it fix this bug? | Repair a reproducible failure in a disposable checkout | Starting revision, patch and relevant test results |
| Does it finish the requested work? | Make a change and run named checks | Tool results, remaining work and requests to continue |
| Did my workflow become slower? | Repeat the same bounded task | Total duration, tool duration where available and interruptions |
A public benchmark can help you decide what to investigate. A task from your own workflow is usually more useful for deciding whether a setting works for that workflow. The cost is maintenance: you need a stable starting state and checks that still represent real work.
Write the acceptance criteria first
For a coding task, specify the behavior that must change, behavior that must remain intact, and the checks that establish completion. Preserve an isolated starting checkout so that every attempt faces the same task.
An example task brief is:
Fix the supplied failing case. Preserve the existing public behavior, run the named checks, and report any work that remains incomplete.
Replace “supplied failing case” and “named checks” with actual inputs and commands. This brief is a test fixture, not a guarantee that the agent will follow it. Score what happened, including omitted checks.
For a visual task, define visible criteria independently of taste. In the pelican benchmark, you can inspect whether the bird is recognizable, whether the bicycle has connected wheels and a frame, and whether the rider connects plausibly to the bicycle. If animation is required, watch the animation as well as its still frame.
Keep the conditions comparable
Record the exact prompt, requested model, reasoning effort, client version, starting revision, active instructions and available tools. Use fresh conversations. If comparing two settings, change one setting and preserve the other conditions.
Configuration deserves a check of its own. OpenAI documents both user and project configuration layers; command-line overrides can take precedence. A value in one file may therefore differ from the setting used by a run. See official configuration basics.
Keep failed attempts and outputs you dislike. If you provide a correction or ask the agent to continue, record that intervention. An initial response and an artifact produced after several follow-ups should have separate labels.
Use a result record you can audit
Save this information alongside each attempt:
Run ID and UTC date:
Exact prompt or prompt file hash:
Requested model and reasoning effort:
Client version and configuration differences:
Starting revision or input files:
Acceptance checks and actual results:
Original output or patch:
Elapsed time and known interruptions:
Tools called:
Retries or manual follow-ups:
Known limitations:
The selected model label records your request. It does not independently identify the backend snapshot. Remove credentials and private material before sharing a record publicly.
Repeat each condition several times if your budget permits, and choose the number of attempts before inspecting the results. Small batches are useful for finding reproducible failure modes. They provide weak evidence for a broad ranking, especially when the task itself is narrow.
Read the Codex Pulse examples within their scope
The September 15 archive contains 12 attempts using the static pelican SVG prompt: two requested models, two efforts and three attempts per combination. All produced SVGs. Tools and runtime instructions remained available, and some runs used tools. This series has no matched earlier baseline for a degradation claim.
The September 16 archive contains three first attempts at a different HTML animation prompt. Its static scores describe the output under one recorded heuristic ruleset. They do not measure general coding ability or independently establish visual correctness.
Use the gallery and benchmark explanation to inspect examples, then choose one task with acceptance checks from your own work. If your concern is a recent change in behavior, start with the quality diagnosis guide. The next useful evidence is a repeat of the same task with the conditions recorded.