AI Infra Bench
How much real AI infrastructure engineering work can frontier models solve?
Results
Each configuration is evaluated four times per task.
| Task details | Model | Effort | Agent | ||||
|---|---|---|---|---|---|---|---|
gpt-6-sol Sep 24 | xhigh | codex 0.156.1 | 69.1% ± 10.1% | 51.2 | 50.2 | $90.86 | |
gpt-6-astra Sep 08 | xhigh | codex 0.153.4 | 64.7% ± 4.8% | 31.3 | 30.3 | $325.01 | |
gpt-6-sol Sep 24 | high | codex 0.156.1 | 61.8% ± 5.9% | 42.6 | 41.6 | $62.65 | |
gpt-6-astra Sep 08 | medium | codex 0.153.4 | 55.9% ± 5.9% | 22.3 | 21.3 | $129.31 | |
gpt-6-astra Sep 08 | high | codex 0.153.4 | 54.4% ± 5.6% | 24.7 | 23.7 | $191.36 | |
gpt-5.6-sol Sep 08 | xhigh | codex 0.153.4 | 53.7% ± 12.8% | 92.9 | 91.7 | $432.48 | |
gpt-6-sol Sep 24 | medium | codex 0.156.1 | 52.9% ± 0.0% | 38.7 | 37.7 | $46.16 | |
gpt-6-astra Sep 08 | low | codex 0.153.4 | 50.0% ± 3.4% | 18.8 | 17.8 | $92.37 | |
gpt-5.6-sol Sep 08 | high | codex 0.153.4 | 48.5% ± 5.6% | 61.2 | 60.1 | $260.95 | |
gpt-6-sol Sep 24 | low | codex 0.156.1 | 45.6% ± 12.1% | 22.6 | 21.6 | $19.18 | |
deepseek-flash Sep 24 | default | claude-code 2.1.278 | 42.6% ± 2.9% | 226.4 | 285.1 | $60.73 | |
gpt-6-luna Sep 24 | xhigh | codex 0.156.1 | 42.6% ± 5.6% | 29.5 | 28.5 | $3.52 | |
gpt-5.6-sol Sep 08 | medium | codex 0.153.4 | 41.2% ± 8.3% | 35.8 | 34.8 | $105.78 | |
gpt-5.6-sol Sep 08 | low | codex 0.153.4 | 36.8% ± 11.1% | 23.5 | 22.5 | $57.35 | |
gpt-6-luna Sep 24 | high | codex 0.156.1 | 35.3% ± 4.8% | 20.6 | 19.6 | $1.21 | |
gpt-6-luna Sep 24 | medium | codex 0.156.1 | 20.6% ± 11.3% | 14.1 | 13.1 | $0.58 | |
gpt-6-luna Sep 24 | low | codex 0.156.1 | 5.9% ± 0.0% | 26.5 | 25.5 | $1.27 |
Tasks
38 offline tasks with execution-based behavioral and e2e tests.
NeMo Gym Anthropic Responses Bridge
Bridge Anthropic Messages and OpenAI Responses for NeMo Gym coding-agent rollouts.
Pi Agent Trace
Export pi's agent loop (runs, turns, provider requests, tool executions, compactions) as an OpenTelemetry trace written as OTLP/JSON lines, with token, cache and timing attributes tied to session entries.
Pi Background Fork
Add a fork tool to pi: a background child pi process that inherits the conversation so far, works on a task while the parent continues, and reports its result back to the parent session exactly once.
Pi Background Processes
Add a background-process extension to pi whose processes can wake the agent, survive session replacement, and never outlive pi.
Pi Context Management
Add model-controlled context windows, persistent notes, and selective history recovery to one Pi session.
Pi Plan Mode
Enhance pi's Plan Mode extension with revision-bound approval, interactive and RPC controls, exact tool restoration, and normal session resume.
Page 1 of 7. Showing tasks 1 to 6 of 38.