Frontier-Bench

Built by the team behind Terminal-Bench and Harbor, Frontier-Bench (which started out as Terminal-Bench 3.0) is an on-going large-scale community effort to source the most diverse, difficult and high-quality tasks.

Our first release contains 74 tasks across 7 domains. The best models achieve ~34% on Frontier-Bench v0.1.

Frontier-Bench is a continuous benchmark. The frontier is wide and dynamic, Frontier-Bench evolves along with it. We are already developing subsequent releases, with new and improved tasks.

Overview

Frontier-Bench expands on Terminal-Bench with more challenging tasks across a wider distribution of valuable work. Frontier-Bench also introduces a new way to continuously improve benchmarks with CI/CD, semantic versioning, and result migrations.

We need a harder benchmark. Many Terminal-Bench tasks have become saturated and leaderboard entries are condensed into a narrow range that may not fully represent capability gaps across models.

We need a more representative benchmark. Terminal-Bench was built on the thesis that code and the CLI were powerful tools for agents to accomplish tasks. Frontier-Bench expands on that idea by introducing more task diversity to measure performance across more types of coding and non-coding work. "Work" has expanded to longer-running tasks across a greater diversity of domains and "computer" has expanded to richer environments, multi-container networks, and GPUs. We anticipate this definition continuing to broaden in future Frontier-Bench versions.

We're fixing the way benchmarks are built. Many benchmarks have known problems, many of these problems go unfixed. It erodes the usefulness of the benchmark and also dilutes the signal. Frontier-Bench is a continuously improving benchmark and open community project. We will regularly release updates to fix and add tasks, making the benchmark more valuable over time.

Tasks

Frontier-Bench was built using our learnings from building Terminal-Bench. Each task goes through a rigorous review process with dense human and agent feedback at each step. Frontier agent, oracle agent, and cheating agent runs ensure that the task is well defined. Domain experts review tasks to probe instruction/verifier alignment, task realism, and manage under/over specification risks.

Frontier-Bench is an open internet benchmark. Capabilities are best demonstrated when agents have access to their full set of tools. We instruct agents not to look for task-specific solutions or hints online, which is surprisingly effective. We also tell the agent how long it has to complete the task.

We separate the agent container and verifier container. Artifacts from the agent container are downloaded at the end of the trial, logged, and uploaded to the verifier container. This prevents many reward hacking vectors allowing for re-grading when task verifiers are updated. The diversity of artifact filetypes reflects the expansion into new domains:

• Database snapshots (.sql, pg.dump, redis.rdb, Kafka .tgz)
• ML checkpoints (.pt, .safetensors)
• RTL designs (.v)
• Formal proofs (Lean .lean, Coq .v)
• Source code and patches (Python, C, CUDA, TypeScript, etc.)
• VM images (.qcow2, .fd)
• Compiled binaries (.jar, ELF executables)
• Music scores (.mxl, .musicxml)
• CAD files (.step, .FCStd)
• Scientific analyses (.json, .csv, .tsv)
• Quantitative computations (.json, .csv, .xlsx)
• Reverse-engineered specs (.json)
• Business-process decisions (.json, .jsonl)

Results

Frontier-Bench discriminates better between frontier and sub-frontier models than Terminal-Bench 2.1. For example, Fable 5 in Claude Code and Opus 4.8 in Claude Code are separated by only 4.9 points on Terminal-Bench v2.1, but 12.7 points on Frontier-Bench v0.1.

While frontier models Fable 5 and GPT-5.6 Sol achieve similar performance on Frontier-Bench v0.1, cost and token usage varies significantly. GPT-5.6 Sol is ~40% cheaper and uses ~50% fewer tokens than Fable 5. All of the GPT-5.6 models (Sol, Terra, Luna) appear on the cost vs pass rate Pareto front for the models we ran.¹

Agents approach problem solving differently, even at similar pass rates. Fable 5 (33.8%) and Opus 4.8 (21.1%) use more tokens and take fewer actions. Meanwhile, GPT-5.6 Sol (34.4%) and Terra (20.8%) use less tokens and take more actions.

Output Tokens vs Steps, Per TrialFrontier-Bench

Roadmap

The Frontier-Bench team is already planning upcoming releases, which we are documenting in our public roadmap. In our next major release, we plan to add new tasks that measure new capabilities, add greater task diversity, and tune timeouts. Tasks are versioned so corresponding trials can be re-used, re-graded, or re-run to minimize the cost and complexity of updating leaderboard results. All of these features are simple commands in Harbor.

Conclusion

Frontier-Bench is a community effort from 100+ task contributors and reviewers. It is the hardest and highest quality benchmark we could make together in the open, but it is just the start. Let's find issues, fix them, and add new tasks together. Let's make this benchmark measure the most valuable frontier capabilities in the most accurate way possible.

Join the effort on Discord and GitHub. Let's continuously and rigorously define the frontier together.

Acknowledgements

Thank you to all the task contributors and reviewers.

Special thanks to our senior reviewers Ivan Bercovich, Benedikt Droste, Tommaso Cerruti, Steven Dillmann, Ruiyang Wang, Dariush Wahdany, Allen Hart, and Karl Krauth and advisors Mike Merrill and Nicholas Carlini.

We appreciate our sponsors who made this project possible including Modal, Anthropic, OpenAI, Google, Scale AI, Snorkel Open Benchmarks, Turing, gNucleus AI, Boolean AI, and Handshake AI.

Frontier-Bench is hosted by Harbor and Laude Institute.

Written by: Ryan Marten, Alex Shaw, Andy Konwinski

Footnotes

1. Grok 4.5 was removed from this chart due to a cost reporting issue with the Cursor CLI harness. ↩