The EdgeBench platform stands out by measuring how agents learn from real-world environments over extended periods, rather than just assessing pre-existing knowledge. With tasks spanning over twelve hours, it captures the compounding nature of experience. The benchmark includes a wide array of tasks, from scientific problems using real research data to software engineering challenges involving large codebases. The involvement of domain experts, who spend an average of fifty-seven hours reviewing tasks, ensures a high level of quality and relevance. For example, the scientific tasks require agents to formulate hypotheses and validate against noisy observations, while the software tasks demand reasoning about cross-module coupling. The feedback mechanisms, including build logs and proof states, are designed to approximate real professional workflows. This structured approach allows for iterative improvement and detailed evaluation. Given the extensive effort involved, it raises the question of how the platform scales for broader community use. The release of fifty-one tasks with a full evaluation framework provides a solid foundation for researchers to analyze agent learning curves effectively.
Questions about EdgeBench
What is EdgeBench?
EdgeBench is an ultra-long-horizon benchmark developed by ByteDance Seed that measures how agents learn from real-world environments rather than what they already know. It consists of 134 executable tasks spanning science, software engineering, optimization, knowledge work, formal math, and games, each designed to run for at least 12 hours of continuous operation. The benchmark includes a leaderboard, an evaluation platform, a paper, a GitHub repository, and a dataset for researchers to study scaling laws of environment learning.
How are the tasks structured and what makes them unique?
The benchmark consists of 134 tasks divided into six diverse families, with most being built from scratch and reviewed by domain experts. Each task requires at least 12 hours of continuous operation, with some selected extended runs continuing beyond 72 hours to allow experience to compound. These tasks are designed to approximate real practice, ensuring that a high score reflects what an agent actually learns over time.
Who is the intended audience for EdgeBench?
EdgeBench is primarily designed for researchers who want to study the scaling laws of environment learning and understand how agents improve through experience. It serves scientists, software engineers, and optimization specialists who need a rigorous way to measure learning from real-world environments rather than simple recall. The benchmark provides a standardized platform for comparing different agents' ability to learn under realistic conditions.
What are the specific categories of tasks included in EdgeBench?
The benchmark covers six distinct task families: scientific problems and machine learning, systems and software engineering, combinatorial optimization, and professional knowledge work. Each family contains a specific number of tasks, such as 39 for scientific problems and 36 for systems and software engineering, totaling 134 tasks. These categories ensure a wide range of real-world scenarios are covered in the evaluation.
How much effort is required to create and review the tasks?
The benchmark emphasizes tasks built from scratch and reviewed by domain experts, with recorded human effort averaging 57.2 hours per task. Among the largest cases, expert work reaches up to 320 hours, ensuring that the tasks are rigorous and require significant domain expertise to complete. This focus on expert-reviewed, scratch-built tasks distinguishes EdgeBench from other benchmarks.
What are some examples of the complex tasks in EdgeBench?
The benchmark includes tasks where agents must formulate hypotheses, choose models, and validate against noisy observations, such as gravitational-wave detection and groundwater plume modeling. In software engineering tasks, agents work on production-grade codebases requiring thousands of lines of change, with over 100,000 lines in the largest cases. These examples illustrate the complexity and real-world relevance of the tasks.
What types of optimization problems are included in EdgeBench?
The benchmark includes tasks where agents must design, tune, and iterate on heuristic search strategies for open-ended, predominantly NP-hard problems. Examples include vehicle routing, SAT/SMT solving, and molecular self-assembly, where even strong solvers have room to improve with additional time and feedback. These tasks require advanced problem-solving skills and iterative refinement.
What professional knowledge work tasks are included in EdgeBench?
The benchmark includes tasks that reproduce real white-collar deliverables across finance, education, and other professional knowledge work areas. These tasks require agents to handle real-world constraints and feedback, such as build logs, test failures, and profiler output in software engineering. The diversity of these tasks ensures a comprehensive evaluation of agent capabilities.
Who develops and maintains EdgeBench?
The benchmark is developed by ByteDance Seed and is associated with academic institutions and research teams. It is used by researchers and developers interested in scaling laws of environment learning and agent performance. The benchmark is accessible through its website and GitHub repository, making it available to a global audience of researchers and practitioners.
What is the official website of EdgeBench?
The official website of EdgeBench is edge-bench.org.
How is EdgeBench rated on Trustburn?
On Trustburn, EdgeBench has a rating of 4 out of 5 based on 5 reviews.
AI-generated from reviews on this page. Verify anything important with the company directly.
EdgeBench offers a sophisticated approach to measuring agent learning by focusing on real-world environment interactions rather than simple knowledge retention. The platform includes tasks that run for twelve hours or more, which is crucial for observing how experience compounds over time. I appreciate the diversity of the task families, ranging from scientific problems like gravitational-wave detection to complex software engineering challenges involving production-grade codebases. The involvement of domain experts ensures that the tasks are realistic and challenging, with expert effort averaging over fifty hours for the most complex cases. The feedback mechanisms, such as build logs and test failures, closely mimic actual professional workflows. This makes the benchmark highly relevant for evaluating agents that need to iterate and improve based on structured critique. The structured review process adds a layer of reliability that is often missing in other benchmarks.
The long-horizon tasks really shine here, allowing agents to learn from feedback over twelve hours. It is a refreshing approach compared to standard benchmarks.
The benchmark appears thorough with tasks lasting over twelve hours, yet the exact resource requirements for running these extended simulations are not explicitly detailed. Since expert effort averages fifty-seven hours per task, it is wise to verify the computational costs and time commitments before assuming the platform fits your specific infrastructure constraints.
EdgeBench presents a compelling framework for evaluating how agents learn from real-world environments rather than just testing static knowledge. The inclusion of ultra-long-horizon tasks, running for at least twelve hours, allows experience to compound in a way that shorter benchmarks cannot achieve. With sixty-five distinct task families spanning science, software engineering, and optimization, the platform offers a diverse landscape for assessment. The fact that most tasks were built from scratch and reviewed by domain experts adds significant credibility. For instance, the scientific tasks utilize real research data, requiring agents to handle noisy observations and refine hypotheses iteratively. Similarly, the software engineering tasks involve production-grade codebases with thousands of lines of changes. This structure seems ideal for measuring genuine learning curves. The availability of an evaluation framework alongside the initial fifty-one tasks makes it accessible for researchers. It is a robust tool for understanding agent development over extended periods.
News
No news yet. This is where the business shares its own updates and press.
Photos
Widgets
Get the widget for your site →About EdgeBench
EdgeBench is a benchmark developed by ByteDance Seed that measures how agents learn from real-world environments. It consists of 134 executable tasks spanning science, software engineering, optimization, knowledge work, formal math, and games, with each task designed to run for at least 12 hours of continuous operation. The benchmark emphasizes tasks built from scratch and reviewed by domain experts, with recorded human effort averaging 57.2 hours per task. EdgeBench includes a leaderboard, an evaluation platform, a paper, a GitHub repository, and a dataset for researchers to study scaling laws of environment learning.
- Website
- edge-bench.org