AgentBench
The facts below are what the maker publishes about AgentBench, checked and dated. Our editors have not written about it yet.
The facts
Each one read from AgentBench's own pages, with the date it was checked.
What it does
- Configuration
- The configuration system uses YAML with extended import, default, and overwrite keywords.checked 9 Oct 2026
- Current tasks
- The current function-calling version supports alfworld, dbbench, knowledgegraph, os_interaction, and webshop tasks.checked 9 Oct 2026
- Current version
- The repository says its current version is AgentBench FC, which uses function-calling prompts and integrates with AgentRL.checked 9 Oct 2026
- Deployment
- The project documents a one-command setup for AgentBench FC tasks using Docker Compose.checked 9 Oct 2026
- Environments
- The original benchmark includes eight environments, including operating systems, databases, knowledge graphs, house-holding, web shopping, and web browsing.checked 9 Oct 2026
- Extensibility
- Users can add a task environment by implementing the Task class and specifying it in the configuration.checked 9 Oct 2026
- FC task coverage
- AgentBench FC supports fully containerized deployment for alfworld, dbbench, knowledgegraph, os_interaction, and webshop.checked 9 Oct 2026
- Hardware limit
- The README warns that the webshop environment requires approximately 16 GB of RAM to start.checked 9 Oct 2026
- Knowledge graph dependency
- The KnowledgeGraph task depends on an online service that the README says is unstable, and it documents local deployment as an option.checked 9 Oct 2026
- Known issue
- The README warns that the current alfworld implementation leaks memory and disk space until its task worker is restarted.checked 9 Oct 2026
- License
- The repository is licensed under Apache License 2.0.checked 9 Oct 2026
- Maintainer
- The THUDM GitHub profile identifies the group as THUKEG and lists FIT Building, Tsinghua University as its location.checked 9 Oct 2026
- Model interfaces
- The framework supports local models served with FastChat and says models offered only through an API can be connected by implementing an interface in the Agent Client.checked 9 Oct 2026
- Model setup
- The original quick start configures an OpenAI API key and gives gpt-3.5-turbo-0613 as its example agent.checked 9 Oct 2026
- Original environments
- The original benchmark covers eight environments: operating system, database, knowledge graph, digital card game, lateral thinking puzzles, household tasks, web shopping, and web browsing.checked 9 Oct 2026
- Other agents
- The README says users can replace the default agent by changing configuration parameters.checked 9 Oct 2026
- Purpose
- AgentBench is a benchmark for evaluating the reasoning and decision-making abilities of LLMs acting as agents.checked 9 Oct 2026
- Requirements
- The README recommends Python 3.9 for dependency installation and requires Docker for the original quick-start setup.checked 9 Oct 2026
- Research release
- The repository describes AgentBench as an ICLR 2024 work.checked 9 Oct 2026
- Resource limits
- The current webshop environment requires about 16 GB of RAM to start, and the README warns that alfworld can leak memory and disk space until its task worker is restarted.checked 9 Oct 2026
- Support
- The project invites users to join its Slack for questions and collaboration, and lists a Google Group for questions about AgentBench FC.checked 9 Oct 2026
The company
- Founded
- 2023checked 28 Sep 2026
Alternatives to AgentBench
- 1
DeepEval
Free planOpen sourceWebAPILinuxmacOS8.3From $200 a month - 2
Promptfoo
Free planOpen sourceWebAPILinuxmacOS7.8Free plan - 3
Braintrust
Free planWebAPISelf-hosted7.7From $249 a month - 4
Giskard
Free planOpen sourceWebAPILinuxSelf-hosted7.6Free plan - 5
TruLens
Free planOpen sourceAPISelf-hosted7.2Free plan - 6
Arena (formerly Chatbot Arena)
Free planOpen sourceWebAPISelf-hosted7.1Free plan