benchmark

A pure C runtime for Gemma 4 E2B CPU inference, with export, benchmarking, and numerical validation tools included.

An open public experiment that tests whether AI agents can complete everyday web errands, publishes the full transcripts, and scores 113 sites for agent-readiness.

ReactBench is a benchmark for evaluating coding agents on realistic React work, with published scores, cost comparisons, and example tasks focused on production-grade frontend issues.

A research article benchmarking bot detection systems against scripted browsers, stealth browsers, and modern AI browsing agents, with comparative results across five detection products.

An open robot football league where frontier AI models and open-source club code manage simulated Unitree G1 humanoids in daily matches, with live streams, match logs, and a public engine for teams to join.

Open-source causal-graph memory for AI agents with MCP support, scoring, active forgetting, and benchmark results published in the repo.

An open benchmark and leaderboard for AI CAD agents, with 308 prompts across 20 categories and layered scoring for geometry, engineering, manufacturability, and cognition.

Smelt is an open-source terminal coding agent with a Rust core and Lua plugins, aimed at developers who want a fast, customizable command-line workflow. The page shows install steps, plugin support, vim mode, themes, and benchmark claims comparing memory and speed against other agents.

A benchmark for coding agents that tests whether they can formally verify properties of real-world Rust web apps in Lean 4. It includes upstream code, i5h ports, extracted Lean models, proof attempts, and task-by-task results.

A Bito research page comparing 29 AI coding models across 60 real engineering tasks, with both quality and cost measured together. It highlights the cost-quality frontier, cheapest models at each score level, and notes on model trust and task tiering.

A benchmark page that aggregates public model lists from AI gateways into one matrix, showing which gateways offer each model and their per-million-token pricing.

Open-source repository for a NanoGPT training speedrun, focused on getting a 124M model to target loss on 8x H100 GPUs in under 90 seconds. The README includes run commands, Docker instructions, and a detailed list of model and systems optimizations.

A visual benchmark that asks AI coding agents to generate a single-file Three.js animation of a banana plant’s full life cycle, then compares the live results side by side.

A research project exploring multi-agent LLM coordination in Pokémon Showdown. It compares talk modes, reports win rates and token costs, and frames the environment as a benchmark for studying collaboration between agents.

A GitHub repo benchmarking TypeSafe’s Jev against Sonnet, GPT-5 nano, and local models on 770 Reddit AITA verdicts, with Brier scores, latency, and cost comparisons.

ObserverBench is an interactive benchmark and guided demo for testing whether internal AI monitoring methods help make safer decisions, not just better predictions. It includes real-model results, benchmark tasks, and a no-login walkthrough showing how warning scores affect review and harm that slips through.

Open-source harness for measuring how consistently hosted LLM APIs return the same output across repeated runs and over time, with raw transcripts, scoring, and a static leaderboard generator.

Open-source benchmark that measures which developer tools LLMs recommend when asked to build real web apps. It runs fixed prompts against a panel of models, parses the tool recommendations, and publishes rankings and head-to-head comparisons.

Public benchmark for comparing agent memory systems, with separate textual and coding tracks, submission flow, documentation, and API guidance.

An open-source benchmark for testing whether LLMs can follow long arithmetic derivations without using tools. It compares models on recall-proof inputs, logs digit accuracy, and reports results across multiple endpoints and model families.

DeepSWE is a benchmark for measuring frontier coding agents on original, long-horizon software engineering tasks. The page shows a leaderboard, methodology overview, task examples, and a full blog explaining the benchmark design and results.

A benchmark for testing whether coding agents can show staff-level engineering judgment, not just produce correct code.

A GitHub repository that publishes a week of controlled runs comparing seven Google models on the same software task, with transcripts, telemetry, screenshots, and scored benchmark artifacts.

A black-box benchmark report on how AI-generated tests detect functional bugs in live APIs across 20 scenarios and 7 systems.