benchmarking

SigMap is a deterministic grounding layer for AI code work. It generates a signature-and-evidence map, helps pick relevant files, validates coverage, and judges whether an AI answer is grounded in the repo.

BoundaryBench is an open-source benchmark for coding agents running under hardened sandbox policies. It compares harnesses like Claude Code, Codex, Terminus 2, and Grok on Terminal-Bench tasks and includes quickstart, policy controls, and result export tools.

Open-source infrastructure for computer-use agents, with sandboxes, SDKs, benchmarks, and desktop automation tooling for macOS, Linux, Windows, and Android. The repo also includes Cua Driver, CuaBot, Cua-Bench, and Lume for VM management.

오픈소스 AI 인프라 산정 도구로, 보유 GPU나 목표 워크로드를 입력해 실행 가능한 모델, 필요 장비, 배치안, TCO 비교까지 계산합니다. LLM, RAG, VLM, 아바타, 음성 워크로드를 다루며 한국어/영어 UI와 여러 추천 경로를 제공합니다.

A community-tracked index of how users feel AI model quality is changing, updated hourly from public feedback and direct submissions.

An interactive guide for choosing local AI hardware and matching open-weights LLMs to specific builds. It offers budget-based build recommendations, a hardware picker, model-to-hardware compatibility, and a state-of-the-local-AI snapshot.

A TypeScript CLI and SDK for testing whether Agent Skills improve model outputs by running with-skill vs baseline evaluations and generating reports.

Ante is a self-contained coding agent for the terminal with local model support, multi-agent orchestration, and programmatic server modes. The docs highlight offline operation, low resource use, public benchmarks, and integrations for providers, MCP, Slack/Discord, and custom engines.

A loadable SQLite extension that adds column-oriented storage and analytics for fast local OLAP-style queries, with benchmark data and build instructions.

A research article from Applied Compute on how agentic, tool-using workloads differ from traditional LLM benchmarks, with production observations, workload profiles, and an open-source harness for replaying traces.

Anypick is an open library for selecting the best LLM across providers using filters for capabilities, pricing, and benchmarks. It includes Python and TypeScript implementations, provider support for OpenRouter and Vercel AI Gateway, and examples showing how to pick a model by constraints and strategy.

A research repo for reducing hallucinations in LLM-generated code using semantic triangulation, with setup, benchmarking, experimentation, and reproducibility instructions.

Open-source runtime for coding agents that sits underneath Claude Code to reduce tool calls, shorten context, and track savings. The page includes installation steps, a local savings check, and benchmark results comparing cost and performance.

A visual benchmark that asks AI coding agents to generate a single-file Three.js animation of a banana plant’s full life cycle, then compares the live results side by side.

A detailed post from pact0 about testing whether blank AI agents could complete a real product workflow, what failed, and the fixes they shipped after the audit.

Caplets is a developer tool that wraps MCP servers into smaller capability-based surfaces for coding agents. The site explains the workflow, setup, example capabilities like OSV, GitHub, and Sourcegraph, and includes benchmark results plus docs links.

A persistent memory layer for AI agents, built as a standard MCP server with local setup, dashboard, and benchmark claims against other memory tools.

arXiv paper describing QUEST, an open family of deep research agents from 2B to 35B parameters, plus a synthetic-task training recipe and released models, data, and scripts.

Open-source LLM uncertainty and hallucination detection tool that estimates epistemic risk in microseconds using exact-match normalized entropy, with Rust gateway and Python package support.

A research repo that replays real Claude Code and Mooncake traces to test prefix-cache eviction policies and compare them against LRU.

Experimental page-aware Metal runtime and reproducibility harness for running oversized sparse mixture-of-experts models on Apple Silicon. The repo documents GPT-OSS 120B support, performance results on 64 GiB and 16 GiB machines, and the runtime/harness structure.

A local Rust CLI for A/B testing two CLAUDE.md files against a fixed SWE-bench Lite smoke set, with doctor, run, and report commands.

A research page comparing 153 autonomous runs across 18 frontier models on the nanoGPT optimizer speedrun, with rankings, trajectories, token/compute stats, and equal-budget comparisons.

Reference implementation for using Linux Pressure Stall Information to trim an LLM KV cache under memory pressure. The repo includes requirements, basic usage commands, a simulator, a llama.cpp runner, and benchmark scripts with example results.