benchmarking

A GitHub release for Microsoft’s Agent Lightning Skill, which helps coding agents optimize other AI agents using benchmarks and iterative improvements across prompts, tools, workflows, models, and reasoning settings.

A Python CLI and research toolkit for simulating AI training math before model training. It estimates memory, training time, token budget, config risks, benchmark speed, and reliability, with optional native Rust and C backends.

An Apple Silicon–optimized inference build of Bonsai 1.7B with custom Metal kernels, benchmark results, quick-start instructions, and a bundled OpenAI-compatible server.

A detailed technical write-up about training a 3.8B-parameter language model from scratch, including setup, throughput improvements, ablations, benchmarks, and cost/performance results.

A blog post about verifiable RAG that benchmarks open-source NLI verifiers against Claude on RAGTruth and describes a Python library for sentence-level citation and claim verification.

A blog post from Augment Code comparing its coding agent, Auggie, against Claude Code on Opus 4.7. It presents benchmark results, token usage, cost comparisons, and an explanation of the Context Engine and Prism router.

A GitHub research project documenting a long-form, multi-model analysis of LLM behavior across Claude, Gemini, ChatGPT, and Grok. The repo includes an executive summary, screenplay, technical white paper, and archive of logs and chat records.

GitHub’s research preview on multi-model orchestration for coding tasks. The page describes adaptive workflow selection and includes benchmark comparisons against the Opus 5 baseline with estimated cost reduction claims.

A technical blog post from the Cua repository describing a process-scoped Metal capability shim for macOS VMs that improves llama.cpp performance on Apple Silicon, with benchmark results and reproduction details.

A research article describing an empirical runtime-monitoring approach for multi-turn LLM agents, backed by 3,175 runs across four benchmarks. It also points to an open-source implementation, state-harness, with Rust/Python support, LangGraph and CrewAI adapters, CLI tooling, and OpenTelemetry export.

A blog post describing an evaluation harness for comparing agentic coding tools and prompts across realistic SWE tasks, with token-cost results and model-specific behavior notes.

LEVI is a harness-first evolutionary framework for code and prompt optimization. It focuses on reducing LLM cost with diversity-preserving search, role-aware model routing, and a proxy benchmark, and presents comparative results against several existing systems.

A Superconductor blog post showing how background coding agents were used to reproduce, diagnose, and fix a Rails memory leak using derailed_benchmarks, with a reusable Agent Skill workflow included.

A Zenodo preprint reporting a 12,160-trial black-box evaluation of GPT-5.4 outputs under two prompt conditions, with detailed token-ceiling sweeps, control/ablation trials, and hash-chained verification.