reproducibility

Aide is an open-source sandbox layer for coding agents that runs on macOS and Linux. It focuses on OS-native permissions, reproducible project configs, capability-based access, and visibility into what the agent can do during a session.

Open-source multi-agent harness from Microsoft for running large self-organized agent teams asynchronously. The repo describes scaling to 1,024 agents, includes a reproduction guide, Docker Compose setup, launch scripts, and an evaluation write-up.

Open-source harness for measuring how consistently hosted LLM APIs return the same output across repeated runs and over time, with raw transcripts, scoring, and a static leaderboard generator.

Open Science is an open-source AI workbench for scientists. It combines literature, code, figures, reports, and review into a local-first desktop workflow with model choice, reproducible artifacts, and scientific agent skills.

Open-source research skill that helps Claude, Codex, and OpenClaw run a more disciplined AI research workflow, from hypothesis and literature review through baselines, leakage checks, analysis, and paper writing.

ARF defines a JSON record format for AI evaluation runs with canonicalization and reproducible SHA-256 digests, so scores can be traced back to the underlying question, model, inputs, and judgment.

CTGT’s research post shows how deterministic LLM serving can still produce inconsistent pipeline outcomes, and introduces Concordance, an Apache 2.0 instrument for tracing where AI workflows diverge. The page includes the problem setup, benchmark details, findings, limitations, and a linked repo for reproduction.

A reproducible GitHub research repo comparing Codex reasoning-token usage between API and ChatGPT Pro access across 960 completed responses, with code, data, and verification steps included.