ai-evaluation

Reference implementation for building and understanding AI evaluation systems, including rule-based evaluation, human review, LLM-as-a-Judge, online evaluation, and experiment comparison for a flight search app.

ObserverBench is an interactive benchmark and guided demo for testing whether internal AI monitoring methods help make safer decisions, not just better predictions. It includes real-model results, benchmark tasks, and a no-login walkthrough showing how warning scores affect review and harm that slips through.

ARF defines a JSON record format for AI evaluation runs with canonicalization and reproducible SHA-256 digests, so scores can be traced back to the underlying question, model, inputs, and judgment.

A write-up of a small benchmark comparing several public AI shopping assistants on niche fashion aesthetics. It shows where assistants confuse subcultures, drift toward neighboring styles, and miss brand-level signals needed for resale or outfit search.

Lagotto Meter is a beta web tool that checks how an AI agent perceives a website versus what the site claims about itself. It positions itself as a semantic truth/readability analyzer with a reproducible analysis method and public tools.

A GitHub repository that proposes a practical methodology for evaluating how AI systems behave during real work, with quick-start, practitioner, and methodology guides included.

A GitHub repo for evaluating agentic AI pipeline systems, with guidance for defining metrics, building eval cases, running repeatable tests, and tracking regressions.