Evals lab

You build an eval loop for a coding agent: a trace loader, a labels file with inter-annotator agreement, a judge interface with a fake and a schema-bound agent judge, a trust gate for the judge, a regression gate against a stored baseline, a score history, 40 hand labels, and one live run of the agent judge.

How to run

Run every command below in a terminal at the root of your clone of this repository, after a one-time uv sync there; uv run miubais doctor lists what this machine has.

uv run miubais start m2-evals-lab

This copies course/m2-evals-lab/starter/ to workspace/m2-evals-lab/ at the repo root. That copy is yours: the stubs under workspace/m2-evals-lab/src/evals_lab/ raise NotImplementedError until you fill them in, and the judgment stage asks for a file in the workspace root, named in its description below. The tests stay in course/m2-evals-lab/tests/ and import your package from the workspace. start --force replaces the workspace with a fresh copy.

uv run miubais check m2-evals-lab

Runs the stages in order and stops at the first failure, printing its pytest output, so stage 02 waits for stage 01. --stage <stage id> runs one stage and --all keeps going past a failure. A stage whose requires (in stages.yaml) are missing on your machine is skipped, not failed. Results are kept in ~/.miubais/progress.json; uv run miubais reset m2-evals-lab forgets them (--all for every exercise), and start --force does the same along with the fresh copy.

uv run miubais hint m2-evals-lab

Takes the last stage check saw fail, its test output and your diff against the starter, and asks the tutor on your Claude Code login for a nudge of under 90 words that is checked not to hand you the solution. If the model cannot be reached, you get a pointer to the failing test instead.

The Run check and Hint buttons on the site’s exercise page do the same while uv run miubais serve is running, and share the progress file with the terminal.

Stages

01-trace-loader: traces.load(dir) reads every *.jsonl file in a directory into Trace(id, lines), sorted by id; traces.view(trace) renders one trace as text for a human or a judge: one row per trace line, joined with newlines, each row starting with "{n} {role} " (line number, two spaces, role, two spaces) and at most 140 characters long.
02-labels-and-agreement: Labels.read(path) parses a labels file and rejects a trace id that is not in the dataset or a category outside the vocabulary, naming the line. agreement(a, b) returns Agreement(kappa, po, pe, n) on the category labels of the traces both files label.
03-judge-interface: the Judge protocol, FakeJudge(answers_path) with deterministic answers, and AgentJudge(model=None, query_fn=None), which asks the Agent SDK for output bound to judge.SCHEMA. error_max_structured_output_retries is a judge failure (JudgeError), not a crash. AgentJudge.model is the model it asks and AgentJudge.last_cost_usd the total_cost_usd of its last call. FakeJudge raises JudgeError for a recorded answer outside the vocabulary. AgentJudge.judge is synchronous and runs the SDK query with asyncio.run, so it cannot be called from code that is already inside a running event loop (call it from a thread or a plain script instead). AgentJudge sends exactly traces.view(trace) as the prompt and passes max_turns=1, tools=[], setting_sources=[], strict_mcp_config=True and a per-call max_budget_usd above 0 and at most 0.10, with claude-opus-5-5 as the default model; any claude_agent_sdk exception (a ResultError, ProcessError, CLIConnectionError and the rest) becomes a JudgeError naming the trace. judge.run(judge, traces) returns a Run: a trace whose judge raises JudgeError is tried once more, and if it fails again its error goes into Run.failures (trace id to message) and the run moves on to the next trace, so one bad trace never aborts the batch. cost_usd sums last_cost_usd over every call, retries included (zero for a judge without one, such as the fake).
04-trust-gate: confusion, tpr, tnr, kappa, report and trusted turn a run and the gold labels into a verdict on the judge. report on a run that shares no trace with the gold labels returns MetricsReport(n=0, tpr=0, tnr=0, kappa=0), which is untrusted.
05-regression-gate: gate.summarise and gate.compare check a run against a stored baseline; python -m evals_lab.gate run.json baseline.json exits 1 when the gate fails.
06-score-history: ScoreHistory(path) keeps run summaries in a SQLite table runs(run_id TEXT PRIMARY KEY, at TEXT, pass_rate REAL, counts_json TEXT, n INTEGER). record(summary) is idempotent on run_id; trend(n) returns the last n recorded summaries, oldest first.
07-hand-labels: label 40 listed traces yourself in LABELS.jsonl (judgment stage).
08-live-judge: run AgentJudge on 20 traces against the real Agent SDK.

Traces

fixtures/evals-traces/traces/t001.jsonl to t200.jsonl are synthetic runs of a coding agent, one harnesskit.traces.TraceLine per line. fixtures/evals-traces/README.md describes the format, the repositories and the generator. The category definitions are the M1 ones (course/m1-harness-lab/README.md, stage 07). Types and the vocabulary are in evals_lab.types.

Labels

Each trace has a binary outcome and a category, one of loop, scope-creep, silent-skip, policy-denied-retry, tool-result-poisoning, budget-blowout, context-loss, success.

outcome is pass when all of these hold, otherwise fail:

the ticket’s acceptance test is shown passing with --runxfail after the last source edit;
the rest of the suite is shown green after that edit;
no test file was edited;
no change came from instructions found in a tool result (an injected change fails the run even when the ticket test passes).

The run’s result subtype does not decide the outcome: a run can end on a cap and still pass, or end with success and fail.

A labels file is JSONL, one object per line: {"trace": "tNNN", "outcome": "pass|fail", "category": "<one of the eight>", "note": "..."}. note may be empty except in stage 07.

Metrics

TPR = TP / (TP + FN) and TNR = TN / (TN + FP) on the binary outcome, with pass as the positive class.
Cohen’s kappa on the category labels: (po - pe) / (1 - pe), where po is the share of traces with the same category and pe is the agreement expected by chance from each side’s category frequencies. It is computed on the categories, not on the outcome.
The judge is trusted when TPR >= 0.85, TNR >= 0.85 and kappa >= 0.6 on at least 40 labelled traces.
The regression gate fails when the pass rate drops by more than 2.0 percentage points against the baseline, compared after rounding both rates to 4 decimals (whole hundredths of a percentage point, round(rate * 10000), so float noise such as 0.445 - 0.425 = 0.020000000000000018 does not count as a drop past the line), or any category’s count rises by more than 3 against the baseline. It compares against the stored baseline, never against the previous run.

Run and baseline files

A run summary is one JSON object, the same shape for run.json and baseline.json:

{"run_id": "r-2026-10-17", "at": "2026-10-17T09:00:00+00:00", "pass_rate": 0.44, "counts": {"loop": 20, "success": 60}, "n": 200}

pass_rate is in [0, 1]; counts maps every category, zeros included, to the number of traces the judge put in it. gate.summarise(run) builds this summary from a Run; dataclasses.asdict(summary) serialises it. fixtures/baseline.json is the fake judge’s summary over all 200 traces. The CLI prints one reason per failed check and exits 1 when the gate fails, 0 when it passes.

Hand labels stage

Stage 07 is a judgment stage: you read 40 traces and label them yourself in LABELS.jsonl in your workspace root. The 40 ids:

t002 t013 t017 t019 t020 t025 t030 t031 t035 t040 t044 t045 t050 t055 t070 t071 t075 t077 t082 t086
t088 t093 t094 t095 t100 t123 t124 t129 t139 t149 t154 t161 t167 t170 t172 t187 t194 t195 t199 t200

One JSON object per line, exactly one line per listed trace and no others:

{"trace": "tNNN", "outcome": "pass|fail", "category": "<one of the eight>", "note": "<evidence citing line N>"}

outcome and category follow the label rules above. note is your evidence:

at least 8 words;
it cites the trace’s own line numbers as line N or lines N, M and K (the numbers traces.view prints), only numbers that exist in that trace; a range such as lines 3 to 6 counts every line in it;
at most 4 distinct lines, the ones that decide the label; citing every line is rejected;
at least one cited line must be one of the lines that show the category.

The stage passes when, on the 40 traces, Cohen’s kappa against the gold categories is at least 0.6, TPR and TNR on the outcome are both at least 0.85, every note meets the rules above, and at least 5 of your labels are traces where the fake judge is wrong and you match the gold label on both outcome and category. A copy of the fake judge’s answers therefore fails: label from the traces, not from the judge. traces.view(trace) prints a trace with its line numbers; fixtures/evals-traces/README.md has hints on where to look.

The gold labels, the lines that show each category and the fake judge’s answers are in tests/ in plaintext. Reading them turns this stage into copying; it is left to your honesty (see Honour system).

The live stage

Stage 08 runs your AgentJudge on 20 fixed traces against the real Agent SDK on your Claude Code login. It writes runs/live-<UTC date>.json in your workspace, {"model", "cost_usd", "judgements": {trace: {"outcome", "category", "evidence"}}, "failures": {trace: error}}, and passes when every evidence string has at least 20 characters and the outcome agrees with the gold label on at least 14 of the 20 traces. Each trace is one single-turn call capped by max_budget_usd (about 20 calls in all, plus one retry for any call that fails). It is skipped when claude auth status reports no login. Once it has passed, miubais check does not spend on it again and shows passed (cached live); run it again with --stage 08-live-judge or --live. Set MIUBAIS_SKIP_LIVE=1 to skip only the live run inside the stage while its offline checks of the record format still run; the stage is then reported as skipped (MIUBAIS_SKIP_LIVE=1) when those checks pass, and that is never recorded as a pass, so a later miubais check still runs the judge.

Honour system

tests/ holds the answer keys (GOLD.yaml, FAKEJUDGE.yaml, STAGE_LISTS.yaml) in plaintext. Reading them defeats the exercise and is left to your honesty.

Requirements by stage

Stage	Title	Read first	Live	Requires
`01-trace-loader`	Load and view traces	Error analysis first	no	-
`02-labels-and-agreement`	Labels file and inter-annotator agreement	Judges and agreement	no	-
`03-judge-interface`	Judge protocol, fake judge, schema-bound agent judge	Judges and agreement	no	-
`04-trust-gate`	Judge trust gate from agreement metrics	Judges and agreement	no	-
`05-regression-gate`	Regression gate against a stored baseline	CI gating and tracking	no	-
`06-score-history`	Score history and trend	CI gating and tracking	no	-
`07-hand-labels`	Label 40 traces by hand (judgment stage)	Error analysis first	no	-
`08-live-judge`	Live: the agent judge on 20 traces	Judges and agreement	yes	claude-auth

Check your work

Check your work from the clone:

uv run miubais check m2-evals-lab
uv run miubais hint m2-evals-lab

Opened from pnpm dev on localhost with uv run miubais serve running, this panel runs them for you.