Evaluation and review

tom@tomrochette.com (Tom Rochette) — Thu, 24 Sep 2026 00:00:00 +0000

Where quality control lives in the agent workflow: CI gates, dashboards, machine review, human annotation, and the academic studies that benchmark the harnesses themselves.

deepeval - the pytest-style eval framework with roughly fifty judge metrics that gates merges in CI.
FrontierHarness Eval - the public benchmark that held the model constant and varied nine harnesses in twelve configurations, cost spread 17.5x, vendor-run caveat attached.
HarnessTax - the UC Berkeley study that held seven models constant across three harnesses on SWE-bench Lite and Terminal-Bench, finding harness swaps cost up to 5x while success barely moves.
Langfuse - the MIT-core tracing and eval platform, 34.7k stars, inside ClickHouse since January with the proprietary ee/ split as the caution.
Phoenix - Arize’s OTel-native observability and eval platform, self-hostable under an Elastic license.
Plannotator - the local review surface that turns your annotations on agent plans and diffs into the agent’s next instruction.
Workshop - Raindrop’s local debugger where the coding agent reads traces, writes evals, and fixes what fails.

Its members are compared on shared rows in the Evaluation and Review Feature Matrix.

Changes
#

2026-08-30 - Added deepeval.
2026-08-30 - Added Phoenix.
2026-08-30 - Added Plannotator.
2026-08-30 - Added Workshop.
2026-09-05 - Added FrontierHarness Eval.
2026-09-16 - Added Langfuse.
2026-09-18 - Added HarnessTax.

evaluation-review on tomrochette.com

Evaluation and review

Changes #

Changes
#