https://amitnaik.com/tags/agent-evaluation/ Recent content in Agent Evaluation on Amit Naik Hugo -- 0.144.2 en-us Mon, 28 Sep 2026 08:59:56 -0700 https://amitnaik.com/til/2026-09-28-the-harness-can-change-the-cost-of-success/ Mon, 28 Sep 2026 08:59:56 -0700 https://amitnaik.com/til/2026-09-28-the-harness-can-change-the-cost-of-success/ Melissa Pan compares seven models across Claude Code, Codex, and Pi harnesses. Her reported results suggest similar task success can hide meaningful cost differences. Evaluating a model in isolation misses the effects of the surrounding coding workflow. https://amitnaik.com/til/2026-09-28-an-agent-evaluation-needs-more-than-a-prompt/ Mon, 28 Sep 2026 08:59:41 -0700 https://amitnaik.com/til/2026-09-28-an-agent-evaluation-needs-more-than-a-prompt/ Alex Lieberman's notes on agent evaluation emphasize the combination of the agent, its system, tasks, and a verifier. Rubrics and reproducible environments matter because a plausible answer is not the same thing as a successfully completed task. https://amitnaik.com/til/2026-09-28-evaluate-documentation-with-reader-agents/ Mon, 28 Sep 2026 08:59:24 -0700 https://amitnaik.com/til/2026-09-28-evaluate-documentation-with-reader-agents/ LangChain's OpenWiki article introduces WikiBench, evaluating generated repository knowledge through questions answered by reader agents. The interesting shift is to test whether documentation helps a downstream user solve a problem, rather than judging the document only by its appearance.