Arun ArumugamAgent Engineering

Themesrising

Teams build evals and judges for agents

Engineering teams are publishing how they test systems whose output is open-ended. Netflix runs LLM judges in production and aligns them with human ratings over time, Vercel lays out an agent test pyramid with seeded task environments and per-constraint verifiers, and Ramp replaced tests with deliberating LLM judges for its internal coding benchmark, lifting judge agreement from 54% to 94%. Reliable evaluation is becoming its own engineering discipline, and a prerequisite for shipping agents with confidence.

First seen
Latest
From
Blogs
Feed

  1. Ramp EngineeringBlogEvalsRamp: making LLM judges agree when grading coding agentsRamp replaced tests with LLM judges for its internal SWE benchmark. Splitting 'list the differences' from 'do they matter', then letting judges deliberate, lifted agreement from 54% to 94%, and engineers backed about 90% of verdicts.

  1. VercelBlogAgentsVercel: how to build end-to-end evals for agents using your API or CLIA worked method for testing software whose users are agents: a test pyramid by LLM-call count, seeded task environments, per-constraint verifiers, and choosing pass@k versus pass^k by what a failure costs.

  1. Netflix Tech BlogBlogResearchThe lifecycle of LLM-as-a-judge: building, aligning and monitoring at scaleNetflix covers running LLM judges in production, from building them to aligning them with human judgment and monitoring them over time. Useful for anyone evaluating agents.