Skip to content

AI Agent Evaluation: How to Know Your Agent Got Worse

Agents degrade with no code change: silent model swaps, edited prompts, tool drift. How to build an eval harness that catches it, and what it cannot see.

The day an agent gets worse, nothing happens. No exception, no alert, no deploy. Tickets rise slightly the following week, and someone eventually says the assistant feels dumber than it used to. By then the change that caused it is three weeks back, spread across systems owned by three different teams, and nobody can name it.

That is the normal case, not the pathological one. Almost everything written about agents is about building them. Almost nothing is written about the part that decides whether they survive a year in production: knowing on an ordinary Tuesday, with evidence, that today's agent is worse than last month's. I build and operate these systems as an independent consultant, and I mark where I am giving professional judgement rather than citing a documented standard.