Eval Study #4 — HITL Agent Regression Test, Passed but for the Wrong Reason
Following the previous post (agent tool selection · multi-step eval), this is a regression test for an HITL agent. This time, instead of a (question · expected tool) test set, I wrote unit-test style functions covering 3 axes of pitfalls (entering the breakpoint for dangerous tools / responding on every turn in multi-turn conversations / not misclassifying safe tools), with sys.exit(1) as the exit code on regression. The first run passed 3/3 — but that wasn't the right answer. The "call only after confirming clear intent" prompt that had been an issue before was still lingering, and the LLM was passing the test via its own self-defense mechanism. When I provoked it with a delete case, our HITL didn't trigger at all — only the LLM's own confirmation remained. Removing the prompt → the failure shifted to the tool simply not being called at all. Eventually, minimizing the prompt to "call the tool that matches the task the user requested" made it stop exactly at the delete_user tool (HITL working, exit code 0). Passing an eval by itself isn't a safety signal — verifying why it passed is the real safety. Pausing the LLM study series here for now, to resume after wrapping up backend studies.
