Eval Study #4 — HITL Agent Regression Test, Passed but for the Wrong Reason

Following Eval Study #3 — Agent Tool Selection · Multi-Step, today’s topic is HITL agent regression testing, targeting the HITL agent I built last time.

Table of contents

Why unit-test style this time

In the previous post, I graded using a (question · expected tool · expected argument) test set. This time is different — I wrote unit-test-style functions where each test function directly states what behavior must occur to pass. I didn’t set up a separate test set.

The reason: with HITL, the behavioral scenario itself is what’s being verified. It’s not “given this question, this tool should be chosen” but rather “when trying to call a dangerous tool, execution should stop at the breakpoint.” Since this is scenario-based verification, unit tests are a natural fit.

The 3 axes of regression tests

The pitfall regression tests I built this time:

1. Dangerous tool → confirm breakpoint entry — instead of a human pressing y/n, directly inspect the breakpoint state
2. Multi-turn → confirm the LLM responds on every turn — when the conversation continues, does the LLM respond every time
3. Safe tool → confirm it isn’t misclassified as dangerous — does the safe tool just execute right away

Why sys.exit(1) is needed

When a regression test fails, it returns an exit code via sys.exit(1). At first, I didn’t understand why this was necessary.

If you’re just looking with your own eyes, it’s “proceed if it passes, stop if it fails” — so why bother with a code?

The answer is automation — systems like CI/CD pipelines run this script and judge success/failure by the exit code. Exit 0 = pass, exit ≠ 0 = fail. Pipelines aren’t human, so they can’t read console logs.

First run — 3/3 passed, but that’s not the right answer

3/3 passed. All pitfall defenses maintained.

But actually, the fact that I expected failure and ran it anyway turned into a problem, because it passed.

The reason — when I built the HITL agent last time, the system prompt that had been an issue — “call only after confirming clear intent” — I thought I’d removed it, but it was still there. Because of this prompt, the LLM was confirming every time as its own self-defense mechanism, so it wasn’t calling the dangerous tool even without our HITL logic being involved.

In other words:

The test passed, but the reason it passed wasn’t our HITL logic — it was a side effect of the LLM prompt.

If I let this slide as “eval passed,” the defense will break the moment the prompt changes later.

Round 2 — provoking with a delete case

To check whether our HITL actually catches this once the prompt-based defense is removed, I switched to a case that explicitly requests “delete” and ran it again.

Test 1 FAIL. next=('execute_tools',) — meaning the next step was set to tool execution, and the actual stop didn’t happen. Our HITL safeguard wasn’t working.

Presumed cause: the problematic prompt was still pushing the LLM into self-defense mode, so the LLM was only doing its own confirmation, bypassing our logic. The pitfall had recurred.

Round 3 — removing the problematic prompt → expanding to multi-step

I removed the phrase “call only after confirming clear intent” from the prompt. But it still failed:

This failure pattern is different — the LLM doesn’t call the tool at all and only guides with text. It calls get_user_info, but doesn’t call the dangerous tool (delete_user), and instead just outputs something like “Are you sure you want to delete this user?” as its final message.

In other words, another defensive phrase remained in the prompt (only modify/delete when the user ID is precisely known), which was still causing the LLM to avoid calling the tool.

Round 4 — minimizing the prompt → an actual pass

I simplified the prompt completely:

• Before: only modify/delete when the user ID is precisely known
• After: call the tool that matches the task the user requested

This time it truly passed — execution stopped exactly at the dangerous tool delete_user (the HITL logic worked). I confirmed exit code 0. Now our safeguard actually catches the danger.

The ongoing value of eval tools

Once you build an eval that works as intended, you can immediately re-verify with this regression set the next time you swap models or change vector DBs. This is the core benefit of automated regression testing — the interest on reusability compounds over time.

Retrospective

Three intuitions I picked up from HITL eval:

1. Passing an eval ≠ a safety signal. Verifying “why it passed” is the real safety. If a test passes while a prompt’s side effect is standing in for the actual defense, the defense breaks the moment the prompt changes. A pass without a known cause isn’t really a pass.
2. It’s better to start regression tests from failure scenarios. Running a test while expecting failure makes you question whether a pass is a genuine pass. If you run it expecting success, you won’t dig into why it passed.
3. The more conditions you add to a prompt, the more tool calls get avoided. The more you write about “when not to call the tool,” the more the LLM leans toward not calling it. For tool-calling logic, a combination of a minimal prompt + a separate safeguard (something like HITL) is more stable.

Wrapping up the LLM study series for now

I originally intended to keep going with fine-tuning, LoRA, production deployment, observability, and so on, but I think the production deployment side will make more sense once my backend foundation is stronger. I’m pausing LLM studies here for now, and plan to come back after wrapping up backend studies (Java → Spring Boot).

The trajectory of my LLM studies so far is organized in the LLM Study Series.

Things to study further

1. Deepening HITL regression test patterns

• Verifying state snapshots using langgraph.checkpoint
• Building an auto-response mock instead of having a real human press y/n
• Regression scenarios that pass through multiple breakpoints in sequence

2. Layering safeguards

• Separating the roles of LLM self-defense (prompts) vs. system-level safeguards (HITL/breakpoints)
• Criteria for deciding what to delegate to which side
• Designing so the two layers don’t neutralize each other when both are present

3. Integrating regression tests into CI

• Running python hitl_regression_test.py on every commit via GitHub Actions
• Blocking PR merges on failure
• Reference: LangSmith Evaluations CI example

4. Fine-tuning vs. prompt engineering

• When to solve things with prompts and when to move to fine-tuning
• LoRA · QLoRA — lightweight fine-tuning methods
• Whether prompt engineering is still needed even after fine-tuning

5. LLM observability

• Comparing LangSmith · Langfuse · Traceloop (trajectory tracing · cost · latency · eval integration)
• What metrics actually get put on dashboards in production
• I’ll pick this up again after finishing backend studies