Monitoring LLM Behavior: Drift, Retries, and Refusal Patterns

Traditional software produces the same output for the same input, but LLMs are probabilistic systems that respond differently each time. Therefore, deploying enterprise AI requires an AI Evaluation Stack rather than a 'vibe check' based on gut feeling.

Layer 1 of evaluation is deterministic assertions. Rules and regex instantly determine whether the JSON schema is correct, whether the right tool call was made, and whether fields like GUIDs or emails were filled in accurately, filtering out structural failures first.

Layer 2 is model-based evaluation such as LLM-as-a-Judge. This approach is used to evaluate qualities that are hard to judge with code, such as helpfulness, politeness, and actionability, and it requires a stronger reasoning model than the production model, clear rubrics, and human-verified golden output. The reasoning behind judgments must also be recorded so that the causes of failures can be debugged.

The offline pipeline is a regression test that catches failures, drift, and latency before deployment. It should be a blocking CI/CD gate at the pull request stage, run against a golden dataset of 200-500 examples.

• Include not just happy paths but also edge cases, jailbreaks, and adversarial inputs.
• Even when using synthetic data, human-in-the-loop verification is essential.
• Weighting can be applied, such as 6 points for deterministic checks and 4 points for model-based checks, but the principle of fail-fast applies: if the structure breaks, the entire test should immediately fail as 0/10.
• Large-scale services should treat 95% as the baseline, while regulated or high-risk domains should aim for 99%+.

After deployment, the online pipeline monitors for drift.

• Explicit signals: thumbs up/down and verbatim feedback.
• Behavioral signals: retry/regeneration, apology rate, refusal rate.
• Synchronous verification: apply synchronous deterministic checks to all traffic.
• Asynchronous verification: keep LLM-Judge off the critical path and sample around 5% of daily sessions.

The key is to feed failure signals caught in production back into the golden dataset, continuously updating the evaluation loop.