Agent Eval Coverage Audit is a skill that inspects an AI agent’s testing infrastructure and produces a structured remediation plan. It examines evaluation configurations, sample datasets, CI/CD hooks, and policy checks to locate gaps that could allow a flawed agent build to pass release gates undetected.
The skill works with any JSON-based evaluation configuration, including formats used by tools such as Promptfoo and LangSmith. It cross-references an agent’s defined success criteria against existing traces and configurations to surface “false greens” — tests that appear to pass but do not catch real failure modes — and missing edge cases that represent production risk.
Once the audit completes, it generates two outputs: a Markdown report formatted for sharing with stakeholders, and a machine-readable JSON file suitable for integration into CI/CD pipelines. The supported runtime environments are PowerShell and Python 3.x.
This skill is appropriate when a team is preparing an agent for production and wants an automated, repeatable check of their eval suite rather than relying on manual review. It is not a runtime monitoring tool and does not execute the agent under test itself; it analyzes the evaluation infrastructure around the agent. Teams that have no existing eval configuration or traces will have limited material for the audit to inspect.