Agent Reliability Audit

Turn raw agent traces and tool logs into professional production-readiness audits and remediation reports.

Install
cmdop skills install agensi-agent-reliability-audit

Agent Reliability Audit is a skill published by agensi that takes exported run logs, execution traces, tool calls, and retry records from AI agents and produces structured audit reports aimed at identifying failure modes before a production rollout.

The skill performs pattern detection across real-world transcripts to surface agent looping, behavioral drift, and latency hotspots that are not visible in unit test results. It also performs tool stability analysis by correlating a tool inventory against actual execution traces, flagging integrations that exhibit flaky or inconsistent behavior across multiple runs.

Audit output is generated in both Markdown and JSON formats. The Markdown report is structured for stakeholder review, while the JSON output is suitable for downstream processing or storage. Both formats include deep dives into recovery failures — cases where an agent failed to recover from an error state — and connect each observed failure pattern to a specific architectural improvement.

The skill is designed for Python-based workflows and is described as compatible with log output from LangChain, CrewAI, and custom OpenAI implementations. It is intended to fit into CI/CD pipelines as well as developer workstation workflows.

This skill is appropriate when preparing an AI agent for production and needing a systematic, multi-run reliability assessment that goes beyond ad-hoc log inspection. It is not a real-time monitoring tool and does not replace runtime observability infrastructure.

Use cases

  • Audit LangChain or CrewAI agent traces before a production launch
  • Identify looping and drift patterns across multiple agent run logs
  • Find flaky tool integrations by correlating tool inventory against execution traces
  • Generate a stakeholder-ready Markdown reliability report from exported agent logs
  • Produce a JSON audit artifact for ingestion into a CI/CD quality gate
  • Get specific architectural remediation guidance tied to observed failure patterns

When to use it

  • Preparing an AI agent for production and needing evidence of reliability across multiple runs
  • Conducting a pre-launch review where stakeholders require a formal, structured audit document
  • Investigating recurring agent failures that standard error logs do not explain
  • Integrating reliability checks into a CI/CD pipeline for agent deployments

When not to use it

  • Real-time or streaming agent monitoring — this skill operates on exported logs, not live data
  • Non-Python workflows where the Python-based integration cannot be used
  • Agents whose logs are not exportable in a format compatible with the skill's input expectations
  • Projects that need a runtime observability platform rather than a point-in-time audit