Chaos Engineering

Design rigorous chaos engineering experiments and resilience audits to verify production system reliability.

Install
cmdop skills install agensi-chaos-engineering

Chaos Engineering is a skill that gives an AI agent the structured methodology of a senior chaos engineer. Rather than producing generic best-practice checklists, it guides the agent through a four-phase procedure — Hypothesize, Perturb, Minimize, Learn — that treats infrastructure as a laboratory subject to controlled, evidence-based experiments.

For experiment design, the skill produces specific chaos experiments with measurable hypotheses, single-variable perturbations, and defined blast radii. This means instead of a vague recommendation like “add retries,” the agent will help design a concrete stress test to determine whether a system collapses under a retry storm.

For resilience auditing, the skill identifies hidden architectural failure modes including thundering herds, gray failures, and synchronized backoffs — patterns that standard reviews often miss.

For operational safety, it defines the human roles required to run experiments safely in production: Lead, Observer, and Abort Authority, along with specific readiness flags that must be satisfied before an experiment begins.

For post-incident work, it analyzes past incidents to produce “never again” experiments that verify a fix actually holds under realistic conditions.

The skill focuses on tail-risk scenarios at the P99 and P99.9 latency percentiles rather than averages, targeting the worst-case conditions that cause real outages. It is a reasoning and design skill; it does not execute experiments or interact with infrastructure directly.

Use cases

  • Design a chaos experiment with a testable hypothesis to verify a circuit breaker holds under upstream failure
  • Audit a microservices architecture for thundering herd and synchronized backoff vulnerabilities
  • Define blast radius and abort criteria before running a production fault injection
  • Assign operational roles (Lead, Observer, Abort Authority) and readiness flags for a safe production experiment
  • Convert a past incident post-mortem into a repeatable chaos experiment that verifies the fix
  • Structure a retry-storm stress test to confirm a service does not cascade under load

When to use it

  • Planning chaos experiments that need falsifiable hypotheses rather than generic recommendations
  • Auditing a system architecture for hidden resilience amplifiers before a production release
  • Preparing operational runbooks and role assignments for safe experiment execution
  • Translating a post-mortem into a structured verification experiment
  • Focusing resilience work on P99/P99.9 tail-risk scenarios rather than average-case behavior

When not to use it

  • Directly executing fault injections or interacting with live infrastructure — this skill produces designs, not automation
  • Environments where no tools or humans are available to carry out the designed experiments
  • Teams looking for a simple checklist rather than a rigorous experimental methodology
  • Scenarios requiring real-time observability data ingestion or automated rollback triggering