Flaky Test Detective For Ai Coding Agents

Diagnoses unreliable tests, identifies root causes, creates stabilization plans, and generates safe AI coding prompts for fixing flaky unit, integration, E2E, and CI tests.

Install
cmdop skills install agensi-flaky-test-detective-for-ai-coding-agents

Flaky Test Detective is a skill for AI coding agents, developers, QA engineers, CI/CD teams, and test automation teams who need to investigate tests that pass and fail inconsistently. It covers a broad range of test types: unit tests, integration tests, E2E tests, browser tests, API tests, and database tests, across frameworks including Playwright, Cypress, Selenium, Jest, Vitest, and pytest. It also handles CI-only failures, which are often the hardest class of flakiness to reproduce locally.

For any failing or unreliable test, the skill produces root cause hypotheses and failure pattern reports. It performs async timing audits to identify race conditions, selector stability reviews for browser tests, and mock leakage checks where shared state bleeds between tests. It reviews database cleanup procedures, compares CI versus local environment configurations, and produces test isolation plans. From those findings it generates stabilization roadmaps and QA tickets that describe what to fix and how to verify the fix.

The skill is explicitly designed to avoid masking failures. It does not recommend retries, arbitrary sleep calls, skipped tests, or weakened assertions as solutions. Instead it produces paste-ready prompts formatted for AI coding tools including Cursor, Claude Code, Codex CLI, OpenCode, Replit, and ChatGPT Agents, so the identified root cause can be handed directly to a coding agent for remediation.

Use cases

  • Identify the root cause of a Playwright test that fails only on CI but passes locally
  • Audit Jest or Vitest test suites for mock leakage between test files
  • Generate a stabilization roadmap for a pytest suite with intermittent async failures
  • Produce a QA ticket and verification plan for a flaky Selenium selector
  • Create paste-ready Cursor or Claude Code prompts to fix a race condition in an integration test
  • Review database cleanup logic in a test suite that produces order-dependent failures

When to use it

  • A test suite has tests that fail non-deterministically across runs
  • Tests pass locally but fail on CI and the cause is unknown
  • The team is masking failures with retries or arbitrary sleeps and wants to fix root causes
  • A browser test fails intermittently due to selector or timing issues
  • An AI coding agent needs structured, scoped prompts to apply test fixes safely

When not to use it

  • The project has no existing tests and needs tests written from scratch
  • The goal is performance benchmarking or load testing rather than reliability diagnosis
  • The test framework in use is not covered by the skill's supported list
  • A fully automated code-editing tool that applies fixes directly to the codebase is needed, as the skill produces prompts and plans rather than executing code changes