Automated Data Comparison for CSV and Excel
Stop manually scrolling through spreadsheets to find changes. This developer-centric skill provides a precise, programmatic way to compare two versions of a dataset. It goes beyond simple file diffing by analyzing schema drift and row-level transformations using keyed or hash-based matching.
What it does
- Schema Drift Detection: Identifies added, removed, or renamed columns and data type changes (dtypes) that break downstream pipelines.
- Row-Level Analysis: Categorizes rows as added, removed, changed, or unchanged. When a primary key is provided, it isolates exactly which cells changed with before → after samples.
- Visual Reporting: Optionally generates interactive Plotly HTML reports to help non-technical stakeholders visualize data drift.
- CI/CD Integration: Uses standard exit codes (0 for identical, 1 for differences), making it an ideal gate for automated data pipeline regression tests.
Why use this skill?
Standard text diffing tools fail on CSVs because row ordering or insignificant formatting changes trigger false positives. This skill understands data structures. It handles NaN values correctly (NaN == NaN) and provides statistical summaries of which columns are the "hottest" (most frequently changed), allowing you to pinpoint logic errors in your ETL scripts instantly.
Supported Formats & Tools
- Files: .csv, .xlsx, .xls
- Frameworks: Built on Pandas, OpenPyXL, and Plotly.
- Features: Composite keys, specific sheet targeting, and machine-readable JSON output.