What it does
The PySpark Cost Linter is a specialized diagnostic tool designed to scan PySpark scripts for expensive anti-patterns that inflate cloud compute bills. It identifies specific code-level inefficiencies—such as driver-side bottlenecks, unoptimized UDFs, and plan-bloating loops—providing severity ratings and idiomatic refactoring advice for each finding.
Why use this skill
Standard linters catch syntax errors, but they don't catch the "silent killers" of Spark performance. This skill is built for data engineers who need to optimize Databricks or EMR jobs without manually auditing thousands of lines of code. It uses deterministic rules (R040–R050) to find patterns that disable Photon acceleration or cause quadratic plan analysis times, saving you from expensive trial-and-error debugging.
Supported tools
- PySpark (Core & SQL)
- Databricks (Photon & AQE optimization checks)
- CI/CD Integration (via JSON output)
- Common data formats (Delta, Parquet, CSV, JSON)
Output
The skill produces a structured report mapping rule IDs to specific line numbers. Each finding includes a description of the cost impact (e.g., "OOM Risk" or "Photon Disabled") and a code-level recommendation to fix the leak.