The Nex Scraper Politeness Pattern provides a production-ready, asynchronous Python fetch layer designed for ethical and sustainable web scraping. It moves beyond simple HTTP requests by implementing a sophisticated coordination layer that prevents your agent from being flagged as a “bad bot.” The skill generates a complete package including a PoliteScraper engine and a specialized email_extractor.

Standard scraping scripts often fail because they lack the “politeness” heuristics that real-world websites expect. Instead of just hammering a URL, this skill implements a global concurrency cap paired with a per-domain throttle. This means you can scrape 100 different websites simultaneously, but your agent will never hit any single site more than once every few seconds. It ensures your bot identifies itself honestly, respects robots.txt, and handles transient network errors with exponential backoff.

You receive a modular Python package (polite_scraper.py, email_extract.py, and requirements.txt). The code is cleanly structured as an async context manager, providing a fetch_many() method that returns structured results without crashing on individual failures. Built by Nex AI. More skills and info at nex-ai.be and slopsome.com.