Scrape

**Judgement:** This skill appears to be a compliance-focused web scraping helper that prioritizes legality, ethics, and server friendliness over raw throughput. It is well-suited for users who want “safer-by-default” scraping patterns rather than maximum speed or feature breadth. **What it does well** - Enforces a **pre-scrape checklist**: fetches and checks `robots.txt`, prompts review of `/terms`, `/tos`, `/legal`, and requires stopping if scraping is explicitly prohibited. - Encourages **API-first** behavior: if an official API exists, the guidance is to use it instead of scraping, which reduces ToS and legal risk. - Implements **request discipline**: recommends 2–3 seconds between requests, realistic User-Agent with contact email, exponential backoff on HTTP 429, and session reuse to reduce server strain. - Integrates **privacy and data-protection principles**: emphasizes stripping PII, avoiding fingerprinting or indirect re-identification, minimizing storage, and maintaining an audit trail. - Provides **legal boundary framing** with references to key cases (e.g., hiQ v. LinkedIn, Van Buren v. US, Meta v. Bright Data) to clarify high-level risk zones. **Key risks and limitations** - **Not a legal guarantee:** The guidance is helpful but cannot substitute for legal advice; laws and case law evolve, and jurisdiction-specific nuances are not fully captured. - **ToS interpretation remains manual:** The tooling can prompt you to check ToS, but it cannot reliably interpret complex legal language or edge cases; users may still violate contractual terms if careless. - **Privacy handling depends on correct use:** While it promotes stripping PII, correct implementation and configuration are user responsibilities; misclassification of PII or misconfigured logging can still create compliance issues. - **Throughput trade-off:** Conservative rate limiting and strict compliance checks mean this is not ideal for high-volume or time-critical scraping workloads. - **Coverage of advanced scenarios unclear:** From the available text, it’s not clear how well it handles JavaScript-heavy sites, dynamic content, or complex anti-bot defenses. **Recommended scenarios** - Teams or individuals scraping **public, non-login content** (e.g., prices, listings, public documents) who want to minimize legal and ethical risk. - Organizations that need **defensible, auditable scraping practices** for internal use, research, or journalism, especially under GDPR/CCPA constraints. - Developers who want a **template or reference implementation** for robots.txt-respecting, rate-limited scraping workflows, with good-faith behavior evident in logs. **Less suitable for** - High-frequency or large-scale crawling where **speed and volume** are more important than conservative legal posture. - Use cases that require **bypassing technical barriers, scraping behind logins, or ignoring robots.txt/ToS**—which this skill explicitly discourages and frames as high-risk.