CSV Data Pipeline

**Judgement** Robust, low-dependency CSV/JSON data pipeline skill that is well-suited for command-line and Python-centric workflows. It covers most common ETL and reporting needs on tabular files and leverages proven standard tools, making it a strong choice for practical data wrangling on a typical Unix-like environment. **Key strengths** - Handles a wide range of operations: inspection, filtering, sorting, grouping/aggregation, joins, deduplication, validation, cleaning, and reporting. - Uses only Python 3 and standard Unix tools (`head`, `tail`, `awk`, `sort`, `tr`, etc.), simplifying deployment on Linux/macOS systems. - Good support for multiple formats (CSV, TSV, JSON, JSON Lines) and conversions between them. - Provides streaming/row-by-row patterns for large files, mitigating memory issues for big CSVs. - Includes data-quality helpers (cleaning and type/email/date validation) plus Markdown summary reporting. **Risks and limitations** - Many examples assume a Unix-like shell and coreutils; on pure Windows environments without WSL or similar, compatibility is reduced. - Several Python patterns load entire files into memory as lists of dicts, which can be problematic for very large datasets if the streaming patterns are not used. - Schema and type handling are basic (simple regex-based date/email checks and explicit type casts); complex schemas or locale-specific formats may need custom extensions. - Operations are mostly imperative scripts/snippets rather than a single high-level declarative interface, so less technical users may find it harder to compose complex workflows. - File operations are standard CLI redirections without safeguards (e.g., overwriting output files), so users must be cautious when running commands on important data. **Recommended scenarios** - You have CSV/TSV/JSON/JSONL files on a Unix-like system and need to quickly filter, join, aggregate, or deduplicate them in a reproducible, scriptable way. - You want to build lightweight ETL pipelines and summary reports (especially in Markdown) without introducing heavy frameworks or external databases. - You must clean and validate exported data (e.g., from SaaS tools or logs) before downstream analysis or loading into a database. - You’re comfortable with shell commands and Python and want a set of practical patterns to assemble into custom data workflows.