Data Anomaly Detector

**Quick judgement** Solid, construction-focused anomaly detector for tabular data built on pandas/numpy/scipy. It offers a good breadth of statistical and rule-based checks (cost, schedule, productivity, duplicates, sequences), but has some edge-case and robustness gaps that users should be aware of when integrating into production pipelines. **Strengths** - Covers several key construction data domains: costs, schedules, productivity, duplicates, and numeric sequences. - Uses established statistical methods (IQR, Z-score, modified Z-score, rolling Z-score) plus business-rule checks (negative costs, impossible dates, zero-duration non-milestones, missing sequences). - Provides a clean data model (Anomaly, AnomalyReport) and a convenient `run_full_detection` orchestration method. - Markdown report generator produces human-readable summaries with severity breakdowns and a table of anomalies. **Key risks / limitations** - **Edge-case robustness:** Some methods may fail or behave poorly when data is degenerate: - Group-based Z-scores in `detect_cost_anomalies` can divide by zero when group std is 0, yielding infinities and potentially excessive false positives. - Productivity detection (`detect_productivity_anomalies`) can divide by zero when MAD is 0, leading to NaNs/inf and unpredictable behavior. - Sequence gap detection (`detect_sequence_gaps`) assumes at least one numeric sequence value; if all values are non-numeric or missing, `min()/max()` will error. - **Config/column assumptions:** `run_full_detection` does not fully guard against missing columns (e.g., a `sequence_column` or `key_columns` specified in config but absent in the DataFrame can raise exceptions). - **In-place DataFrame mutation:** Several methods add/overwrite columns (e.g., `duration`, `productivity`, `seq_num`, converted date types) directly on the provided DataFrame, which may surprise callers that expect inputs to remain unchanged. - **Domain thresholds not fully leveraged:** The defined construction-specific threshold dictionaries are not consistently used across detection methods, limiting domain-tuning out of the box. - **Time-series detector not wired into `run_full_detection`:** `detect_time_series_anomalies` must be called manually; it’s not integrated into the main pipeline. **Recommended usage scenarios** - Offline or batch QA/QC on construction project exports (cost/schedule/productivity tables) before reporting, forecasting, or executive reviews. - As a data quality gate in ETL pipelines for construction ERPs, cost management systems, or scheduling tools, with appropriate try/except wrapping and input validation. - Exploratory anomaly scanning and auditing on historical project datasets to identify potential data issues, cost overruns, or schedule anomalies for further manual investigation. - Environments where Python + pandas/numpy/scipy are already standard, and where minor customization (e.g., additional guards, threshold tuning, or integration of time-series detection into workflows) is acceptable.