diff --git a/docs/cookbook/imputation-strategies-guide.md b/docs/cookbook/imputation-strategies-guide.md new file mode 100644 index 0000000..8646492 --- /dev/null +++ b/docs/cookbook/imputation-strategies-guide.md @@ -0,0 +1,268 @@ +--- +title: Choose and audit missing-value imputation +description: Compare simple and MissForest imputation, configure column rules, protect declared roles, and inspect actual repairs. +--- + +# Choose and audit missing-value imputation + +Missing values need a decision: retain the gap, fill it from the observed +column, or estimate it from other features. An imputed value is an estimate, +not a recovered observation. Start with the simplest strategy that fits your +data and inspect its effects before using the result downstream. + +## Choose a strategy + +| `impute` value | Behavior | Tradeoff | +| --- | --- | --- | +| `"mean"` | Fill numeric columns with their observed mean. Non-numeric columns are skipped. | Sensitive to extreme values; repeated fills reduce variation. | +| `"median"` | Fill numeric columns with their observed median. Non-numeric columns are skipped. | More resistant to extreme values, but still a constant estimate. | +| `"mode"` | Fill from the most frequent observed value, including text columns. | Can reinforce the majority category; inspect ties and rare categories. | +| `"auto"` | Use median for numeric columns and mode for other columns. | A simple per-column rule, not model selection. | +| `"missforest"` | Use iterative random-forest regression/classification with other columns as predictors. | Ineligible data can take an audited simple fallback; model-eligible data requires the ML dependencies. | + +Simple imputation uses only the observed values of the column being filled. +FreshData's MissForest-inspired implementation initializes gaps, fits a model +for each eligible column, and iterates up to `missforest_max_iter`, stopping +earlier if it converges. More complexity does not guarantee better estimates. +Compare choices on representative data using validation suited to your task. + +`strategy` controls the broader cleaning engine; `impute` and `impute_strategy` +are explicit missing-value overrides: + +| Combination | What to expect | +| --- | --- | +| `strategy="conservative"`, `impute=None` | Representation repairs; no decision-engine imputation. | +| `strategy="balanced"`, `impute=None` | Accuracy-first decisions based on column roles, missingness, and available evidence. Gaps may remain. | +| `strategy="aggressive"`, `impute=None` | More active filling and column-drop decisions, with default outlier flagging. Capping requires `outlier_action="cap"`. | +| Any strategy with an explicit `impute` choice | Apply that override to eligible columns; declared ID/target protection still applies. | +| Any strategy with `impute_strategy={...}` | Rules for the named columns override the global `impute` choice. | + +The examples use `"conservative"` to make explicit imputation easy to inspect. +Other strategies can make additional decisions; none guarantees that every +gap will be filled. See the [cleaning engine](../cleaning-engine.md). + +## Apply global and per-column rules + +Fill age with its median and group with its mode. Deliberately request fills +for declared ID and target columns to verify that those requests are ignored +and audited. One age and one group are missing at `row-c`. + +```python +import pandas as pd +import freshdata as fd + +frame = pd.DataFrame( + { + "record_key": ["A", None, "C", "D", "E"], + "age": [20.0, 30.0, None, 40.0, 90.0], + "group": ["red", "blue", None, "red", "red"], + "outcome": [1.0, 0.0, 1.0, None, 0.0], + }, + index=["row-a", "row-b", "row-c", "row-d", "row-e"], +) + +cleaned, report = fd.clean( + frame, + strategy="conservative", + impute="median", # global numeric rule + impute_strategy={ + "group": "mode", # overrides the global rule + "record_key": "mode", # ignored: declared identifier + "outcome": "mean", # ignored: declared target + }, + id_columns=("record_key",), + target_column="outcome", + fix_dtypes=False, + drop_empty_rows=False, + drop_empty_columns=False, + return_report=True, + verbose=False, +) + +assert cleaned.loc["row-c", "age"] == 35.0 +assert cleaned.loc["row-c", "group"] == "red" +pd.testing.assert_series_equal(cleaned["record_key"], frame["record_key"]) +pd.testing.assert_series_equal(cleaned["outcome"], frame["outcome"]) +print(cleaned[["age", "group"]]) +print(report.warnings) +``` + +The warnings identify ignored rules for `record_key` and `outcome`. Declare +important roles explicitly: explicit simple imputation protects `id_columns` +and `target_column`, so do not rely on name heuristics to protect arbitrary +keys or labels. Context-protected columns also remain untouched. All-missing +columns have no observed fill value, and numeric rules do not apply to text; +those gaps can remain. + +Use the post-normalization names for `impute_strategy`, `id_columns`, and +`target_column` when the default `column_names=True` renames input headers. +For `"Customer Age"`, use `"customer_age"`. This is essential for MissForest: +its role gate matches the normalized column name exactly; declaring a custom +target under its original spaced name can leave it unprotected. Alternatively, +use `column_names=False` and declarations matching the unchanged headers. +Configure these options through `fd.clean` keywords or `fd.CleanConfig`. + +## Inspect the audit and the changed cells + +Continue with `frame`, `cleaned`, and `report` from the previous example: + +```python +for action in report.actions: + if action.step == "impute": + print(action.column, action.count, action.description, action.metadata) + +# Actions report a column and affected count, not every cell's location. +# This example keeps the index, columns, and row order, so the frames align. +pd.testing.assert_index_equal(cleaned.index, frame.index) +assert all(column in cleaned.columns for column in frame.columns) +filled = frame.isna() & cleaned[frame.columns].notna() +repairs = [ + { + "row_position": position, + "row_label": frame.index[position], + "column": column, + "value": cleaned[column].iloc[position], + } + for column in frame.columns + for position in range(len(frame)) + if filled[column].iloc[position] +] +print(repairs) +assert [(item["row_label"], item["column"]) for item in repairs] == [ + ("row-c", "age"), + ("row-c", "group"), +] + +# Persist the structured aggregate audit when you need a portable record. +audit_json = report.to_json() +``` + +The filled cells are age `35.0` and group `"red"` at `row-c`; ID and target gaps +remain. Imputation actions report one affected cell per fill and zero-count +notes for protected columns. `CleanReport` also captures warnings and overall +before/after missing counts. `columns_imputed` is a decision-engine summary; +use the actual `impute` actions for explicit overrides. + +This comparison identifies missing-to-present repairs. If your plan drops or +reorders rows or renames columns, align the frames using a stable record +identity first. It is not a history of every non-missing value change. See +[audit automated cleaning](audit-automated-data-cleaning.md) and the +[audit schema](../audit-trail.md) for supported report fields. + +## Try model-based imputation + +Install the optional ML dependencies: + +```bash +pip install "freshdata-cleaner[ml]" +``` + +This separate example has 80 rows and related numeric/categorical predictors. +It fills eight missing charges with a small forest and at most two iterations. +No identifier or target column is included in this model example. + +```python +import pandas as pd +import freshdata as fd + +training = pd.DataFrame( + { + "hours": [float(i % 12) for i in range(80)], + "charges": [10 + 2 * (i % 12) + 0.5 * (i % 5) for i in range(80)], + "kind": ["standard", "priority"] * 40, + } +) +training.loc[training.index[::11], "charges"] = float("nan") + +estimated, model_report = fd.clean( + training, + strategy="conservative", + impute_strategy={"charges": "missforest"}, + missforest_n_estimators=10, + missforest_max_iter=2, + missforest_random_state=42, + missforest_add_indicators=False, + fix_dtypes=False, + return_report=True, + verbose=False, +) + +assert estimated["charges"].notna().all() +for action in model_report.actions: + if action.step == "impute" and action.column == "charges": + print(action.count, action.metadata) + assert action.count == 8 +``` + +Inspect `selected_model_type`, `iterations`, `converged`, and `fallback_reason` +in action metadata. With ML dependencies installed, this example reports +`selected_model_type="regressor"` and `fallback_reason=None`. Reaching the +iteration limit is recorded separately from convergence. Numerical estimates +and model diagnostics can vary with dependency versions. + +`missforest_min_rows_for_model` defaults to 50. Smaller inputs take a recorded +simple-imputation fallback. Use the earlier five-row `frame`: + +```python +small, small_report = fd.clean( + frame, + strategy="conservative", + impute_strategy={"age": "missforest"}, + id_columns=("record_key",), + target_column="outcome", + missforest_add_indicators=False, + fix_dtypes=False, + drop_empty_rows=False, + drop_empty_columns=False, + return_report=True, + verbose=False, +) + +assert small.loc["row-c", "age"] == 35.0 +fallback_actions = [ + action for action in small_report.actions + if action.step == "impute" and action.column == "age" +] +assert len(fallback_actions) == 1 +print(fallback_actions[0].metadata["fallback_reason"]) +assert "fewer rows" in fallback_actions[0].metadata["fallback_reason"] +``` + +Insufficient predictors and ineligible columns can also lead to fallback or +preservation; inspect the audit rather than assuming a requested model ran. +If a column is model-eligible but scikit-learn is absent, cleaning raises +`ImportError` with the ML installation hint. No result or report is returned +for that failed call; install `freshdata-cleaner[ml]` before retrying. + +The following example protects a custom target whose input header is renamed. +The declaration and imputation rule both use its post-normalization name: + +```python +raw = pd.DataFrame( + { + "Observed Hours": [float(i % 12) for i in range(80)], + "Future Sales": [10 + 2 * (i % 12) + 0.5 * (i % 5) for i in range(80)], + } +) +raw.loc[raw.index[::11], "Future Sales"] = float("nan") + +protected, protected_report = fd.clean( + raw, + strategy="conservative", + impute_strategy={"future_sales": "missforest"}, + target_column="future_sales", # matches after header normalization + fix_dtypes=False, + return_report=True, + verbose=False, +) + +pd.testing.assert_series_equal( + protected["future_sales"], raw["Future Sales"].rename("future_sales") +) +assert protected["future_sales"].isna().sum() == 8 +``` + +With matching declarations, ID and target gaps are preserved. Role protection +is not a train/test leakage guarantee: cleaning learns from the supplied frame. +Isolate training data and validate your downstream workflow before estimating +values in evaluation data. diff --git a/docs/cookbook/index.md b/docs/cookbook/index.md index ac7a2fc..46677a1 100644 --- a/docs/cookbook/index.md +++ b/docs/cookbook/index.md @@ -24,6 +24,10 @@ Every recipe is self-contained with runnable code, realistic sample outputs, and Role-aware imputation: why identifiers and targets must be preserved while numerics are safely imputed. *(Search intent: safe missing value imputation, fillna without corruption)* +- :material-help-box: **[Choose and audit missing-value imputation](imputation-strategies-guide.md)** + Compare simple and model-based fills, declare protected roles, and inspect actual repairs. + *(Search intent: mean median mode imputation, missforest, per-column imputation)* + - :material-content-copy: **[How to detect and resolve duplicate rows](detect-and-resolve-duplicates.md)** Report duplicate clusters without silent row drops, and apply controlled deduplication policies. *(Search intent: detect duplicate rows pandas, safe deduplication python)* diff --git a/docs/fallback-matrix.md b/docs/fallback-matrix.md index 27dfd9a..d507cb0 100644 --- a/docs/fallback-matrix.md +++ b/docs/fallback-matrix.md @@ -84,7 +84,6 @@ from pprint import pprint import polars as pl import freshdata as fd -from freshdata.execution import PlanGenerator frame = pl.DataFrame({"value": [1, 2]}) backend = "polars" diff --git a/mkdocs.yml b/mkdocs.yml index 6193425..84d47e3 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -146,6 +146,7 @@ nav: - Overview: cookbook/index.md - Clean pandas DataFrame: cookbook/clean-pandas-dataframe.md - Safely handle missing values: cookbook/safely-handle-missing-values.md + - Imputation strategies: cookbook/imputation-strategies-guide.md - Detect & resolve duplicates: cookbook/detect-and-resolve-duplicates.md - Prepare data for ML: cookbook/prepare-data-for-machine-learning.md - Clean Polars DataFrames: cookbook/clean-polars-dataframes.md