5.1 Understanding and Handling Missing Data

Chapter 4 treated parsing and schema validation as a boundary contract. One result of that contract is honest missingness: values that were absent, invalid, or not applicable remain visible instead of being disguised as zero or empty text. Cleaning begins by asking why information is missing, not by immediately making every cell non-empty.

Missing is a state, not a value to compare

pandas encounters several missing markers because columns can use different dtypes:

• pd.NA appears in nullable pandas dtypes such as Int64, boolean, and the nullable string dtype.

• NaN is the traditional floating-point “not a number” marker.

• NaT represents missing datetime or timedelta values.

• Python None may appear in object-like input and is usually recognized as missing.

Use isna and notna across these representations:

missing = deliveries.isna()
known_minutes = deliveries["minutes"].notna()

Do not test value == pd.NA. With nullable dtypes, comparison to unknown generally remains unknown:

pd.NA == pd.NA  # pd.NA
pd.NA > 40      # pd.NA

pd.NA follows three-valued logic: true, false, and unknown. For example, True | pd.NA is true because either possible value of the unknown leaves the result true. False | pd.NA remains unknown. This prevents an unavailable fact from being silently treated as false.

Reductions such as mean and sum commonly skip missing values by default:

mean_minutes = deliveries["minutes"].mean()
strict_mean = deliveries["minutes"].mean(skipna=False)

The first mean may use fewer rows than the table contains. Always report or inspect the valid denominator:

valid_n = deliveries["minutes"].count()
total_n = len(deliveries)

Profile columns, rows, groups, and patterns

A column missingness table is a useful first view:

mask = deliveries.isna()

column_profile = pd.DataFrame({
    "missing_count": mask.sum(),
    "missing_rate": mask.mean(),
}).sort_values("missing_rate", ascending=False)

But a single percentage hides structure. Add row burden:

row_missing = mask.sum(axis=1)
review = deliveries.loc[row_missing >= 2]

Then inspect whether fields disappear together:

patterns = (
    mask.astype("int8")
        .value_counts(dropna=False)
        .rename("rows")
        .reset_index()
)

If minutes and address vanish together during one afternoon, a collection outage is more plausible than two independent random events. Compare missing rates across meaningful groups and time:

by_zone = deliveries.groupby("zone")["minutes"].apply(
    lambda values: values.isna().mean()
)

Do not collapse structurally not applicable into unknown. A pickup order may legitimately have no delivery address. Store a reason code or process type so analysts can distinguish “not required” from “should exist but was not captured.”

Missingness mechanisms are assumptions about the process

Statistical discussions often distinguish three mechanisms:

• Missing completely at random (MCAR): missingness is unrelated to observed or unobserved values relevant to the analysis.

• Missing at random (MAR): after conditioning on observed information, missingness no longer depends on the missing value itself.

• Missing not at random (MNAR): missingness still depends on the unobserved value or another unobserved cause.

These names do not mean that MAR is “random enough” or that a pattern chart can prove a mechanism. They are assumptions about how data was generated. For example, long deliveries may be less likely to receive a completed-time scan. If the likelihood of missingness depends on the unobserved duration even after using recorded variables, the process is MNAR.

Use system knowledge, collection logs, source owners, and sensitivity analysis. A missingness test or dashboard can challenge an assumption, but rarely establishes it by itself.

Choose a response that matches the analytical question

Leaving values missing is a valid choice when a method handles them or when filling would create unjustified facts. Other common responses include deletion and imputation.

Complete-case analysis drops rows missing fields required for a specific calculation:

complete = deliveries.dropna(subset=["minutes", "zone"])

Use subset rather than dropping every row with any missing field. A missing optional note should not remove an otherwise valid delivery. Record how many rows and which groups were removed; complete cases can represent a different population.

Constant filling is appropriate only when the constant has defined meaning:

notes = deliveries["driver_note"].fillna("not_recorded")

Filling missing minutes with zero is normally wrong because zero minutes is a real duration. It changes means, threshold classifications, correlations, and later models.

Statistic-based imputation can preserve row count:

was_missing = deliveries["minutes"].isna()
minutes_numeric = deliveries["minutes"].astype("Float64")
median_minutes = minutes_numeric.median()

analysis = deliveries.assign(
    minutes_was_missing=was_missing,
    minutes_clean=minutes_numeric.fillna(median_minutes),
)

The indicator preserves evidence that the value was imputed. Median filling reduces variation and can distort relationships, so compare results with an unfilled or complete-case scenario.

Groupwise filling may use relevant observed information:

zone_median = minutes_numeric.groupby(deliveries["zone"]).transform("median")
analysis["minutes_zone_fill"] = minutes_numeric.fillna(zone_median)

Small groups can have unstable or entirely missing medians. Define a fallback hierarchy and minimum group size rather than accepting missing fills unexpectedly.

Forward fill and interpolation rely on ordering assumptions. They may be reasonable for regularly sampled sensor measurements, but usually not for unrelated customer rows:

sensor = sensor.sort_index()
sensor["temperature_linear"] = sensor["temperature"].interpolate(
    method="time",
    limit=2,
)

The limit prevents a long outage from being painted over. Never interpolate identifiers or unordered categories.

Prevent leakage and preserve lineage

In a prediction workflow, estimate medians, category frequencies, or model-based imputations using training data only, then apply the learned rule to validation and test data. Computing a fill value from the full dataset leaks information across the evaluation boundary. Chapter 8 will package this rule in a machine-learning pipeline.

For every repair, preserve:

• The original field or immutable raw dataset.

• A Boolean repair indicator and, when useful, a reason code.

• The method, parameters, grouping fields, and software version.

• Counts before and after the operation.

• Sensitivity results under plausible alternative policies.

The next section applies the same evidence-first approach to repeated records. Exact duplicates can be detected mechanically, but deciding which record represents an entity requires explicit identity rules.