5.4 Identifying and Handling Outliers

After normalizing dates and units, some observations will still be far from the rest. An outlier is an observation that is unusual under a chosen reference distribution or rule. It is not synonymous with error. A five-hour delivery could be a duplicated timestamp, a road closure, a unit problem, or a genuine extreme event—the distinction matters more than the flag.

Separate impossible values from statistically unusual values

Begin with domain constraints:

python

minutes = deliveries["minutes"]

domain_invalid = (minutes.lt(0) | minutes.gt(24 * 60)).fillna(False)

A negative elapsed time may indicate reversed timestamps or clock problems. A duration over one day may violate this table's process contract. These are not statistical judgments; they follow from how the variable is defined. Confirm the contract before correcting or excluding them.

Next summarize the distribution with more than a mean:

python

summary = minutes.describe(
    percentiles=[0.01, 0.25, 0.5, 0.75, 0.95, 0.99]
)

Compare mean and median, inspect quantiles, and view records at both tails. A large mean–median gap can indicate skew or extreme values, but it does not locate their cause.

Use detection rules as investigation lenses

The interquartile range (IQR) is

IQR = Q_3 - Q_1.

A common screening rule flags values below

Q_1 - 1.5\,IQR

or above

Q_3 + 1.5\,IQR.

In pandas:

python

valid = minutes.mask(domain_invalid)
q1, q3 = valid.quantile([0.25, 0.75])
iqr = q3 - q1

iqr_flag = (
    valid.lt(q1 - 1.5 * iqr)
    | valid.gt(q3 + 1.5 * iqr)
)

The factor 1.5 is a convention, not a law. In a strongly skewed distribution it may flag many legitimate tail values. In a small sample, quartiles can be unstable.

A classical z-score uses mean $\bar{x}$ and standard deviation $s$ :

z_i = \frac{x_i - \bar{x}}{s}.

But the observations being flagged can pull both $\bar{x}$ and $s$ , weakening the rule. A robust alternative uses the median and median absolute deviation (MAD):

MAD = \operatorname{median}(|x_i - \operatorname{median}(x)|).

A commonly scaled modified score is

z_i^* = 0.6745\frac{x_i - \operatorname{median}(x)}{MAD}.

When MAD is zero, division is undefined. That can happen when at least half the observations share one value. Use domain rules, inspect unique values, or choose another justified scale estimate rather than forcing a score.

Loading interactive lab...

Loading concept check...

Loading practice...

Context changes what counts as unusual

A 90-minute delivery may be extreme in a compact city zone and ordinary in a rural zone. Compute group-specific references only when groups have enough observations and genuine process differences:

python

zone_stats = deliveries.groupby("zone")["minutes"].agg(
    n="count",
    median="median",
    q1=lambda values: values.quantile(0.25),
    q3=lambda values: values.quantile(0.75),
)

Very small groups produce unstable bounds and can hide cross-group anomalies. Compare both global and group-conditioned views.

Univariate rules also miss unusual combinations. A delivery may have an ordinary distance and ordinary duration, yet an implausibly high speed when the two are combined. Derived checks can encode relationships:

python

hours = deliveries["minutes"].div(60)
speed_kmh = deliveries["distance_km"].div(hours)
relationship_invalid = speed_kmh.gt(160) | speed_kmh.lt(0)

Again, a flag starts an investigation. A very high calculated speed may expose a unit mismatch or timestamp problem rather than a vehicle traveling at that speed.

Investigate provenance before choosing treatment

For each flagged record, review:

The immutable source row and source-system event history.

Units, parsing outcomes, time zone, and transformation lineage.

Duplicate or entity-resolution decisions affecting the record.

Operational context such as weather, promotions, outages, or route closures.

Whether the event is impossible, erroneous, rare but valid, or still unresolved.

Store separate flags and a review status. A useful vocabulary is confirmed_error, verified_extreme, and unresolved. Do not turn one statistical Boolean into a deletion command.

Treatments answer different questions

Keep: Verified extremes belong to the observed population. Report robust summaries such as the median and IQR alongside mean-based statistics.

Correct: If the source proves a recoverable transcription or unit error, create a corrected analytical field while preserving the raw value, correction reason, and rule version.

Exclude: Remove a confirmed invalid record from a specific analysis, not from historical evidence. State how the analysis population changes.

Cap or winsorize: Replace tail values in an analytical scenario with declared bounds:

python

lower, upper = verified.quantile([0.01, 0.99])
winsorized = verified.clip(lower=lower, upper=upper)

This limits influence but changes totals and tail behavior. It is not a repair unless the cap has domain justification.

Transform: A logarithm can make a positive right-skewed variable easier to model:

python

import numpy as np

log_minutes = np.log1p(deliveries["minutes"])

The transformation changes scale, not truth. It does not make an invalid negative value valid, and results must be interpreted on the transformed or back-transformed scale.

Show sensitivity instead of hiding judgment

Build parallel scenarios without overwriting the raw column:

python

raw_source = deliveries["minutes"].copy()
raw_analysis = raw_source.astype("Float64")
confirmed_error = deliveries["outlier_status"].eq("confirmed_error")
verified = raw_analysis.mask(confirmed_error)

lower, upper = verified.quantile([0.01, 0.99])

scenarios = {
    "keep_all": raw_analysis,
    "remove_confirmed_errors": verified,
    "winsorize_verified_1_99": verified.clip(lower, upper),
}

For each scenario, report sample size, mean, median, upper quantiles, and totals. Then repeat the primary business comparison or model. If a conclusion reverses under a plausible policy, that sensitivity is part of the result.

Loading interactive lab...

Loading concept check...

Loading practice...

Chapter 5 cleaning contract

A defensible cleaned dataset should retain:

Immutable raw fields or a versioned raw source.

Missingness, parsing, duplicate, normalization, and outlier flags.

Corrected or standardized analytical fields beside raw representations.

Mapping tables, thresholds, conversion factors, survivor rules, and their versions.

Before/after counts and validation results.

Review decisions, unresolved cases, and sensitivity reports.

Cleaning is not the act of making data visually uniform. It is the controlled production of a more usable representation while preserving enough evidence to explain every material change. Chapter 6 will use that cleaned representation for transformations, ranking, encoding, and reshaping.

5.4 Identifying and Handling Outliers

Separate impossible values from statistically unusual values

Begin with domain constraints:

python

minutes = deliveries["minutes"]

domain_invalid = (minutes.lt(0) | minutes.gt(24 * 60)).fillna(False)

Next summarize the distribution with more than a mean:

python

summary = minutes.describe(
    percentiles=[0.01, 0.25, 0.5, 0.75, 0.95, 0.99]
)

Compare mean and median, inspect quantiles, and view records at both tails. A large mean–median gap can indicate skew or extreme values, but it does not locate their cause.

Use detection rules as investigation lenses

The interquartile range (IQR) is

IQR = Q_3 - Q_1.

A common screening rule flags values below

Q_1 - 1.5\,IQR

or above

Q_3 + 1.5\,IQR.

In pandas:

python

valid = minutes.mask(domain_invalid)
q1, q3 = valid.quantile([0.25, 0.75])
iqr = q3 - q1

iqr_flag = (
    valid.lt(q1 - 1.5 * iqr)
    | valid.gt(q3 + 1.5 * iqr)
)

The factor 1.5 is a convention, not a law. In a strongly skewed distribution it may flag many legitimate tail values. In a small sample, quartiles can be unstable.

A classical z-score uses mean $\bar{x}$ and standard deviation $s$ :

z_i = \frac{x_i - \bar{x}}{s}.

But the observations being flagged can pull both $\bar{x}$ and $s$ , weakening the rule. A robust alternative uses the median and median absolute deviation (MAD):

MAD = \operatorname{median}(|x_i - \operatorname{median}(x)|).

A commonly scaled modified score is

z_i^* = 0.6745\frac{x_i - \operatorname{median}(x)}{MAD}.

Loading interactive lab...

Loading concept check...

Loading practice...

Context changes what counts as unusual

A 90-minute delivery may be extreme in a compact city zone and ordinary in a rural zone. Compute group-specific references only when groups have enough observations and genuine process differences:

python

zone_stats = deliveries.groupby("zone")["minutes"].agg(
    n="count",
    median="median",
    q1=lambda values: values.quantile(0.25),
    q3=lambda values: values.quantile(0.75),
)

Very small groups produce unstable bounds and can hide cross-group anomalies. Compare both global and group-conditioned views.

python

hours = deliveries["minutes"].div(60)
speed_kmh = deliveries["distance_km"].div(hours)
relationship_invalid = speed_kmh.gt(160) | speed_kmh.lt(0)

Again, a flag starts an investigation. A very high calculated speed may expose a unit mismatch or timestamp problem rather than a vehicle traveling at that speed.

Investigate provenance before choosing treatment

For each flagged record, review:

The immutable source row and source-system event history.

Units, parsing outcomes, time zone, and transformation lineage.

Duplicate or entity-resolution decisions affecting the record.

Operational context such as weather, promotions, outages, or route closures.

Whether the event is impossible, erroneous, rare but valid, or still unresolved.

Store separate flags and a review status. A useful vocabulary is confirmed_error, verified_extreme, and unresolved. Do not turn one statistical Boolean into a deletion command.

Treatments answer different questions

Keep: Verified extremes belong to the observed population. Report robust summaries such as the median and IQR alongside mean-based statistics.

Correct: If the source proves a recoverable transcription or unit error, create a corrected analytical field while preserving the raw value, correction reason, and rule version.

Exclude: Remove a confirmed invalid record from a specific analysis, not from historical evidence. State how the analysis population changes.

Cap or winsorize: Replace tail values in an analytical scenario with declared bounds:

python

lower, upper = verified.quantile([0.01, 0.99])
winsorized = verified.clip(lower=lower, upper=upper)

This limits influence but changes totals and tail behavior. It is not a repair unless the cap has domain justification.

Transform: A logarithm can make a positive right-skewed variable easier to model:

python

import numpy as np

log_minutes = np.log1p(deliveries["minutes"])

The transformation changes scale, not truth. It does not make an invalid negative value valid, and results must be interpreted on the transformed or back-transformed scale.

Show sensitivity instead of hiding judgment

Build parallel scenarios without overwriting the raw column:

python

raw_source = deliveries["minutes"].copy()
raw_analysis = raw_source.astype("Float64")
confirmed_error = deliveries["outlier_status"].eq("confirmed_error")
verified = raw_analysis.mask(confirmed_error)

lower, upper = verified.quantile([0.01, 0.99])

scenarios = {
    "keep_all": raw_analysis,
    "remove_confirmed_errors": verified,
    "winsorize_verified_1_99": verified.clip(lower, upper),
}

Loading interactive lab...

Loading concept check...

Loading practice...

Chapter 5 cleaning contract

A defensible cleaned dataset should retain:

Immutable raw fields or a versioned raw source.

Missingness, parsing, duplicate, normalization, and outlier flags.

Corrected or standardized analytical fields beside raw representations.

Mapping tables, thresholds, conversion factors, survivor rules, and their versions.

Before/after counts and validation results.

Review decisions, unresolved cases, and sensitivity reports.