2.4 Aggregation, Missing Values, and Numerical Stability

Vectorization transforms elements while preserving array shape. Aggregation moves in the opposite direction: it summarizes many values into fewer values by collapsing one or more axes. The result is meaningful only when you know which axis disappeared and which labels remain.

Use the familiar date-by-zone matrix:

times = np.array([
    [31.0, 38.0, 44.0, 36.0],
    [29.0, 41.0, 47.0, 35.0],
    [33.0, 39.0, 42.0, 37.0],
])

Its shape is (3 dates, 4 zones). With no axis, NumPy aggregates all 12 values:

overall_mean = times.mean()

With an axis, that axis is consumed:

zone_means = times.mean(axis=0)  # shape (4,): one per zone
daily_means = times.mean(axis=1) # shape (3,): one per date

axis=0 does not mean “return rows.” It means “collapse axis 0.” Dates disappear and zones remain. axis=1 collapses columns, leaving one result for each date.

For $n$ valid values $x_1, x_2, \ldots, x_n$, the arithmetic mean is:

The formula makes the denominator visible. Missing and excluded observations change $n$, sometimes differently for each group.

Preserve dimensions when the next operation needs them

Aggregation usually removes the collapsed axis. keepdims=True retains it with length 1:

zone_means_row = times.mean(axis=0, keepdims=True)
print(zone_means_row.shape)  # (1, 4)

centered = times - zone_means_row

The (1, 4) mean broadcasts directly over (3, 4). Keeping dimensions can make axis intention more visible and reduce reshaping later.

Other common reductions include sum, min, max, std, argmin, and argmax. argmax(axis=0) returns the position of the maximum within each column, not the maximum value itself. Positional results need the original axis labels to become meaningful dates or zone names.

NaN represents a missing floating value

NumPy commonly represents a missing floating-point observation with np.nan, meaning “not a number.” It is a special floating value, not zero and not an empty string:

times_with_missing = np.array([
    [31.0, np.nan, 44.0],
    [29.0, 41.0, np.nan],
    [33.0, np.nan, 42.0],
])

Ordinary arithmetic generally propagates NaN:

print(times_with_missing.mean(axis=0))
# [31. nan nan]

Propagation is useful because it prevents missingness from silently disappearing. When the analysis explicitly allows available-case summaries, use a NaN-aware function and report valid counts:

valid_counts = np.sum(~np.isnan(times_with_missing), axis=0)
zone_means = np.nanmean(times_with_missing, axis=0)

print(valid_counts)  # [3 1 2]
print(zone_means)    # [31. 41. 43.]

The Central mean of 41 uses one observation; the North mean uses three. Displaying only means would hide this difference in evidence.

np.nanmean does not infer missing values or prove they are harmless. It only excludes them from the calculation. Ask why values are absent, whether missingness differs by group, and whether the remaining observations still represent the intended population.

Floating-point numbers are approximations

Computers store floating-point values with finite precision. Many decimal fractions cannot be represented exactly in binary:

print(0.1 + 0.2 == 0.3)  # False

For comparisons, use a tolerance appropriate to the domain:

np.isclose(0.1 + 0.2, 0.3)
np.allclose(array_a, array_b)

Tolerance is not permission to ignore any difference. Choose it from measurement precision and the decision being made. A tolerance suitable for delivery minutes may be unacceptable for financial reconciliation.

Precision also depends on magnitude. float32 keeps roughly seven decimal digits of precision. Near 100,000,000, it may be unable to distinguish values separated by one:

large = np.array([100_000_000.0, 100_000_001.0], dtype=np.float32)
print(large[1] - large[0])  # may be 0.0

Both stored values can round to the same representable number. This is not random corruption; it follows from finite precision.

Reformulate unstable calculations

A calculation is numerically stable when small representation or rounding errors do not grow into a large error in the result. Subtracting two nearly equal, very large numbers is risky because their shared leading digits consume precision before the small difference is taken. This effect is often called cancellation.

When meaning allows, subtract a common origin before converting to lower precision:

timestamps64 = np.array([100_000_000.0, 100_000_001.0], dtype=np.float64)
relative32 = (timestamps64 - timestamps64[0]).astype(np.float32)

print(relative32[1] - relative32[0])  # 1.0

The relative representation stores [0, 1], where the important difference is large relative to the values themselves. Other stability practices include using an adequate dtype, avoiding unnecessary round trips between types, checking overflow before integer arithmetic, and using established library functions instead of inventing fragile formulas.

Chapter 2 has established NumPy’s core mental model: a homogeneous array has a shape and typed axes; selection uses coordinates or aligned masks; vectorization expresses whole-array transformations; broadcasting aligns compatible shapes; and aggregation collapses named dimensions under an explicit missingness and precision policy. Chapter 3 will reshape, stack, sort, sample, and profile these arrays while examining views, copies, memory, and performance in greater depth.