11.3 Time Indexes, Resampling, and Rolling Windows

Clustering and PCA treated rows as exchangeable points. Time series break that assumption: order matters, spacing carries meaning, and a feature available tomorrow cannot explain a prediction made today. Before forecasting, we need a temporal data contract.

Our running series is daily order volume. Every value must state its timezone, interval ownership, availability time, aggregation rule, and whether a missing timestamp means zero activity or missing collection.

Turn event timestamps into trustworthy calendar bins

Use pd.to_datetime deliberately. A timestamp string without an offset is timezone-naive; one with a timezone is timezone-aware. tz_localize declares the timezone of naive wall-clock values, while tz_convert changes an already aware timestamp to another zone. Daylight-saving transitions can create ambiguous or nonexistent local times, so keep raw UTC and document conversion policy.

After parsing, audit the index:

• Is it a DatetimeIndex in the intended timezone?

• Is it sorted monotonically?

• Are duplicate event IDs errors, while duplicate timestamps are legitimate simultaneous events?

• Are gaps expected inactivity, sensor downtime, or missing batches?

• Is the timestamp event time, ingestion time, or the time the value became usable?

resample is a time-based groupby followed by aggregation. Turning event rows into hourly operations can use different reductions for different meanings:

hourly = (
    events.set_index("event_time")
    .resample("1h", closed="right", label="right", origin="start_day")
    .agg(
        orders=("order_id", "size"),
        revenue_yuan=("revenue_yuan", "sum"),
        avg_wait_min=("wait_min", "mean"),
    )
)

Counts and flows often use sum or size; state variables such as temperature may use mean, last, min, or max. Aggregation is semantic. Summing an average wait or averaging a total order count creates a number but not the intended measure.

Two parameters solve different boundary questions. closed decides which interval owns an event exactly on an edge. label decides whether the result is timestamped with the interval's left or right edge. A careless choice can aggregate later observations and label them at an earlier time, creating look-ahead in a forecasting table.

The resampling lab turns the timeline into a workbench. Carve your own unequal interval boundaries directly into the raw event stream, then write a small executable aggregation expression such as count(), sum(revenue), or mean(wait). Boundary events are orange, so changing endpoint ownership visibly moves real rows between bins while output labels remain an independent choice.

Downsampling combines many fine observations into fewer coarse intervals. Upsampling creates a finer index and therefore new missing positions. asfreq changes frequency without aggregating. Forward fill, interpolation, or zero fill makes a claim about the data-generating process; none is a neutral formatting step. Preserve an observation or coverage flag so imputed values are distinguishable from measured values.

Make every rolling feature obey an availability boundary

A rolling statistic summarizes a moving local history. With a fixed count window, rolling(7) uses seven rows. With a time-offset window, rolling("7D") uses all rows inside seven calendar days. The two differ when dates are missing or observations are irregular.

For a trailing mean with window $w$,

The definition deliberately ends at $t-1$. If the feature predicts $y_t$, including $y_t$ in its own mean leaks the answer. In pandas, shift the target before rolling:

past_orders = daily["orders"].shift(1)

daily["orders_lag_1"] = daily["orders"].shift(1)
daily["orders_lag_7"] = daily["orders"].shift(7)
daily["orders_mean_7d"] = past_orders.rolling(
    "7D",
    min_periods=4,
).mean()

min_periods states how much evidence is required. Leading NaN values are honest: early rows do not possess enough history. Dropping them changes the eligible training population; filling them with a global mean can leak future information. Choose a policy that could run at the original prediction time.

center=True labels a window at its midpoint, which is useful for descriptive smoothing because observations on both sides contribute. It is generally invalid as a real-time forecasting feature because the right half lies in the future. Exponentially weighted means place more weight on recent history, but still need the same shift and availability audit.

The rolling lab asks you to paint the exact observations read by a feature, including custom and non-contiguous footprints, and to place the forecast origin yourself. Each selected dependency is checked against both event time and a configurable availability delay. Future, target, and late-arriving sources turn orange, making the distinction between a row existing offline and being available at decision time concrete.

Rolling windows require more than a clean formula. Production data may arrive late or be revised. If yesterday's final total is unavailable until 02:00, a midnight forecast cannot use it even though its event date is earlier. Track both event time and availability time, and insert a gap where the prediction protocol requires one.

Always visualize the raw series beside the derived feature. A rolling curve can hide gaps, structural breaks, spikes, and imputed stretches. Add coverage or count panels rather than allowing a smooth line to imply complete evidence.

We now have a regular series and leakage-safe history features. The final section separates trend, seasonality, and remainder, then evaluates forecasting rules on future windows that were never used to fit them.