6.1 Vectorized Column Transformations

Chapter 5 produced cleaner columns while preserving raw evidence. The next task is to derive useful analytical variables: revenue from quantity and price, delay from timestamps, or a review label from several conditions. A vectorized transformation describes such a relationship for an entire Series or array at once.

“Vectorized” does not mean that every operation is magically fast. It means the expression works with whole pandas or NumPy objects instead of manually asking Python to visit one row at a time. This usually makes the data relationship visible in the code and lets optimized array operations do the repetitive work.

Think in aligned columns

Suppose orders has units, unit_price, and cost columns. The row-level formula

becomes a column expression:

orders["profit"] = (
    orders["units"] * orders["unit_price"]
    - orders["cost"]
)

The Series share the DataFrame index, so values align by label before arithmetic. That protection matters when combining independently created Series: equal lengths do not guarantee equal row identity. If two indexes differ, pandas forms their label union and produces missing results where a label has no partner.

For a sequence of dependent columns, assign makes the pipeline explicit without changing the source object:

analysis = orders.assign(
    revenue=lambda d: d["units"] * d["unit_price"],
    profit=lambda d: d["revenue"] - d["cost"],
    margin=lambda d: (
        d["profit"] / d["revenue"]
    ).where(d["revenue"] > 0),
)

Each callable receives the DataFrame produced so far, so profit can use revenue. The margin expression retains missingness where revenue is zero or negative rather than inventing an infinite ratio. Parentheses are also doing real work: they group a multi-line expression without backslashes and make the numerator and guard visible.

Do not round intermediate values merely for display. Store full-precision analytical values, and round only when formatting a table or chart. Early rounding can alter thresholds and accumulated totals.

Build decisions from Boolean masks

A comparison such as orders["delay_minutes"] > 30 returns a Boolean Series aligned with the orders. Masks can be named and combined:

invalid_amount = orders["amount"].isna() | orders["amount"].lt(0)
severe_late = orders["delay_minutes"].ge(60)
high_value_late = (
    orders["amount"].ge(500)
    & orders["delay_minutes"].gt(30)
)

Use &, |, and ~ for element-wise AND, OR, and NOT. Put every comparison in parentheses because Python's operator precedence does not read an English-looking expression the way a person might expect:

# Correct
eligible = (orders["amount"] >= 100) & orders["customer_id"].notna()

For two outcomes, where or np.where is often sufficient:

orders["service_level"] = orders["delay_minutes"].le(30).map(
    {True: "on_time", False: "late"}
)

Be careful: comparisons involving nullable Boolean values can remain pd.NA. Decide whether unknown should remain unknown, become a separate label, or enter review. Silently treating it as false changes the policy.

For several mutually exclusive outcomes, np.select uses the first true condition:

import numpy as np

conditions = [invalid_amount, severe_late, high_value_late]
choices = ["invalid_amount", "severe_late", "high_value_late"]

reason = np.select(conditions, choices, default="ok")
analysis = orders.assign(
    review_reason=pd.Series(reason, index=orders.index, dtype="string"),
    needs_review=invalid_amount | severe_late | high_value_late,
)

Conditions can overlap. A negative-amount order may also be severely late. Therefore, list order is a business rule, not a formatting detail. Name masks, document priority, and build a cross-tab that tests important overlaps.

Other useful vectorized transforms include:

• clip(lower=..., upper=...) to create a declared capped scenario without overwriting the raw value.

• where(condition) to retain values where a condition is true and mark the rest missing.

• mask(condition, replacement) to replace values where a condition is true.

• NumPy universal functions such as np.log1p for mathematical transforms, after checking their input domain.

np.log1p(x) computes $\log(1+x)$ accurately near zero, but it is undefined for $x < -1$. A vectorized function still needs a domain contract.

Validate a transformation, not just its syntax

A successful expression can still encode the wrong population or units. For each derived column, record its inputs, formula, missing-value policy, units, and valid range. Then test useful invariants:

assert analysis.index.equals(orders.index)
assert analysis["revenue"].ge(0).all(skipna=True)
assert analysis.loc[analysis["revenue"].le(0), "margin"].isna().all()

Compare counts before and after, inspect boundary rows, and keep the source fields. The next section focuses on the cases where a built-in expression is not enough—and on choosing among map, apply, aggregation, and other APIs without reaching for a row-wise callback by habit.