1.3 Observations, Variables, and Datasets

The lifecycle tells us to understand data before cleaning or calculating. This section provides the vocabulary for doing that. A rectangular table may look obvious, but many serious errors begin with an unanswered question: What does one row represent?

Consider a delivery table:

An observation is one recorded instance of the entity or event being studied. Here, each row is one order, so the observational unit is an order. A variable is a property recorded consistently across observations, such as zone or delivery minutes. A value is one variable’s entry for one observation, such as 52 for order A105’s delivery time.

The meaning “one order per row” is the dataset’s grain, also called its level of detail. State it as a sentence. If you cannot finish “Each row represents…,” do not aggregate yet.

Mixed grain creates invisible duplication

Suppose a new export contains one row per item inside an order. Order A104 bought three products, so it appears three times. The new grain is an order line, not an order. Counting rows now counts items, not orders.

This becomes dangerous when an order-level value is repeated on every item row:

Summing order_total returns 126 even though the order was worth 42. The arithmetic is valid; the data model is wrong for that arithmetic. To calculate order revenue, keep one order-level value or aggregate line amounts to order grain first.

Datasets can use several related grains:

• A customer table has one row per customer.

• An order table has one row per order.

• An order-line table has one row per product within an order.

• A daily summary has one row per date and perhaps per zone.

Moving between them requires explicit aggregation or joins. Later chapters will implement these operations in pandas. For now, learn to announce the grain before every count, average, or merge.

Storage type is not analytical meaning

A data type describes how a value is stored and which operations are valid. A data role describes what the field means in the analysis. These ideas overlap, but they are not identical.

An employee ID such as 10021 may be stored as an integer, but its mean is meaningless. A rating from 1 to 5 has order, yet the distance between 1 and 2 may not represent exactly the same experience change as the distance between 4 and 5. A date stored as text cannot reliably support duration calculations until parsed as a datetime.

A schema is a contract

A schema documents expected fields, roles, types, allowed values, units, and missingness. It turns assumptions into checks. For the delivery data, part of a schema might say:

• order_id is a non-empty identifier and must be unique at order grain.

• delivery_minutes is numeric, measured in minutes, and must be non-negative when present.

• zone is a category from {North, Central, South}.

• on_time is Boolean and is derived from the documented threshold.

The schema does not claim that reality is clean. It defines what the analysis expects and how violations will be handled. A new zone might be a valid business change rather than bad data. Good validation reports the violation with context; it does not blindly delete every surprise.

Tidy structure makes questions easier to express

A useful default is tidy data: each variable is a column, each observation is a row, and each kind of observational unit is a table. It is not the only possible storage design, but it aligns naturally with filtering, grouping, plotting, and modeling tools.

Wide monthly columns such as sales_jan, sales_feb, and sales_mar encode month names in headers. A tidy version uses a month column and a sales column. Chapter 6 will show how to reshape between these forms. The important idea now is that structure determines which operations are simple and which mistakes are easy to make.

You now have a declared grain and schema. The final section prepares an environment where those declarations, the code that checks them, and the results can be rerun from a clean state.