1.2 The Data Analysis Lifecycle

The previous section built a chain from decision to question, evidence, finding, and action. We now need a working process that preserves that chain while data changes, defects appear, and assumptions are revised.

Data analysis is often drawn as a straight pipeline. In practice, it is a lifecycle: a sequence with deliberate return paths. Discovering a problem and revisiting an earlier stage is not wasted work. It is how an analysis stops a defect from becoming a confident conclusion.

The six stages and their artifacts

Every stage should produce an artifact, something inspectable that survives beyond the analyst’s memory.

The delivery company’s raw export might arrive as deliveries_2026-08-01.csv. Preserve that file unchanged. Cleaning should create a new processed dataset rather than silently overwriting the evidence. A provenance note records the source system, extraction time, filters, and responsible owner. This is data provenance: the documented origin and history of data.

Why the lifecycle loops

Suppose exploration shows that the South zone has many missing delivery times. That is not merely an inconvenience for a chart. It may invalidate the metric, so the analysis returns to understanding and cleaning.

Different discoveries point to different return paths:

• If the audience needs a different decision, return to framing.

• If an important group was never collected, return to acquisition.

• If units, time zones, missingness, or duplicates are wrong, return to understanding and cleaning.

• If two outputs disagree, inspect transformations and calculations before communicating.

• If a finding is too unstable across reasonable definitions, narrow the claim or gather better evidence.

Iteration should be explicit. Record what triggered the return, what changed, and which outputs became obsolete. Otherwise an old chart may survive after its input or metric has changed.

Preserve a lineage from raw input to result

Data lineage is the trace from a source through transformations to an output. A simple project might use this structure:

delivery-analysis/
├── data/
│   ├── raw/          # unchanged source snapshots
│   └── processed/    # rebuildable cleaned data
├── notebooks/        # exploration and explanation
├── src/              # reusable loading and validation code
├── reports/          # generated figures and summaries
└── README.md         # question, source, environment, and run steps

The raw directory is not “whatever file I currently use.” It is the closest preserved copy of what the source provided. Processed files are disposable outputs: if they are deleted, the code should rebuild them from raw inputs.

For each important result, ask a traceability question: “Which source rows, exclusions, transformations, and metric definition produced this number?” If that path cannot be reconstructed, the result may be visually polished but is not auditable.

Separate exploration from confirmation

Exploration is where analysts learn what the data contains, notice unexpected groups, test summaries, and form hypotheses. That freedom is valuable, but it creates a risk: after trying many definitions, one striking pattern can appear by chance.

A disciplined workflow records exploratory choices and distinguishes them from confirmatory checks—checks specified before seeing the result or repeated on fresh data. You do not need a formal experiment for every chart. You do need honest language about whether a result was anticipated or discovered after many attempts.

Also record negative results. If three plausible comparisons showed no meaningful difference, keeping only the fourth, dramatic chart creates a misleading history of the work.

The next section zooms into the central artifact of most analyses: the dataset. Before using NumPy or pandas, you must know precisely what one row represents, what each column means, and whether a number is a quantity or merely a label.