9.3 Bayes' Theorem

Section 9.2 showed that the direction of a condition matters. $P(E\mid H)$ asks how likely evidence $E$ is when hypothesis $H$ is true, whereas $P(H\mid E)$ asks how plausible the hypothesis is after the evidence appears. These two probabilities are not generally equal.

That distinction appears whenever we diagnose a cause from an observation:

• a medical condition may cause a positive test;
• a faulty machine may produce a defective item;
• an email source may produce certain words;
• a hidden game strategy may produce a visible move.

Bayes' theorem converts probability in the first direction into probability in the second direction.

Hypotheses, evidence, and four useful names

An hypothesis is an event representing a possible hidden explanation. Let $H$ mean “the person has the condition.” Let $E$ mean “the test result is positive.”

Four quantities play different roles:

• $P(H)$ is the prior probability, the probability of $H$ before observing $E$.
• $P(E\mid H)$ is the likelihood, the probability of the observed evidence under $H$.
• $P(E)$ is the evidence probability, the overall probability that $E$ occurs.
• $P(H\mid E)$ is the posterior probability, the updated probability of $H$ after observing $E$.

These names describe positions in an update; they are not four new probability rules.

Deriving Bayes' theorem from one intersection

The multiplication rule computes the same intersection in two orders:

and

Set the right sides equal and divide by $P(E)$, assuming $P(E)>0$:

This is Bayes' theorem. Its structure is worth reading in words:

The numerator $P(E\mid H)P(H)$ is $P(E\cap H)$, the probability mass belonging to both the hypothesis and the evidence. Dividing by all evidence mass $P(E)$ restricts attention to cases where the evidence occurred.

Finding the missing denominator

Often $P(E)$ is not given directly. Suppose exactly one of two hypotheses must hold: $H$ or its complement $H^c$. The evidence can arrive through either route:

The two pieces are mutually exclusive, so their probabilities add. Applying the multiplication rule to each gives the law of total probability:

Substitute this expression into Bayes' theorem:

Every term now describes one of the two routes that can produce the evidence.

A screening example: why the base rate matters

Assume the following facts:

• $1\%$ of a population has a condition, so $P(H)=0.01$;
• among people with the condition, the test is positive $90\%$ of the time, so $P(E\mid H)=0.90$;
• among people without the condition, the test is negative $95\%$ of the time.

The last statement is called specificity. To use Bayes' formula, we need the false-positive probability:

We could substitute decimals directly. A population table makes the denominator easier to see. Imagine $10{,}000$ people:

Among the $585$ positive results, only $90$ are true positives. Therefore,

So the posterior probability is about $15.38\%$, not $90\%$. The test's $90\%$ sensitivity is $P(E\mid H)$, not $P(H\mid E)$. The low prior rate creates a large group without the condition, and even a $5\%$ false-positive rate produces many positive results in that group. This is the base-rate effect.

Tune prevalence, sensitivity, and specificity for a virtual population. The population splits into true positives, false positives, false negatives, and true negatives, so the posterior is visible as a selected part of all positive results rather than as an unexplained formula output.

More than two hypotheses

The hidden cause may have several possibilities. Events $H_1,H_2,\ldots,H_k$ form a partition of $Omega$ when they are mutually exclusive and together cover the entire sample space:

The union notation says that at least one of the hypotheses occurs; mutual exclusion says that no two occur together.

The evidence probability is the sum of all possible evidence routes:

The symbol $\sum$ means to add one term for every index $i$ from $1$ through $k$. For a particular hypothesis $H_j$, Bayes' theorem becomes

Example: locating a defective machine

Three machines make all components in a factory:

Let $D$ be the evidence “the selected component is defective.” First calculate the joint contribution of every route:

Then add those contributions:

Finally, normalize machine 3's contribution by the total defective mass:

Machine 3 produces only $20\%$ of all components, yet it is responsible for about $42.11\%$ of defective components because its defect rate is higher.

Sequential evidence means repeated updating

After one update, the posterior can become the prior for the next piece of evidence. Suppose $E_1$ is observed first and then $E_2$. The update follows this pipeline:

To perform the second step, we need the probability of $E_2$ after accounting for $E_1$ under each hypothesis. If a model states that $E_1$ and $E_2$ are conditionally independent given $H_i$, then their joint likelihood factors:

Conditionally independent means independent inside the restricted world where $H_i$ is assumed. This is an additional model assumption, not something Bayes' theorem guarantees. If the clues influence one another even after fixing the hypothesis, multiplying their separate likelihoods would count the evidence incorrectly.

Investigate three suspects by selecting clues in any order. Each clue redistributes posterior probability bars. The case board exposes the prior, each likelihood multiplier, and the normalization step—and allows contradictory-looking evidence to undo an earlier lead.

A disciplined Bayes workflow

1. Define the hypothesis events and the evidence event. 2. Check that the hypotheses form a partition. 3. Record every prior $P(H_i)$ and likelihood $P(E\mid H_i)$ in the correct direction. 4. Multiply to obtain the joint contributions $P(E\cap H_i)$. 5. Add all contributions to obtain $P(E)$. 6. Divide the target contribution by the total evidence probability. 7. Check that all posterior probabilities are between $0$ and $1$ and sum to $1$.

Bayes' theorem updates probabilities of events. Section 9.4 introduces a different tool: a random variable will attach a numerical value to each outcome, allowing us to describe distributions and long-run average values.