Stanford CS221 | Autumn 2025 | Lecture 13: Bayesian Networks and Gibbs Sampling

Key Concepts

• Bayesian Networks (BNs): Graphical models representing joint probability distributions over a set of variables, defined by a Directed Acyclic Graph (DAG) and local conditional probability tables (CPTs).
• Probabilistic Inference: The process of computing the probability of query variables given evidence (e.g., $P(B|A=1)$).
• Rejection Sampling: An approximate inference method that generates samples from the joint distribution and discards those that do not match the evidence.
• Gibbs Sampling: An MCMC (Markov Chain Monte Carlo) algorithm that iteratively updates one variable at a time conditioned on the current values of all other variables.
• Markov Blanket: The set of nodes (parents, children, and co-parents) that shield a variable from the rest of the network, allowing for more efficient local updates in Gibbs sampling.
• Conditional Independence: A property where two variables become independent given the value of a third variable.
• d-separation: A graphical criterion used to determine if two sets of variables are conditionally independent in a Bayesian network.

---

1. Bayesian Networks: Structure and Inference

A Bayesian network is defined by:

1. Variables: A set of random variables ${X_1, \dots, X_n}$.
2. Graph Structure: A DAG representing dependencies (e.g., Alarm depends on Burglary and Earthquake).
3. Local Conditional Distributions: CPTs for each node given its parents.
4. Joint Distribution: The product of all local conditional distributions: $P(X_1, \dots, X_n) = \prod P(X_i | \text{Parents}(X_i))$.

Inference Process:

• Exact Inference: Involves forming the full joint distribution tensor, slicing it based on evidence, marginalizing out irrelevant variables, and normalizing. This is computationally expensive ($O(2^n)$ for $n$ binary variables).
• Approximate Inference: Used when exact inference is intractable.

---

2. Sampling Methodologies

Rejection Sampling

• Process: Generate a full sample from the BN. If the sample matches the evidence, keep it; otherwise, discard it.
• Pros/Cons: Simple to implement and unbiased. However, it is highly inefficient if the evidence is rare (low probability), leading to a high rejection rate.

Gibbs Sampling

• Process: Start with an initial assignment that satisfies the evidence. Iteratively cycle through non-evidence variables, updating each by sampling from its conditional distribution given the current values of all other variables.
• Efficiency: Uses the Markov Blanket to restrict the computation to only the variables that directly interact with the target node, significantly reducing the cost per iteration.
• Limitations: Samples are correlated (Markov Chain). It struggles with highly correlated variables (e.g., $A=B$), where the chain may get "stuck" and fail to explore the state space effectively.

---

3. Conditional Independence and Graph Properties

The lecture highlights that graph connectivity does not always imply dependence, and vice versa.

• Independence vs. Conditional Independence:
  • Case 1 (Chain/Fork): $A \to C \to B$. $A$ and $B$ are marginally dependent but conditionally independent given $C$.
  • Case 2 (Common Cause): $A \leftarrow C \to B$. $A$ and $B$ are marginally independent but conditionally dependent given $C$ (the "explaining away" effect).
• d-separation Algorithm: To check if $A \perp B | C$:
  1. Shade the evidence nodes ($C$).
  2. Remove unshaded leaf nodes recursively.
  3. "Marry" parents (connect parents of the same child).
  4. Check if a path exists between $A$ and $B$ that does not pass through shaded nodes.

---

4. Notable Quotes and Perspectives

• On Inference: "Think about joint distributions... as a database that specifies how the world works."
• On Gibbs Sampling: "It's like doing a three-legged race with someone... you can't really move until the other person moves."
• On Algorithmic Choice: "Rejection sampling is the fastest path to defining a Bayesian network... but it is not the most efficient way to do inference."

---

5. Synthesis and Conclusion

The lecture establishes that while Bayesian networks provide a powerful framework for modeling uncertainty, inference is the primary challenge. Rejection sampling is intuitive but fails with rare evidence, while Gibbs sampling is more robust but sensitive to variable correlation. The Markov Blanket serves as a critical optimization for Gibbs sampling. Finally, understanding conditional independence via d-separation is essential for interpreting the causal and probabilistic relationships encoded within the graph structure. Future lectures will address parameter learning—how to derive the CPTs from data.