Stanford CS221 | Autumn 2025 | Lecture 14: Bayesian Networks and Learning

Key Concepts

• Bayesian Network: A directed acyclic graph (DAG) representing a joint probability distribution over a set of random variables, defined by local conditional distributions.
• Parameter Learning: The process of estimating the values (probabilities) within the local conditional distribution tables from data.
• Maximum Likelihood Estimation (MLE): A statistical principle used to find parameter values that maximize the likelihood of the observed training data.
• Parameter Sharing: A technique where multiple nodes in a network use the same local conditional distribution table, reducing the number of parameters and the amount of data required for learning.
• Laplace Smoothing: A method to prevent overfitting and zero-probability estimates by adding "pseudo-counts" to the observed data.
• Expectation-Maximization (EM): An iterative algorithm used for parameter estimation when some variables in the dataset are unobserved (hidden).

---

1. Bayesian Network Fundamentals

A Bayesian network defines a joint distribution by multiplying local conditional distributions: $P(X_1, ..., X_n) = \prod P(X_i | \text{Parents}(X_i))$.

• Inference: The task of querying the network (e.g., $P(\text{Burglary} | \text{Alarm}=1)$) using exact inference (marginalization) or approximate methods like Rejection Sampling and Gibbs Sampling.
• Conditional Independence: Two variables $A$ and $B$ are independent given $C$ if every path between them is blocked. Paths are blocked by:
  • Causal chains: $X \to C \to Y$ (if $C$ is observed).
  • Common causes: $X \leftarrow C \to Y$ (if $C$ is observed).
  • V-structures: $X \to Z \leftarrow Y$ (blocked unless $Z$ or its descendants are observed).

---

2. Parameter Learning: Fully Observed Setting

When all variables are observed, learning parameters is a "count and normalize" process.

• Methodology:
  1. Count: Iterate through the training data and tally the occurrences of each variable assignment given its parents.
  2. Normalize: Divide the counts by the total occurrences of the parent configuration to ensure the probabilities sum to 1.
• Bookkeeping: For complex networks, maintain a nested dictionary where the first key is the parameter name, the second is the parent assignment, and the third is the node value.
• Parameter Sharing: When multiple nodes (e.g., ratings from different users) share the same distribution, they point to the same table. This is useful when data is sparse, as it pools evidence across nodes.

---

3. Advanced Estimation Techniques

Laplace Smoothing

To avoid assigning zero probability to unseen events (overfitting), add a smoothing factor $\lambda$ (pseudo-counts) to all possible outcomes.

• Effect: As $\lambda \to 0$, the estimate approaches MLE. As $\lambda \to \infty$, the distribution becomes uniform. A common choice is $\lambda = 1$ or $0.1$.

Expectation-Maximization (EM)

Used when data is partially observed (e.g., missing the "Genre" variable).

• E-Step (Expectation): Compute the distribution over the hidden variables given the current parameters and observed data. This creates a "weighted" dataset.
• M-Step (Maximization): Use the weighted counts from the E-step to update the parameters via the standard "count and normalize" approach.
• Properties: EM is guaranteed to increase the likelihood at each iteration and converge to a local maximum. It requires non-uniform initialization to break symmetry.

---

4. Notable Quotes and Perspectives

• On Parameter Sharing: "The real way to think about this is that you have these local conditional tables that are floating out... you hook these tables up to nodes to power them to give them probabilistic life."
• On EM: "It's a chicken and egg problem. If I knew the parameters, I could compute the hidden variables. If I knew the hidden variables, I could estimate the parameters."
• On MLE: "You don't need to do iterative optimization at all. There's just a closed-form solution which is called count and normalize."

---

5. Synthesis

The learning process for Bayesian networks transitions from simple counting in fully observed scenarios to iterative optimization (EM) in partially observed ones. The core principle remains Maximum Likelihood Estimation, which provides a rigorous statistical foundation for the intuitive "count and normalize" approach. By incorporating parameter sharing and Laplace smoothing, one can build robust models that generalize well even with limited data or complex, multi-node structures like Hidden Markov Models.