The Tea Lady Experiment - Numberphile

Key Concepts

• Hypothesis Testing: A statistical method used to determine whether there is enough evidence to reject a null hypothesis in favor of an alternative hypothesis.
• Null Hypothesis (H0): A statement that there is no effect or no difference. It's the "boring" hypothesis that researchers try to disprove.
• Alternative Hypothesis (H1): A statement that contradicts the null hypothesis, suggesting there is an effect or a difference.
• Test Statistic: A value calculated from sample data that is used to determine whether to reject the null hypothesis.
• Asymptotic Properties: The behavior of a statistic as the sample size approaches infinity.
• Permutation Test: A type of resampling method used to determine the significance of a test statistic by rearranging the observed data.
• Resampling Method: A statistical technique that involves repeatedly drawing samples from the original data to estimate the sampling distribution of a statistic.
• Significance Level (α): The probability of rejecting the null hypothesis when it is actually true (Type I error). Commonly set at 5% (0.05).
• P-value: The probability of obtaining a test statistic as extreme as, or more extreme than, the one observed, assuming the null hypothesis is true.
• Quantiles/Percentiles: Values that divide a distribution into equal parts.
• Binomial Distribution: A discrete probability distribution that describes the probability of obtaining a certain number of successes in a fixed number of independent trials.
• Completely Connected Graph: A graph in which every pair of distinct vertices is connected by a unique edge.

Hypothesis Testing Fundamentals

The video begins by explaining the basic framework of hypothesis testing.

1. Data Collection: You start with a set of data samples (x1, x2, ..., xn) drawn from a distribution. For example, the data might be heights of four-year-olds.
2. Formulating Hypotheses: Define a null hypothesis (H0) and an alternative hypothesis (H1).
   • Example:
     • H0: The average height of four-year-olds is greater than 36 inches.
     • H1: The average height of four-year-olds is not greater than 36 inches.
3. Choosing a Test Statistic: Select a test statistic (T(x)) that summarizes the data in a way that is relevant to the hypothesis.
   • Example: The average height of the sampled four-year-olds.
4. Decision Rule: Establish a rule for rejecting the null hypothesis based on the value of the test statistic. This often involves comparing the test statistic to a threshold (c).
   • Example: If T(x) > c, reject H0.
5. Asymptotic Properties: Utilize asymptotic properties to approximate the distribution of the test statistic for large sample sizes. This often involves using a normal distribution.
6. Significance Level: Choose a significance level (alpha), typically 5%, which represents the probability of rejecting the null hypothesis when it is true.
7. P-value Calculation: Calculate the p-value, which is the probability of observing a test statistic as extreme or more extreme than the one observed, assuming the null hypothesis is true.
8. Decision: If the p-value is less than the significance level, reject the null hypothesis.

Case Study: Disease Spread on Networks

The video then discusses a research project involving modeling infectious disease spread on networks.

1. Problem: The goal was to determine whether a specific network configuration was more likely than a completely connected graph (where everyone is symmetrically related) based on observed infection data.
2. Data: The data consisted of a single observation of who was infected in a group of individuals (e.g., classmates).
3. Challenge: Unlike standard hypothesis testing, there were not multiple independent observations of the same phenomenon. The sample size was limited to the number of individuals in the network (e.g., five people).
4. Solution: Permutation Testing: The researchers used a permutation test to address the challenge of limited data and non-standard data structure.

Fischer's Lady Tasting Tea Experiment: A Detailed Explanation of Permutation Testing

The video provides a detailed explanation of permutation testing using Fischer's famous "lady tasting tea" experiment.

1. Scenario: A lady claimed she could distinguish whether milk was added before or after tea.
2. Null Hypothesis: The lady has no special ability and is guessing randomly.
3. Alternative Hypothesis: The lady has a special ability to distinguish the tea.
4. Experimental Setup: Fischer prepared eight cups of tea, four with milk first and four with tea first. The lady had to identify which cups had milk first.
5. Test Statistic: The number of cups the lady correctly identified.
6. Permutation Approach: Instead of relying on a standard distribution, Fischer considered all possible arrangements of four "milk first" cups and four "tea first" cups.
7. Calculating the Distribution:
   • There are 8 choose 4 (70) possible arrangements.
   • For each arrangement, the number of correct identifications can be calculated.
   • This creates a discrete distribution of the test statistic under the null hypothesis.
8. P-value Calculation: The p-value is the probability of observing a test statistic as extreme or more extreme than the one observed, assuming the lady is guessing randomly.
   • Example: If the lady correctly identified six cups, the p-value is the probability of getting six or eight cups correct by random chance.
9. Decision: If the p-value is less than the significance level (e.g., 5%), reject the null hypothesis and conclude that the lady has a special ability.

Example Calculation:

• If the lady gets six cups correct, the p-value is (16/70 + 1/70) = 17/70 ≈ 24%.
• Since 24% > 5%, you would not reject the null hypothesis.

Applying Permutation Testing to Disease Spread

The video returns to the disease spread example and explains how permutation testing was applied.

1. Test Statistic: The number of edges between infected nodes in the alternative hypothesis graph.
2. Permutation: Generate all possible infection scenarios on the same five nodes, keeping the number of infected individuals constant (e.g., three infected).
3. Distribution: Calculate the test statistic for each permutation, creating a discrete distribution.
4. P-value: Determine the probability of observing a test statistic as extreme or more extreme than the one observed, assuming random infection.
5. Decision: Based on the p-value, decide whether to reject the null hypothesis in favor of the alternative hypothesis.

Conclusion

The video provides a detailed explanation of hypothesis testing and permutation testing, emphasizing the importance of creating more data through resampling when dealing with limited observations or non-standard data structures. Fischer's lady tasting tea experiment serves as a clear illustration of the permutation testing methodology. The disease spread example demonstrates a real-world application of this technique. The video also highlights the role of the Lever Hume Trust in supporting Blue Skies Research.