(ML 15.3) Logistic regression (binary) - intuition

Key Concepts:

• Logistic Regression: A classification method, despite its name.
• Actuary: A statistician who assesses risk, particularly for insurance and finance.
• Probability Modeling: Creating models to predict the likelihood of events.
• Linear Combination: A sum of variables multiplied by coefficients.
• Sigmoid Function: A generic term for an S-shaped curve used to map values to a range between 0 and 1.
• Logistic Function: A specific sigmoid function, 1 / (1 + e^(-a)), used in logistic regression.
• Binary Classification: Classification with two possible outcomes.
• Level Sets: Lines or surfaces where a function has a constant value.

1. Introduction to Logistic Regression

• Logistic regression is a classification method, not a regression method, despite its name.
• It is widely used, especially in medicine, biostatistics, and social sciences.
• It provides a foundation for more complex methods like neural networks and generalized linear models.
• The video aims to provide an intuitive understanding of the logistic regression model.

2. Motivating Example: Actuarial Modeling

• Scenario: An actuary wants to model the probability of a person dying in the next 10 years (P(death|X)).
• Variables (X):
  • X1: Age of the person.
  • X2: Gender (Male/Female).
  • X3: Cholesterol level.
• Goal: To find a simple model with few parameters.

3. Building a Linear Model

• A linear combination of the variables is considered: w0 + w1*X1 + w2*X2 + w3*X3.
• The weights (w1, w2, w3) determine the influence of each variable on the outcome.
• For example, a positive w1 (for age) suggests that as age increases, the probability of death increases.
• This linear combination can be written as a vector dot product: W^T * X, where X = [1, X1, X2, X3].
• Problem: The linear combination is not a probability (it can be any real number).

4. Applying the Sigmoid Function

• To convert the linear combination into a probability, a sigmoid function is applied.
• Sigmoid Function: A generic term for an S-shaped curve that maps values to the range [0, 1].
• The model becomes: P(death|X) = sigmoid(W^T * X).
• This ensures the output is always between 0 and 1, representing a probability.

5. The Logistic Function

• A standard choice for the sigmoid function in logistic regression is the logistic function.
• Logistic Function: sigma(a) = 1 / (1 + e^(-a)).
• The complete logistic regression model is: P(death|X) = 1 / (1 + e^(-W^T * X)).
• The video focuses on binary classification, but logistic regression can be extended to multiclass problems.

6. Visualizing the Probabilities

• Simplified Scenario: X2 (gender) is fixed to 0, and w0 is set to 0.
• Only X1 (age) and X3 (cholesterol) are considered.
• The vector W = [w1, w3] represents the direction of increasing probability.
• Level Sets: Lines orthogonal to W, representing constant values of W^T * X.
• In 3D (age, cholesterol, probability), the sigmoid function creates a surface that starts near 0, climbs to 0.5 along the line where W^T * X = 0, and approaches 1 as X increases in the direction of W.
• The offset w0 shifts this surface.

7. Conclusion

• The video provides an intuitive understanding of how logistic regression models probabilities using a linear combination of variables and a sigmoid (logistic) function.
• The visualization helps to understand how the model assigns probabilities based on the input features.
• The next video will formalize logistic regression and discuss how to find maximum likelihood estimates using Newton's method.