Deep Learning in Economics II

Key Concepts

• Unstructured Data: High-dimensional, complex data (text, images) that cannot be directly used in economic analysis.
• Structured Data: Low-dimensional, interpretable data extracted from unstructured data, suitable for statistical analysis.
• Deep Neural Networks (DNNs): State-of-the-art tools for large-scale feature extraction from unstructured data.
• Measurement Error: The error introduced when using DNNs to predict structured data, which is unlikely to be classical.
• MARS (Missing At Random Structured data): A framework developed to conduct valid, efficient, and robust inference on estimates that incorporate unstructured data.
• Ground Truth: Data obtained through a costly process (e.g., annotation by experts) that serves as a reliable measure of the structured data.
• Missing At Random (MAR): An assumption that after adjusting for observables, annotated and unannotated data are comparable in their ground truth values.
• Annotation Score Function: The probability of annotating a given observation, given observables.
• Semiparametric Efficiency: Achieving the lowest possible asymptotic variance of an estimator.
• Robustness: An estimator's ability to relax rate requirements for first-step estimates in multi-step procedures.
• Efficient Influence Function: A function used to construct robust and efficient estimators.
• Debiasing: Correcting for the bias introduced by using predictions from DNNs.

1. Motivation and Background

• Economists increasingly use unstructured data (text, images, satellite data) to derive structured information for analysis.
• Traditionally, extracting structured information was costly, relying on large-scale external initiatives (e.g., conflict data sets).
• Advances in computing and deep learning have reduced the costs of extracting structured information, enabling economists to create their own data sets.
• However, neural networks do not generically produce unbiased predictions in finite samples, and measurement error is unlikely to be classical.
• Biases in predictions from neural networks propagate to estimators, affecting both point estimates and uncertainty quantification.
• The availability of off-the-shelf neural networks raises concerns about p-hacking.
• It's crucial to take the potential bias from neural networks seriously and correct standard errors and point estimates accordingly.
• A key question is determining how good the neural network predictions need to be, which depends on how the bias propagates to the economic quantity of interest.

2. The MARS Framework

• The MARS framework addresses the challenges of using unstructured data by framing inference as a missing data problem.
• It builds upon Rubin's classic 1976 missing at random mechanism.
• The key idea is to collect a validation sample with ground truth values to estimate the bias in the imputed data and adjust estimates accordingly.
• Ground truth is not necessarily truth in some deep sense, but rather data obtained through a costly process (e.g., annotation by experts).
• The validation sample must meet the missing at random assumption, meaning that after adjusting for observables, annotated and unannotated data should be comparable in their ground truth values.
• If the missing at random assumption is not met, there is no information to correct the biases in the unlabeled data.
• The MARS framework directly addresses the fundamental limitation of neural networks, which is that their bias shifts in complex ways with the training data.

3. Theoretical Contributions

• The MARS framework unifies existing work on inference with black-box AI models and integrates it with older literatures on measurement error, missing data, and causal inference.
• It identifies estimators that are both unbiased and efficient, addressing the lack of focus on efficiency in the existing literature.
• It addresses a wide variety of settings that come up in economics that the existing literature does not address.

4. Efficiency Considerations

• To achieve asymptotic efficiency, the imputation of missing structured data should depend not only on the unstructured data but also on context-specific structured variables that help to estimate the target parameter.
• This is related to deep and wide learning, where both deep learning and structured data are incorporated.

5. Robustness Considerations

• In the context of semiparametric inference, robustness means that the estimator relaxes the rate requirements for first-step estimates in multi-step procedures.
• MARS is weakly doubly robust, meaning that it can tolerate arbitrarily large bias in the estimation of the neural network because it is trading off that bias against the fact that we know our annotation function.

6. Implementation Steps

1. Ensure that the target parameter would be identified if there were no missing structured data.
2. Derive the efficient influence function.
3. Construct the robust and efficient estimator by adding a one-step correction to a plug-in estimator based off the efficient influence function.
4. Do sample splitting for estimation, using the data for debiasing only for that purpose.

7. Estimators

• The framework is illustrated on commonly used estimators, including descriptive means, linear regression, treatment effects identified through linear IV models, modern differences in differences, and regression discontinuity.
• All these estimators are very similar because they are essentially all mean functionals.

7.1. Descriptive Means

• The robust efficient one-step estimator is the augmented inverse propensity weighted estimator.
• It can be rewritten as a correction term added to the mean of the predictions from the big sample.
• It can also be expressed as the estimate in the ground truth sample adjusted with a term that leverages the full sample of data.

7.2. Linear Regression

• The estimator takes the form of standard linear regression, but with the outcome variable adjusted for the measurement error.
• The imputation function should be a function not just of the unstructured data but also of other control variables relevant to the regression.

7.3. Linear IV

• The estimator takes the form of standard IV regression, but with the treatment variable adjusted for the measurement error.

7.4. Differences in Differences and RDD

• These estimators are also based on differences in means and involve adjusting the outcomes for the imputation error.

8. Extensions

8.1. Aggregated Predictions

• If the regression is not at the level of the text or images, the predictions from the neural network can be aggregated and potentially transformed nonlinearly.
• In this case, the mean of the variable can be estimated with the MARS mean estimator, and then the regression can be run with the unbiased measure, adjusting for the classical measurement error.

8.2. Rare Events

• If the data of interest is a rare event, it is important to have both positive and negative examples in the annotated data.
• Economists often use a keyword method to deal with this issue, but this can violate the strong overlap assumption.
• The statistical literature suggests using an importance sampling approach.

9. Empirical Examples

9.1. Economic Policy Uncertainty (EPU) Index

• The framework is applied to the EPU index, which measures economic policy uncertainty based on newspaper articles.
• The keyword search measure used in the original paper is debiased, and a neural network is trained and used for debiasing as well.
• The results show that the confidence intervals are much tighter if the fact that the data were generated is ignored.
• The bias is not huge, but there is some systematic downward bias relative to the debiased estimates.
• The framework is also applied to a regression of the change in employment on the change in log EPU interacted with an intensity measure.
• The results show that accounting for the classical measurement error attenuates the coefficients.

9.2. Geopolitical Risk Index

• The framework is applied to a geopolitical risk index, which measures geopolitical risk based on newspaper articles.
• The results show substantial downward bias in the keyword classifier.
• The framework is also applied to a regression of disaster aid on geopolitical risk.
• The results show that it does not make a huge amount of difference to adjust for the classical measurement error.

10. Conclusion

• Deep learning provides powerful tools for processing data, but it is important to take into account that these models are not perfect.
• The MARS framework provides a principled way to incorporate the measurement error from neural networks.
• Accounting for imputation bias is important and can influence both point estimates and uncertainty quantification.