Deep Learning in Economics I

Key Concepts

• Deep Learning for Unstructured Data
• Transformers
• Transfer Learning
• Contextual Embeddings
• Vector Representations
• Classification
• Regression (Machine Learning Definition)
• Contrastive Learning
• Record Linkage
• OCR (Optical Character Recognition)
• Generative AI

Deep Learning for Unstructured Data

• Main Topic: The lecture focuses on using deep learning to process unstructured data (text, images, audio) and incorporating these tools into econometric research.
• Key Points:
  • Deep learning revolutionizes unstructured data processing by extracting information for analysis.
  • In economics, this involves extracting sentiment or topics from text.
  • The goal is to convert complex data into a computable format.
  • The transformer architecture is central to processing various data types.
• Technical Terms:
  • Unstructured Data: High-dimensional, complex data that cannot be directly used in analysis.
  • Transformer: A neural network architecture used for processing sequential data like text and images.
  • Continuous Vector Space: Mapping high-dimensional data into lower-dimensional vectors with numerical values.
• Logical Connections: The lecture introduces the concept of deep learning for unstructured data and sets the stage for discussing specific applications and econometric considerations.

Why Neural Networks?

• Main Topic: The lecture explains the advantages of using neural networks over traditional methods for unstructured data.
• Key Points:
  • Transfer Learning: Deep neural networks leverage information from massive datasets, reducing the need for problem-specific data.
  • Context: Neural networks consider the context of words or pixels, improving accuracy.
  • Scalability: Continuous vector computations are highly optimized, enabling large-scale analysis.
• Example: State-of-the-art language models are trained on trillions of tokens, providing a strong foundation for specific tasks.
• Argument: Neural networks are powerful due to their ability to compress vast amounts of information and adapt to specific problems.
• Notable Quote: "Deep neural networks are the state-of-the-art tool for estimating really complicated functions."

Classification vs. Embeddings

• Main Topic: The lecture differentiates between classification and embeddings for imputing information from unstructured data.
• Key Points:
  • Classification: Imputing a prespecified discrete class (e.g., economic policy uncertainty).
  • Regression: Imputing a continuous number (e.g., coordinates of a street vendor).
  • Embeddings: Imputing relationships in data where classes are not specified in advance.
  • Generative AI performs classification by predicting the most likely word in the vocabulary.
• Step-by-Step Process:
  1. Encode unstructured data into low-dimensional vector representations.
  2. Add a classification layer to map vectors to class probabilities or a regression layer to map vectors to a continuous number.
  3. Alternatively, work directly with vector representations (embeddings).
• Logical Connections: The lecture builds on the previous sections by providing a framework for different types of imputation problems and introducing the concept of embeddings.

Classification in Detail

• Main Topic: The lecture provides a detailed explanation of classification using transformer neural networks.
• Key Points:
  • Transformer neural networks use the same pre-trained language model as the backbone for various classification tasks.
  • The BERT model is used as an example.
  • Classification can be done at the text level or to classify the relationship between two texts.
• Technical Terms:
  • BERT: Bidirectional Encoder Representations from Transformers, a transformer-based language model.
  • Class Token: A special token that represents the entire text.
  • SEP Token: A special token that separates two texts.
• Logical Connections: The lecture expands on the concept of classification by explaining how transformer models can be used for different classification tasks.

Generative AI for Classification

• Main Topic: The lecture discusses the use of generative AI (e.g., GPT) for classification tasks.
• Key Points:
  • Generative AI can be used for classification by providing a prompt and asking it to classify the text.
  • Custom-trained models are often more accurate, but GPT can be effective in some cases.
  • It is important to engineer the prompt on a separate set of data from the test set to avoid overfitting.
• Example: A JL article compared GPT to custom-trained models for 19 different topic classification tasks.
• Argument: It is cheap and easy to try GPT for classification tasks, but it is important to evaluate its performance.
• Notable Quote: "Always say like you know why not try uh GPT, try Gemini, try Claude, see how it does."

Embeddings in Detail

• Main Topic: The lecture provides a detailed explanation of embeddings and their applications.
• Key Points:
  • Embeddings are the dense vectors produced by a deep neural network.
  • It is generally not recommended to use embeddings of off-the-shelf transformer models.
  • Contrastive training can be used to make the distances between the embeddings meaningful.
• Technical Terms:
  • Embeddings: Dense vector representations of data.
  • Contrastive Training: Training a model such that instances that belong to the same class have similar embeddings and instances that belong to different classes have dissimilar embeddings.
• Logical Connections: The lecture expands on the concept of embeddings by explaining how they can be used for different tasks.

Applications of Embeddings

• Main Topic: The lecture provides several examples of how embeddings can be used to measure things and perform tasks.
• Key Points:
  • Detecting reproduced content in historical newspaper articles.
  • Identifying the biggest news story of the year.
  • Record linkage for merging data sets.
  • OCR (Optical Character Recognition).
• Examples:
  • The Newswire Data Set: A data set of 138 million front-page articles from historical US newspapers.
  • Link Transformer: A package for using large language models for tasks like merging, dduplication, and clustering.
  • Clippings: A multimodal model for linking Japanese firms across time.
• Logical Connections: The lecture provides concrete examples of how embeddings can be used to solve real-world problems.

Regression

• Main Topic: The lecture provides a brief overview of regression in machine learning.
• Key Points:
  • Regression refers to the prediction of continuous numbers.
  • It works just like classification, but the layer is predicting a continuous number instead of class scores.
  • Document image analysis is an example of a regression problem.
• Example: Layout Parser: An open-source package for document image analysis.
• Logical Connections: The lecture provides a brief overview of regression and its applications.

Conclusion

• Main Takeaways:
  • Deep learning provides powerful tools for processing unstructured data.
  • Transformers are a central architecture for processing various data types.
  • Embeddings are a powerful tool for measuring things and performing tasks.
  • It is important to account for the fact that these models are not perfect in research.
• Synthesis: The lecture provides a comprehensive overview of deep learning for unstructured data, covering the key concepts, techniques, and applications. It emphasizes the importance of understanding the limitations of these models and accounting for them in research.