AI Engineering with Chip Huyen

Key Concepts

AI Engineering, Machine Learning Engineering, Foundation Models, Prompt Engineering, Retrieval Augmented Generation (RAG), Fine-tuning, Data Set Engineering, Model Evaluation, Context Length, Multimodality, API Calls, Product Focus, Data-Driven Approach, Model Hosting, Open Source Alternatives, Evaluation Metrics, Functional Correctness, AI as a Judge, Comparative Evaluation, BM25, Customer Support Automation, CrR Framework, Human-in-the-Loop, Data Inspection, System Thinking, Complex Adaptive Systems.

Defining AI Engineering

Chip Huyen defines AI engineering as a field that focuses on building applications leveraging machine learning or AI through direct API calls to foundation models. This lowers the entry barrier, reducing the need for extensive data and specialized AI degrees. It represents a shift from machine learning to engineering and product development.

AI Engineering vs. Machine Learning Engineering

| Feature | Machine Learning Engineering | AI Engineering

Core Differences in Responsibilities

• Machine Learning Engineer: Focused on the entire lifecycle of a model, from data gathering and annotation to training and deployment.
• AI Engineer: Focuses on product and data, leveraging existing AI capabilities (often via APIs) to build and integrate AI-powered features into applications.

Steps to Build an AI Application

1. Product-First Approach: Start with a demo and validate the product idea.
2. Data Acquisition: Gather data to improve the application, focusing on prompt examples and evaluation data.
3. Model Integration: Initially, use API calls to services like OpenAI, Anthropic, or Google.
4. Model Development (Optional): If API costs become prohibitive, consider hosting open-source models or fine-tuning existing models.

Practical Ways for Software Engineers to Get Started

1. Understand Good vs. Bad Responses: Define clear guidelines for model behavior.
2. Prompt Engineering: Craft effective prompts and add examples.
3. Evaluation: Create a set of queries and expected responses, using both automated metrics and human evaluations.
4. Retrieval Augmented Generation (RAG): Augment model context with relevant documents.
5. Fine-tuning (Last Resort): Consider fine-tuning only after exhausting other options.

RAG Implementation Details

• Start Simple: Begin with keyword retrieval instead of immediately jumping to vector search.
• Chunking: Break documents into smaller chunks to fit within context length limits.
• Data Preparation: Focus on data preparation techniques like keyword extraction and metadata addition.
• Hybrid Search: Combine term-based retrieval (e.g., BM25) with vector databases for improved results.
• BM25 Benchmark: Benchmark retrieval systems against BM25 for a baseline comparison.

Challenges with Fine-tuning

• Hosting and Maintenance: Fine-tuned models require hosting and maintenance.
• Resource Intensive: Fine-tuning can be computationally expensive.
• Model Obsolescence: Rapid advancements in AI models can quickly make fine-tuned models obsolete.

Customer Support Automation: A Case Study

1. Identify Bottlenecks: Determine the key challenges in the current customer support process.
2. Consider Alternatives: Explore solutions beyond GenAI, such as routing requests to the right department.
3. CrR Framework: Implement a phased approach, starting with human-in-the-loop systems and gradually increasing automation.

Common Mistakes in Building AI Applications

1. Using GenAI When Not Needed: Applying GenAI to problems that can be solved with simpler methods.
2. Giving Up Too Early: Abandoning GenAI due to initial failures without proper evaluation and optimization.
3. Overcomplicating Solutions: Jumping to complex techniques like vector databases or fine-tuning prematurely.
4. Ignoring Data Quality: Neglecting data preparation and quality control.
5. Blindly Adopting Frameworks: Using AI agent frameworks without understanding their underlying mechanisms and potential drawbacks.

Evaluating AI Systems

• Challenges: As AI becomes more intelligent, human evaluation becomes more difficult.
• Functional Correctness: Evaluate based on how well the application performs its intended task.
• AI as a Judge: Use AI models to evaluate other AI models.
• Comparative Evaluations: Compare different versions of outputs to identify the better one.
• Human Evaluations: Maintain consistent human evaluations to correlate with automated metrics.
• Data Inspection: Manually inspect data to detect patterns and understand user behavior.

Learning AI Engineering

• Project-Based Learning: Choose a project and work through it, solving problems as they arise.
• Structured Learning: Complement project-based learning with courses, books, and research papers.
• Tutorials: Use tutorials, but critically analyze each step and ask "why" questions.
• Personal Automation: Identify tasks that can be automated with AI and build tools to automate them.

The Future of Software Engineering

• AI will automate coding tasks, but it will not replace software engineers.
• Software engineering is about problem-solving and creating executable programs, not just writing code.
• AI will enable software engineers to build more complex and sophisticated systems.

Exciting Use Cases for AI

• Education: AI can personalize learning and help people learn faster.
• Entertainment: AI can create intellectually stimulating content and adapt content across different mediums.
• Enterprise: AI can improve organizational efficiency by automating information aggregation and decision-making processes.

Rapid Fire Questions

• Programming Languages: Python and JavaScript.
• Favorite LLM: No specific favorite; uses different models for different tasks.
• Neat AI Tool: A custom tool for researching papers and extracting relevant information.
• Recommended Books:
  • Complex Adaptive Systems
  • The Selfish Gene
  • Antifragile

Conclusion

AI engineering is an evolving field that requires a product-focused, data-driven approach. While AI offers powerful capabilities, it's crucial to understand the underlying problems, evaluate solutions rigorously, and avoid common pitfalls. By combining practical experience with structured learning, software engineers can effectively leverage AI to build innovative and impactful applications.