Five Steps to Create a New AI Model

Key Concepts

Foundation Models, Fine-tuning, Data Pile, Tokenization, Model Card, Benchmarking, Prompt Engineering, Watsonx (watsonx.data, watsonx.governance, watsonx.ai), Hybrid Cloud, Red Hat OpenShift.

What is a Foundation Model?

A foundation model is a centralized base model that can be adapted to specialized models through fine-tuning. This approach significantly accelerates AI model development compared to building models from scratch for each specific use case. Example: Using a foundational model and fine-tuning it with programming language data to create an AI model for programming language translation.

Five Stages of AI Model Workflow

Stage 1: Prepare the Data

• Objective: Create a base data pile for training the AI model.
• Process:
  • Gather large amounts of data (potentially petabytes) from various sources, including open-source and proprietary data.
  • Perform data processing tasks:
    • Categorization: Classify data (e.g., English, German, Java, Ansible).
    • Filtering: Remove unwanted content like hate speech, profanity, copyrighted material, and private/sensitive information.
    • Deduplication: Eliminate duplicate data entries.
  • Output: A versioned and tagged base data pile, documenting the data used and filters applied for governance purposes.

Stage 2: Train the Model

• Objective: Train the selected foundational model using the prepared data pile.
• Process:
  • Model Selection: Choose a foundational model based on the use case (e.g., generative, encoder-only, lightweight, high parameter).
  • Tokenization: Convert the data pile into tokens, which are the units foundation models operate on. A data pile can result in trillions of tokens.
  • Training: Train the model using the tokens. This process can be computationally intensive and time-consuming, potentially taking months with thousands of GPUs for large-scale models.
• Key Point: The training stage is the most computationally expensive part of the process.

Stage 3: Validate

• Objective: Benchmark the trained model to assess its performance and quality.
• Process:
  • Run the model against a set of benchmarks.
  • Create a model card documenting the model and its benchmark scores.
• Persona: Primarily performed by data scientists.

Stage 4: Tune

• Objective: Fine-tune the model for specific applications and improve its performance.
• Process:
  • Application developers (not necessarily AI experts) engage with the model.
  • Generate prompts to elicit good performance.
  • Provide additional local data to fine-tune the model.
• Key Point: This stage is significantly faster than building a model from scratch, taking hours or days.
• Persona: Primarily performed by application developers.

Stage 5: Deploy

• Objective: Deploy the model for use in real-world applications.
• Process:
  • Deploy the model as a service offering in a public cloud.
  • Embed the model into an application running closer to the edge of the network.
  • Continuously iterate and improve the model over time.

IBM Watsonx Platform

IBM has announced the watsonx platform to support all five stages of the AI model workflow. It consists of three elements:

• watsonx.data: A modern data lakehouse that connects to data repositories for Stage 1 (data preparation).
• watsonx.governance: Manages data cards (Stage 1) and model cards (Stage 3), enabling well-governed AI processes and lifecycles through fact sheets.
• watsonx.ai: Provides a platform for application developers (Stage 4) to interact with and fine-tune the model.

Watsonx is built on IBM's hybrid cloud platform, Red Hat OpenShift.

Conclusion

Foundation models are revolutionizing AI model development by enabling faster creation of specialized AI models through fine-tuning. The 5-stage workflow, supported by platforms like IBM's watsonx, allows teams to build sophisticated AI applications more rapidly.