Supercharge your AI/ML: Designing effective GCP storage

Key Concepts:

• AI/ML Workloads
• Vertex AI Ecosystem
• Google Cloud Storage (GCS)
• PaliGemma Model (Baseline & Fine-tuned)
• Visual Question Answering (VQA)
• Data Ingestion, Preparation, Training, Validation, Serving, Archiving
• Storage Transfer Service
• Vertex AI Colab Enterprise Notebook
• Observability Metrics
• Filestore
• GCS FUSE
• VM Metrics (GPU, Memory, Network)
• GKE Cluster
• Anywhere Cache

1. Introduction: The Role of Storage in AI/ML Workloads

• The demonstration focuses on the critical role of storage solutions within AI/ML workloads, specifically within the Vertex AI ecosystem.
• It addresses the challenges of managing large-scale datasets and AI models.
• Google Cloud Storage (GCS) is presented as an efficient solution for these complexities, particularly in the context of PaliGemma models.
• The demo illustrates how to optimize the management of datasets for efficient model deployment and training.

2. PaliGemma Model: Baseline vs. Fine-tuned

• Baseline PaliGemma:
  • The initial capabilities of the PaliGemma model within the Google Cloud Model Garden are showcased before any custom fine-tuning.
  • An image of Machu Picchu is uploaded, and the Visual Question Answering (VQA) mode is used to describe the image.
  • The baseline model provides a simplified description: "ancient ruins."
  • This demonstrates the model's ability to quickly analyze visual data and provide a relevant initial response.
• Fine-tuned PaliGemma:
  • The same Machu Picchu image is used with the fine-tuned PaliGemma model.
  • The enhanced custom model delivers a rich, detailed description that far exceeds the baseline response.
  • This highlights the improvement achieved through fine-tuning.

3. Demo Architecture: Data Pipeline

• The demo architecture outlines the complete process of data ingestion, preparation, model training and validation, serving, and archiving.

4. Data Ingestion

• A Vertex AI Colab Enterprise notebook script is used to automate the data transfer process from AWS S3 to Google Cloud Storage (GCS) using Storage Transfer Service.

5. Data Preparation

• The dataset is accessed from Google Cloud Storage (GCS).
• Observability metrics are used to track data access patterns.
• This includes monitoring:
  • Data transfer from AWS into GCS.
  • Unpacking of images.
  • Writing of processed data back to GCS.
  • Dataset split within the GCS bucket.

6. Model Training and Validation

• The PaliGemma model is copied to GCS.
• A Filestore instance is created for checkpoints to save the training progress, allowing for resuming training if needed.
• VM resources are configured, and the file system is mounted to enable data access and storage connections for the training workload.

7. Monitoring Training Performance

• Real-time visibility into training workload performance is achieved by monitoring VM metrics.
• Network spikes correlate with checkpoint writes to Filestore.
• GPU and memory utilization are aligned with training cycles.
• Monitoring dashboards show training behavior, including VM boot, data caching using GCS FUSE, and checkpoint writes to Filestore.
• The impact of restarting the training job and clearing the VM cache is observed.
• Training data loads from the GCS FUSE cache, not directly from GCS, demonstrating the performance benefits of caching.

8. Model Serving

• The lowest loss checkpoint is selected post-training.
• This checkpoint is copied from Filestore to Google Cloud Storage for serving.
• The serving GCS bucket is configured, and Anywhere Cache is enabled to optimize read performance.
• The GKE cluster is configured for serving.
• The point of access is copied for later serving requests.

9. Enhanced Image Descriptions

• The enhanced image descriptions generated by the fine-tuned PaliGemma model are demonstrated.
• The results are notably richer than those produced by the initial baseline model.
• The trained model's response is tested on more historical landmarks:
  • Mount Rushmore National Memorial
  • Taj Mahal
  • Colosseum
  • Easter Island

10. GCS Anywhere Cache Demonstration

• To demonstrate the advantages of GCS Anywhere Cache, the GKE cluster is expanded.
• Data loads directly from Anywhere Cache during serving tasks, improving performance.

11. Archiving

• Storage Transfer Service is used to transfer training checkpoints from the Filestore instance to Google Cloud Storage for archiving.

12. Conclusion

• The demonstration concludes by encouraging viewers to explore the power of Google Cloud for AI/ML workloads, including Vertex AI, and its robust storage solutions.
• Viewers are recommended to delve deeper into the features and products showcased to unlock their full potential.