MLFlow Crash Course: MLOps in Python

Key Concepts

• MLflow: An open-source AI engineering platform and MLOps toolkit for monitoring, evaluating, debugging, and deploying models.
• GenAI Mode: Features for LLM tracing, prompt management, AI gateways, and LLM evaluation.
• Model Training Mode: Features for hyperparameter tuning, model checkpointing, versioning, and the Model Registry.
• Autologging: A feature that automatically captures metrics, parameters, and artifacts from frameworks like OpenAI, LangChain, Mistral, scikit-learn, and PyTorch.
• Model Registry: A centralized repository to manage model versions and lifecycle stages.
• AI Gateway: A proxy layer for LLM requests that supports load balancing and fallback models.
• Scorers: Custom or built-in functions (e.g., correctness, conciseness) used to evaluate LLM outputs.

---

1. Project Setup and Infrastructure

To begin using MLflow, the user sets up a Python environment (using uv or pip) and installs mlflow, python-dotenv, and relevant AI libraries.

• Server Initialization: Running mlflow server starts both the backend and the web-based dashboard (defaulting to localhost:5000).
• Tracking URI: Python scripts must point to the server using mlflow.set_tracking_uri("http://localhost:5000").
• Experiments: Used to group related runs. mlflow.set_experiment("name") organizes logs, traces, and evaluations.

2. GenAI: Tracing and Evaluation

MLflow provides deep observability into LLM interactions.

• Tracing: By using mlflow.openai.autolog() or mlflow.langchain.autolog(), developers can capture token usage, latency, and request/response payloads.
• Evaluation: The mlflow.genai.evaluate function allows for automated testing of agents.
  • Methodology: Define a predict function and a list of scorers.
  • Custom Scorers: Using the @score decorator, users can define custom logic (e.g., checking if a response is under five words).
• Prompt Management: Prompts can be stored as versioned objects in MLflow. They support variables (e.g., {{num_bullet_points}}) and can be loaded via mlflow.genai.load_prompt("prompts:/name/version").

3. AI Gateway

The AI Gateway acts as a centralized interface for LLM requests, enabling:

• Load Balancing: Distributing traffic across multiple models.
• Fallback Mechanisms: Automatically routing requests to a secondary model (e.g., Mistral) if the primary model (e.g., OpenAI) fails.
• Usage Tracking: Monitoring costs and performance across all gateway-routed requests.

4. Agent Deployment

MLflow facilitates turning agents into production-ready services:

• Framework: Uses FastAPI under the hood.
• Process: Decorate an asynchronous function with @invoke, wrap it in an AgentServer, and run it. This exposes an endpoint (e.g., /invocations) that can be queried via curl or HTTP requests.

5. Classic Machine Learning (scikit-learn & PyTorch)

MLflow manages the traditional ML lifecycle:

• Autologging: mlflow.sklearn.autolog() captures metrics like F1-score, precision, recall, and AUC automatically.
• Model Registry: After training, models are saved as artifacts. They can be registered, versioned, and loaded later using mlflow.pyfunc.load_model.
• Hyperparameter Tuning: When combined with Optuna, MLflow logs every trial's parameters and metrics, allowing users to sort by performance (e.g., lowest error) to identify the best model version.
• Deep Learning Monitoring: For PyTorch, MLflow tracks loss/accuracy curves per epoch and system metrics (CPU/RAM usage) via psutil.

6. Notable Quotes

• "MLflow is an MLOps toolkit because it's all about monitoring, evaluating, analyzing, observing, debugging, and deploying models."
• "If you can express it in language or in Python code, you can use it as a scorer and automatically evaluate your agents."

Synthesis

MLflow serves as a unified bridge between GenAI and traditional machine learning workflows. Its primary value lies in observability—providing a single dashboard to track LLM token costs and latency alongside traditional model training metrics like accuracy and loss. By standardizing the logging of artifacts and parameters, it simplifies the transition from experimental code to production-ready, versioned model deployments.