Monitor Your Python Applications with Prometheus & Grafana

Key Concepts

• Prometheus: A tool for monitoring applications and extracting metrics.
• Grafana: A tool for visualizing metrics obtained from Prometheus.
• Flask: A Python web framework used to create the example application.
• Docker & Docker Compose: Containerization tools used to deploy the application, Prometheus, and Grafana.
• Metrics: Numerical data points that represent the state or behavior of an application.
• Counter: A Prometheus metric type that represents a cumulative value that only increases.
• Summary: A Prometheus metric type that aggregates data over time, providing statistics like averages and quantiles.
• Endpoints: Specific URLs in a web application that handle requests.
• Error Handling: Mechanisms for catching and managing exceptions in an application.
• Scraping: The process by which Prometheus collects metrics from applications.
• Data Source: A connection in Grafana to a source of data, such as Prometheus.
• Dashboard: A collection of visualizations in Grafana that display metrics.

Monitoring Applications with Prometheus and Grafana

Introduction

The video demonstrates how to professionally monitor applications using Prometheus and Grafana, two open-source tools commonly used in production environments. The goal is to provide a basic setup using a Flask application, Docker, and Docker Compose, focusing on getting started rather than an in-depth tutorial.

Prometheus and Grafana Overview

• Prometheus: Collects metrics from applications.
• Grafana: Visualizes the metrics collected by Prometheus.
• The application sends metrics to Prometheus, and Grafana retrieves and displays them.

Setting up the Flask Application

1. Requirements File (requirements.txt):
   • Includes Flask and prometheus_client packages.
2. Flask Application (app.py):
   • Imports necessary modules: flask, request, response from flask, and counter, generate_latest, CONTENT_TYPE_LATEST from prometheus_client.
   • Creates a Flask app instance: app = Flask(__name__).
   • Defines several endpoints:
     • /hello: Returns "Hello, World!".
     • /print_number: Accepts a number as a URL parameter and returns it, also tracking the numbers for average calculation.
     • /crash: Raises a KeyError to simulate an application crash.
     • /metrics: Returns Prometheus metrics in the required format.
3. Exception Handling:
   • Uses app.errorhandler to catch unhandled exceptions.
   • Creates a Counter metric called exceptions to track the total number of unhandled exceptions, grouped by endpoint and exception type.
   • The catch_all function increments the exceptions counter each time an unhandled exception occurs.
4. Request Counting:
   • Uses app.before_request to count all incoming requests.
   • Creates a Counter metric called request_count to track the total number of requests, grouped by method (GET, POST, etc.) and endpoint.
   • The count_requests function increments the request_count counter before each request.
5. Number Summarization:
   • Creates a Summary metric called print_number to track the numbers passed to the /print_number endpoint.
   • The observe method of the Summary object is used to record each number.
6. Metrics Endpoint:
   • The /metrics endpoint returns a response object containing the generated metrics data using generate_latest().
   • The CONTENT_TYPE_LATEST is used to set the correct MIME type for the response.

Docker and Docker Compose Setup

1. Dockerfile:
   • Uses python:3.12-slim as the base image.
   • Sets the working directory to /app.
   • Copies requirements.txt and installs the dependencies using pip install -r requirements.txt.
   • Copies app.py.
   • Exposes port 5000.
   • Sets the command to run the application: python app.py.
2. Prometheus Configuration (prometheus.yaml):
   • Sets the scrape_interval to 5 seconds.
   • Configures a job_name called "flask_app".
   • Specifies the target as flask:5000, where "flask" is the hostname of the Flask application container.
3. Docker Compose File (docker-compose.yaml):
   • Defines three services: flask, prometheus, and grafana.
   • Flask:
     • Builds the application using the Dockerfile in the current directory.
     • Sets the container name to "flask_app".
     • Maps port 5000 to port 5000.
   • Prometheus:
     • Uses the prom/prometheus:latest image.
     • Sets the container name to "prometheus".
     • Mounts the prometheus.yaml file to /etc/prometheus/prometheus.yaml as read-only.
     • Maps port 9090 to port 9090.
   • Grafana:
     • Uses the grafana/grafana:latest image.
     • Sets the container name to "grafana".
     • Maps port 3000 to port 3000.
     • Specifies that it depends_on Prometheus.

Running the Application

1. Build the Docker images using docker compose build.
2. Start the containers using docker compose up.
3. Access the Flask application at localhost:5000.
4. Access Prometheus at localhost:9090.
5. Access Grafana at localhost:3000.

Configuring Grafana

1. Log in to Grafana with the default credentials (admin/admin).
2. Add a data source:
   • Select Prometheus as the data source.
   • Set the URL to http://prometheus:9090.
   • Save and test the connection.
3. Create a new dashboard and add visualizations:
   • Select Prometheus as the data source.
   • Choose a metric to visualize, such as app_requests_total or exceptions_total.
   • Use the code editor to aggregate metrics using functions like sum.
   • Save the dashboard.

Examples and Demonstrations

• The video demonstrates how to visualize the total number of requests over time using the app_requests_total metric.
• It shows how to visualize the total number of exceptions, grouped by exception type, using the exceptions_total metric.
• It explains how to calculate the average of the numbers passed to the /print_number endpoint by dividing the app_print_number_summary_sum metric by the app_print_number_summary_count metric.
• The video also shows how to trigger exceptions by accessing the /crash endpoint and passing non-numeric values to the /print_number endpoint.

Conclusion

The video provides a practical guide to setting up a basic monitoring system using Prometheus and Grafana with a Flask application and Docker Compose. It covers the essential steps for collecting, visualizing, and analyzing application metrics, enabling professional monitoring and troubleshooting. The presenter encourages viewers to explore further customization and more advanced features of Prometheus and Grafana.