Python Rust Extensions: Massively Speed Up Your Code

Key Concepts

• Rust Extensions for Python: Writing performance-critical components in Rust and utilizing them within Python projects.
• UV: A Rust-based Python package manager simplifying the process of building and managing Rust extensions.
• Materin: A tool used to build Rust packages for Python, automatically installing them into the Python virtual environment.
• PyO3: A framework facilitating the creation of Rust bindings for Python.
• PI Functions: Rust functions designed to be exposed to Python as callable functions.
• Performance Optimization: Leveraging Rust's speed and efficiency to accelerate computationally intensive Python tasks.
• Sieve of Eratosthenes: An algorithm for finding all prime numbers up to a specified integer.

Extending Python with Rust: A Detailed Guide

This video demonstrates how to write Rust extensions for Python, offering a method to significantly improve performance in computationally intensive tasks. The presenter emphasizes a practical approach, focusing on minimal examples to illustrate the core principles.

1. Setting Up the Development Environment

The initial step involves setting up the necessary tools. The presenter recommends using UV, a Rust-based Python package manager (installable via pip install uv or system package managers). While not mandatory, UV simplifies the process. The core building tool is Materin (uv tool install materin), which compiles Rust code into a Python package and automatically installs it into the project’s virtual environment.

2. Project Structure and Initialization

A new Rust extension project is created using matin new <project_name> --bindings pio3. This command generates a directory structure containing a src directory with a lib.rs file (the Rust extension code) and a space for Python code. Best practice, as noted, is to keep the Python code alongside the extension code within the same project. A virtual environment is initialized using uv venv and activated using source venv/bin/activate.

3. Writing the Rust Extension Code

The lib.rs file contains the Rust code that will be exposed to Python. Functions intended for Python use are defined as PI functions. These functions must return a PyResult, which indicates success or failure.

Example 1: Simple Greeting Function

#[pyfunction]
fn greet_name(name: &str) -> PyResult<String> {
    Ok(format!("Hello {}!", name))
}

This function takes a string reference (&str) as input and returns a formatted greeting string. The Ok() wrapper signifies successful execution.

Example 2: Summing a List of Numbers

#[pyfunction]
fn sum_list(numbers: Vec<i64>) -> PyResult<i64> {
    Ok(numbers.iter().sum())
}

This function takes a vector of 64-bit integers (Vec<i64>) and returns their sum as a PyResult<i64>.

4. Building and Importing the Extension

After writing the Rust code, matrin develop --release compiles the code and installs it as a Python package within the active virtual environment. This allows the Rust functions to be imported and used directly in Python code.

5. Using the Extension in Python

Within the Python script (e.g., main.py), the Rust extension is imported using import <project_name>. The PI functions can then be called as if they were native Python functions.

Example:

import simple_extension

print(simple_extension.greet_name("Florian"))
print(simple_extension.sum_list([1, 2, 3, 4, 5]))

6. Performance Comparison: Prime Number Generation

A more substantial example demonstrates the performance benefits of using Rust. The presenter compares a Python implementation of the Sieve of Eratosthenes algorithm (for finding prime numbers) with its Rust equivalent.

Key Findings:

• The Rust implementation consistently outperforms the Python implementation, even for relatively small values of n (e.g., 1 million).
• For n = 1 million, the Rust version was approximately 15 times faster.
• For n = 100 million, the Rust version was approximately 8 times faster (0.05 seconds vs. 0.4 seconds).
• The speedup is calculated as Python Time / Rust Time.

The presenter notes that the speedup may decrease as n increases, but the performance advantage of Rust remains significant.

7. Technical Details & Terminology

• #[pyfunction]: An attribute in Rust that marks a function as being callable from Python.
• PyResult<T>: A type in Rust representing the result of a function that can either succeed with a value of type T or fail.
• &str: A string slice in Rust, representing a reference to a string.
• Vec<T>: A dynamically sized vector in Rust, capable of storing a collection of elements of type T.
• time.perf_counter(): A Python function used to measure elapsed time with high precision.

Logical Connections

The video follows a logical progression: setting up the environment, creating a project, writing Rust code, building the extension, and finally, using it in Python. The performance comparison serves to highlight the primary motivation for using Rust extensions – speed optimization. The simple examples build understanding before moving to a more complex, performance-critical application.

Conclusion

This video provides a clear and concise introduction to writing Rust extensions for Python. It demonstrates the process from initial setup to performance evaluation, emphasizing the potential for significant speed improvements in computationally intensive tasks. The use of UV and Materin simplifies the development workflow, making it accessible to developers familiar with both Rust and Python. The presenter encourages viewers to explore further and utilize this technique to optimize their Python projects.