Massively Speed Up Python Code with C Extensions

Key Concepts

• C Extensions for Python
• Global Interpreter Lock (GIL)
• Multi-threading in Python
• Factorial Calculation
• Python C API
• setup.py and pyproject.toml for building C extensions

Speeding Up Python with C Extensions

Introduction

The video demonstrates how to significantly improve the performance of Python applications by using C extensions. The speedup is attributed to two primary factors: the inherent speed advantage of C over Python and the manual release and reacquisition of the Global Interpreter Lock (GIL).

Example: Factorial Calculation

A simple example is used to illustrate the performance gains: calculating the factorial of 20 millions of times in multiple threads. The performance is compared across three scenarios:

1. Pure Python implementation
2. Python with a C extension (GIL enabled)
3. Python with a C extension (GIL released)

Python Implementation (Baseline)

A basic Python function factorial(n) is defined to calculate the factorial of a given number n. A worker function is created to run the factorial calculation multiple times. The main section creates 16 threads, each running the worker function with 5 million repetitions. The execution time is measured using time.perf_counter(). The initial run takes approximately 63.446 seconds, establishing a baseline for comparison.

C Extension Implementation

Creating the C Extension (fast_factorial.c)

1. Includes: The code includes Python.h to access the Python C API. define pi_s size_t clean is used to avoid truncation issues with sizes.
2. C Factorial Function (C_Factorial): A static function C_Factorial is defined in C to calculate the factorial. It takes an unsigned integer n as input and returns an unsigned long long integer (64-bit) to accommodate the factorial of 20. The logic mirrors the Python implementation.
3. Python Wrapper Functions (pi_factorial_repeat_withgil, pi_factorial_repeat_withoutgil):
   • These functions act as interfaces between Python and the C factorial function. They take Python objects (piObject pointers) as input and return a Python object.
   • piArg_ParseTuple is used to parse the arguments passed from Python (n and repetitions) into C variables.
   • The C factorial function is called repeatedly within these wrappers.
   • piLong_FromUnsignedLongLong converts the unsigned long long result from C back into a Python long object.
   • GIL Handling: The pi_factorial_repeat_withoutgil function includes piBegin_AllowThreads() before the loop and piEnd_AllowThreads() after the loop to release and reacquire the GIL, enabling true multi-threading.
4. Method Definition (fast_factorial_methods): A piMethodDef array defines the methods exposed to Python. Each entry specifies the Python function name, the C function to call, the calling convention (meth_varargs), and an optional description.
5. Module Definition (fast_factorial_module): A piModuleDef struct defines the module itself, including the module name, documentation, and the methods it exposes.
6. Module Initialization (piInit_fast_factorial_module): The piModInit_Func function is the entry point for the module. It calls piModule_Create to create and return the module object.

Building the C Extension

1. pyproject.toml: This file specifies the build system requirements (setuptools and wheel) and the build backend.
2. setup.py: This file uses setuptools to define the extension module, its name, version, and the source file (fast_factorial.c).
3. Virtual Environment: A virtual environment is created using python3 -m venv .venv.
4. Installation: The C extension is installed using pip3 install . (from the directory containing pyproject.toml and setup.py).

Integrating the C Extension into Python

A new Python file (main2.py) is created to use the C extension. The fast_factorial_module is imported. The worker function is modified to call the factorial_withgil or factorial_withoutgil functions from the C extension.

Performance Results

The C extension with the GIL enabled provides a significant speedup compared to the pure Python implementation. When the GIL is released, the performance improves dramatically, achieving near-linear scaling with the number of threads. For example, with 500 million calculations, the C extension with GIL enabled takes 1.39 seconds, while the version without the GIL takes only 0.219 seconds with 16 threads.

Notable Quotes

• "Part of the speed up will of course be just because C is faster in general... but another part of this speedup will also be the result of us manually releasing and reacquiring the so-called global interpreter lock or JIL."
• "This is crazy. You can see how much more stuff I do here. But just by using the C implementation and just by releasing the global interpreter lock when doing the repetitions, this happens much much faster..."

Technical Terms and Concepts

• C Extension: A module written in C that can be imported and used in Python.
• Global Interpreter Lock (GIL): A mutex that allows only one thread to hold control of the Python interpreter at any one time. This limits the true parallelism of Python threads in CPU-bound tasks.
• Python.h: The header file that provides access to the Python C API.
• piObject: A pointer to a Python object.
• piArg_ParseTuple: A function to parse arguments passed from Python to C.
• piLong_FromUnsignedLongLong: A function to convert an unsigned long long integer in C to a Python long object.
• piMethodDef: A structure that defines a method exposed to Python.
• piModuleDef: A structure that defines a Python module.
• piModInit_Func: A function that initializes a Python module.
• piBegin_AllowThreads(): A macro that releases the GIL.
• piEnd_AllowThreads(): A macro that reacquires the GIL.
• setup.py: A Python script used to build and install Python packages, including C extensions.
• pyproject.toml: A configuration file that specifies the build system requirements for a Python project.

Logical Connections

The video progresses logically from establishing a baseline performance with Python code, to implementing the same logic in C, wrapping the C code for use in Python, and then demonstrating the performance improvements gained by using C extensions and releasing the GIL. The example is kept simple to clearly illustrate the concepts and the impact of each optimization.

Conclusion

The video effectively demonstrates how C extensions can be used to significantly speed up Python applications, especially when combined with manual GIL management. While the example is basic, it highlights the potential for substantial performance gains in CPU-bound tasks by leveraging the speed of C and enabling true multi-threading. The video provides a clear and actionable introduction to writing and using C extensions in Python.