Python Multiprocessing Explained in 7 Minutes

Key Concepts:

1. Multipprocessing vs. Multithreading:

Python has both multithreading and multipprocessing.
Multithreading is limited by the Global Interpreter Lock (GIL), preventing true parallelism for CPU-bound tasks.
Multipprocessing uses multiple processes, enabling true parallelism.

2. Pool Class:

The Pool class from the multipprocessing module allows parallel execution of a function across multiple input values.
Example: Calculating factorials of a range of numbers.
- Serial execution (single process) took 9.1 seconds.
- Parallel execution (5 processes using Pool) took 2.9 seconds.
- with Pool(5) as p: results = p.map(factorial_function, range(12000))
The map function applies the given function to each item in the iterable in parallel.

3. Process Class:

The Process class allows creating and managing individual processes.
Example: A "watchdog" function that prints a heartbeat message every 5 seconds while a calculation runs in the main process.
This demonstrates parallel execution of the watchdog and the factorial calculation.

4. Queue (Q):

The Queue class facilitates communication and data sharing between processes.
Example: A producer-consumer model.
- The producer enqueues items (numbers) into the queue every 0.2 seconds.
- The consumer dequeues and "consumes" items from the queue every 1 second.
- Demonstrates how to process items in a specific order using multiple processes.
Adding a second consumer shows that the queue maintains the order of items.

5. Pipe:

A Pipe creates a communication channel between two processes (parent and child).
Example: A worker function sends information to the parent process through the pipe.
conn.send() sends data, and conn.recv() receives data.
Setting duplex=True enables two-way communication.

6. Lock and Value Class:

The Lock class synchronizes access to shared resources, preventing race conditions.
The Value class creates a shared variable accessible by multiple processes.
Example: Multiple processes incrementing a shared counter.
- Without a lock, the final counter value is inconsistent due to concurrent access.
- Using a lock ensures that only one process can modify the counter at a time, resulting in the correct final value (400,000).
- with lock: counter.value += 1

7. Semaphore:

The Semaphore class limits the number of processes that can access a resource concurrently.
Example: Limiting the number of simultaneous workers to two.
Workers acquire the semaphore before accessing the resource and release it afterward.
The semaphore value decreases as workers acquire it and increases as they release it.
with semaphore: # Access the resource

8. Synthesis/Conclusion:

The video provides a rapid overview of key classes in Python's multipprocessing module.
It demonstrates how to achieve true parallelism using Pool and Process.
It explains how to manage communication and synchronization between processes using Queue, Pipe, Lock, Value, and Semaphore.
The examples illustrate common use cases for each class, such as parallel computation, producer-consumer patterns, and resource management.