Python Multiprocessing Explained in 7 Minutes

Key Concepts:

• Multipprocessing vs. Multithreading
• Pool Class
• Process Class
• Queue (Q)
• Pipe
• Lock
• Value Class
• Semaphore

1. Multipprocessing vs. Multithreading:

• Python has both multithreading and multipprocessing.
• Multithreading is limited by the Global Interpreter Lock (GIL), preventing true parallelism for CPU-bound tasks.
• Multipprocessing uses multiple processes, enabling true parallelism.

2. Pool Class:

• The Pool class from the multipprocessing module allows parallel execution of a function across multiple input values.
• Example: Calculating factorials of a range of numbers.
  • Serial execution (single process) took 9.1 seconds.
  • Parallel execution (5 processes using Pool) took 2.9 seconds.
  • with Pool(5) as p: results = p.map(factorial_function, range(12000))
• The map function applies the given function to each item in the iterable in parallel.

3. Process Class:

• The Process class allows creating and managing individual processes.
• Example: A "watchdog" function that prints a heartbeat message every 5 seconds while a calculation runs in the main process.
• This demonstrates parallel execution of the watchdog and the factorial calculation.

4. Queue (Q):

• The Queue class facilitates communication and data sharing between processes.
• Example: A producer-consumer model.
  • The producer enqueues items (numbers) into the queue every 0.2 seconds.
  • The consumer dequeues and "consumes" items from the queue every 1 second.
  • Demonstrates how to process items in a specific order using multiple processes.
• Adding a second consumer shows that the queue maintains the order of items.

5. Pipe:

• A Pipe creates a communication channel between two processes (parent and child).
• Example: A worker function sends information to the parent process through the pipe.
• conn.send() sends data, and conn.recv() receives data.
• Setting duplex=True enables two-way communication.

6. Lock and Value Class:

• The Lock class synchronizes access to shared resources, preventing race conditions.
• The Value class creates a shared variable accessible by multiple processes.
• Example: Multiple processes incrementing a shared counter.
  • Without a lock, the final counter value is inconsistent due to concurrent access.
  • Using a lock ensures that only one process can modify the counter at a time, resulting in the correct final value (400,000).
  • with lock: counter.value += 1

7. Semaphore:

• The Semaphore class limits the number of processes that can access a resource concurrently.
• Example: Limiting the number of simultaneous workers to two.
• Workers acquire the semaphore before accessing the resource and release it afterward.
• The semaphore value decreases as workers acquire it and increases as they release it.
• with semaphore: # Access the resource

8. Synthesis/Conclusion:

• The video provides a rapid overview of key classes in Python's multipprocessing module.
• It demonstrates how to achieve true parallelism using Pool and Process.
• It explains how to manage communication and synchronization between processes using Queue, Pipe, Lock, Value, and Semaphore.
• The examples illustrate common use cases for each class, such as parallel computation, producer-consumer patterns, and resource management.