🔴Live: Coding Drone Obstacle using Pysimverse and AI

Key Concepts

• AI-powered Drone Programming: Using AI tools like Cursor to generate and modify drone code.
• Pymverse: A drone simulator for learning, competitions, and real-world scenario testing.
• Obstacle Avoidance: Programming a drone to autonomously navigate through hoops.
• Computer Vision: Using image recognition (YOLO) to detect hoops in the drone's camera feed.
• PID Control: Implementing a Proportional controller (P) for autonomous drone movement.
• Data Collection and Labeling: Gathering and annotating images for training object detection models.
• Kickstarter Campaign: Launching a crowdfunding campaign to fund Pymverse development.

Pymverse Simulator and its Purpose

The speaker introduces Pymverse, a drone simulator launching on August 5th (a Tuesday). It aims to bridge the gap between drone hardware and programmable control, offering a software solution for learning and experimenting with drone programming in real-world scenarios.

• Key Features:
  • Multiple scenarios for challenges, competitions, and learning.
  • Half-screen design for simultaneous coding and simulation viewing.
  • Connection to simulated drone for feed access.
  • Example codes (basics, image capture, YOLO) for different scenarios.
• Pricing Model:
  • Some scenarios will be free to download and run.
  • Competitions and live sessions will be part of a paid subscription.
  • Lifetime deal available for Kickstarter backers (one-time payment for all future content and live sessions).
• Motivation:
  • The TE drone is discontinued and hard to find.
  • Many companies offer drones as ready-made products, limiting programmability.
  • Pymverse provides a safe and cost-effective environment for learning and testing drone code.
  • Simulates real-world physics to replicate drone behavior accurately.

Obstacle Course Challenge: Autonomous Hoop Navigation

The core of the session involves programming a drone to autonomously navigate an obstacle course consisting of hoops.

• Goal: The drone should detect hoops, identify the largest one, and move towards it.
• Methodology:
  1. Data Collection:
     • Write code to move the drone using keyboard input (arrow keys).
     • Capture images of the drone's camera feed using the 'S' key.
     • Store the images in an output folder for labeling.
  2. Model Training:
     • Label the collected images by drawing bounding boxes around the hoops.
     • Train an object detection model (YOLO) using the labeled data.
     • The speaker refers to their free object detection course on YouTube for detailed instructions.
  3. Autonomous Navigation:
     • Use the trained YOLO model to detect hoops in real-time from the drone's feed.
     • Implement a Proportional (P) controller to adjust the drone's yaw (rotation) based on the hoop's position in the image.
     • Maintain a constant forward speed.
     • Count the number of hoops passed and land the drone after passing two hoops.

Step-by-Step Code Development

The speaker demonstrates the coding process using Cursor, an AI-powered code editor.

1. Collecting Data (collecting_data.py):
   • Import necessary libraries (e.g., pyimverse, cv2).
   • Connect to the drone and turn on the camera stream.
   • Implement keyboard controls for drone movement (W/A/S/D for forward/left/backward/right, T for takeoff, L for land).
   • Save images to an output folder when the 'Z' key is pressed.
2. Hoop Detection (hoop_detection.py):
   • Integrate the YOLO model (model.pt) with the image capture code.
   • Display the drone's camera feed with bounding boxes around detected hoops.
3. Autonomous Hoop Course (autonomous_hoop_course.py):
   • Implement a P controller to adjust the drone's yaw based on the hoop's center X coordinate.
   • Define a PIDController class with KP, KI, and KD values (initially using only KP).
   • Implement logic to count the number of hoops passed and land the drone after passing two.

Technical Details and Challenges

• RC Control: Using drone.rc_control to control the drone's speed and direction.
• Image Processing: Using OpenCV (cv2) to process the drone's camera feed and draw bounding boxes.
• PID Tuning: Adjusting the KP value to achieve stable and accurate hoop tracking.
• Challenges:
  • Initial issues with the AI-generated code (e.g., incorrect key mappings, takeoff command not working).
  • Drone moving in the wrong direction due to incorrect P controller implementation.
  • Inaccurate hoop detection leading to premature or missed hoop counts.
  • Drone continuing to move forward for too long after passing a hoop.
  • Balancing speed and accuracy to avoid collisions.

P Controller Implementation

The speaker implements a Proportional (P) controller to autonomously adjust the drone's yaw.

• Goal: To center the drone on the largest detected hoop.
• Method:
  • Calculate the error as the difference between the image center X coordinate (320) and the hoop's center X coordinate.
  • Multiply the error by the KP value to get the yaw output.
  • Clamp the yaw output to a range of -3 to 3 to limit the drone's rotation speed.
  • Apply the yaw output to the drone's rc_control.
• Tuning:
  • The speaker initially sets KI and KD to zero and focuses on tuning KP.
  • They start with a small KP value (0.01) and gradually increase it until the drone tracks the hoop effectively.
• Issue: The initial implementation caused the drone to move in the wrong direction, which was fixed by inverting the error calculation.

Key Arguments and Perspectives

• Importance of Simulation: The speaker emphasizes the value of using a simulator like Pymverse for learning and testing drone code before deploying it on real hardware.
• AI as a Tool: The speaker advocates for using AI tools like Cursor to accelerate the coding process, but also acknowledges the need for human oversight and debugging.
• Bridging the Gap: The speaker aims to bridge the gap between drone hardware and programmable control, making drone technology more accessible to students and researchers.
• Future of Drones: The speaker believes that drones will become increasingly prevalent in various industries and applications, creating a demand for skilled drone programmers and engineers.

Notable Quotes

• "AI is taking over and now we need to use it as a tool."
• "We are trying to create a software which will be easy for everyone to use and then you will be able to go through uh different scenarios and real world scenarios."
• "If you were to learn how to pilot, right? If you wanted to become a pilot, then you would not go directly fly a plane. You would first learn for a few hundred hours how exactly can you fly."

Synthesis/Conclusion

The YouTube session demonstrates the use of AI-powered coding tools and a drone simulator (Pymverse) to develop an autonomous hoop navigation system. The speaker walks through the process of data collection, model training, and P controller implementation, highlighting the challenges and solutions encountered along the way. The session emphasizes the importance of simulation for safe and cost-effective drone programming and promotes the upcoming Pymverse Kickstarter campaign. The key takeaway is that AI can be a valuable tool for drone programming, but human expertise is still essential for debugging and refining the code. The speaker successfully demonstrates a functional autonomous drone system within a short timeframe, showcasing the potential of Pymverse as a learning and development platform.