Realtime AI videos, Grok 4, open-source robots, add sound to video, 3D videos, 4K upscaler

Key Concepts

• AI Video Generation: Creating videos from text prompts or images.
• AI Audio Generation: Adding realistic and synchronized sound to silent videos.
• 4D Slow-Mo: Generating slow-motion 3D videos from multiple asynchronous video inputs.
• AI Photo Retouching Agent: Automating photo editing tasks in Adobe Lightroom using AI.
• Agentic AI: AI systems capable of autonomous tasks and workflows.
• 3D Object Generation: Creating 3D models from images with editable and controllable parts.
• Humanoid Robotics: Robots with human-like form and capabilities, including dual-mode locomotion.
• Open-Source Robotics: Programmable robots with open-source hardware and software.
• Large Language Models (LLMs): Advanced AI models for text and image processing, reasoning, and problem-solving.
• AI Deblurring and Upscaling: Enhancing the quality of blurry or low-resolution images.

Omni V: Customizable Video Generation

• Main Function: Omni V is an AI tool that customizes videos by changing subjects, scenes, or actions based on text prompts and reference images.
• Key Features:
  • Image-to-Video: Generates videos based on a reference image and a text prompt describing the desired action or scene.
    • Example: Inputting an image of a woman and the prompt "the woman in this image is talking to a man on the street" generates a video of that scenario.
  • Micro Editing: Allows for specific edits within a video, such as changing clothing or adding elements.
    • Example: Swapping the t-shirt of a boy in an image with a blue shirt and having him pray in a temple.
  • Scene Combination: Combines elements from two different images into a single video.
    • Example: Adding a man running (from one image) in front of a church (from another image).
  • Camera Control: Enables control over the camera movement in the generated video by specifying a 3D trajectory.
  • Reference Input: Accepts depth videos and mask videos as references to guide the video generation process.
• Current Status: The GitHub repository for Omni V has been released, but the models and code are not yet available.

Think Sound: AI-Powered Audio for Video

• Main Function: Think Sound is an AI that automatically adds synchronized sound to silent videos.
• Key Features:
  • Action-Specific Sound: Generates audio that is precisely synced to the actions in the video.
    • Example: Generating chopping sounds that match the timing of a person chopping wood.
  • Contextual Awareness: Detects the scene and adjusts the audio accordingly.
    • Example: Generating train sounds that change in volume and proximity as a train approaches.
• Comparison with MM Audio: Think Sound is generally higher quality and more in sync compared to MM Audio, a popular open-source alternative.
• Benchmark Performance: Think Sound outperforms other similar tools across various benchmark metrics for video-to-audio generation.
• Accessibility: A free Hugging Face space is available for online use, and a GitHub repository provides instructions for local installation.

4D Slowmo: High-Quality 3D Slow-Motion Video

• Main Function: 4D Slowmo creates high-quality, slow-motion 3D videos from fast-moving scenes.
• Key Features:
  • Asynchronous Video Input: Accepts multiple videos of a scene from different angles, even if they are taken at slightly different times.
  • High Frame Rate: Achieves frame rates of 100-200 frames per second, allowing for significant slow-down without specialized high-speed cameras.
  • 3D Perspective: Generates a 3D video that can be viewed from different angles.
• Quality Comparison: 4D Slowmo offers significantly better quality and smoother results compared to existing 3D video generators.
• Future Availability: The code and data will be released after the paper is accepted.

Jarvis Art: AI Agent for Adobe Lightroom

• Main Function: Jarvis Art is a free AI agent that controls Adobe Lightroom to automatically retouch photos.
• Key Features:
  • Autonomous Editing: Adjusts settings like brightness, contrast, saturation, and white balance based on text prompts.
  • Micro Editing: Allows users to draw a box around a specific area to edit only that region of the image.
  • AI Planning: Analyzes the image and plans out the steps needed to achieve the desired effect.
  • Preset Generation: Generates Lightroom preset templates that can be applied to images.
• Accessibility: A free Hugging Face space is available for generating preset configurations, and a GitHub repository provides inference code for local use.
• Dependency: Requires Adobe Lightroom to function fully.

Emergent: Agentic AI Coding Platform

• Main Function: Emergent is an agentic AI coding platform that autonomously helps users create applications from text prompts.
• Key Features:
  • Autonomous Development: Handles both front-end and back-end development.
  • Testing and Refinement: Tests the application to ensure it runs smoothly.
  • Versatile Application: Can be used to build various types of apps, including habit trackers, meditation apps, and interactive piano apps.

Comet: AI-Powered Browser by Perplexity

• Main Function: Comet is an AI browser developed by Perplexity that features a built-in AI assistant for agentic tasks.
• Key Features:
  • Browser Integration: The AI assistant lives within the browser and can be used on any webpage.
  • Agentic Capabilities: Automates tasks such as scheduling meetings, sending emails, and summarizing content.
• Availability: Currently available to Max subscribers, with invite-only access rolling out to the waitlist over the summer.

Omniart: Segmented 3D Object Generation

• Main Function: Omniart creates 3D objects from images with separate, editable, and controllable parts.
• Key Features:
  • Part Segmentation: Segments the 3D model into meaningful parts that can be edited individually.
  • Mask-Guided Generation: Allows users to control the segmentation process using mask images.
  • Granularity Control: Provides ultimate control over the granularity of the segmentation.
• Accessibility: A free Hugging Face space is available for online use, allowing users to segment images and generate 3D models.

Humanoid Robot News: AGIBOT X2N and Hugging Face's Reichi Mini

• AGIBOT X2N: A humanoid robot developed by Zu Yuen (founded by a former Huawei engineer) that can seamlessly switch between walking on legs and rolling on integrated foot wheels. It can carry loads of up to 12 lbs.
• Reichi Mini: An open-source desktop robot from Hugging Face, available in wireless ($450) and wired ($300) versions. It is programmable using Python and can be linked to open-source AI tools for tasks like face recognition and chatbot interactions.

Gro 4: State-of-the-Art AI Model by XAI

• Main Function: Gro 4 is a multimodal AI model (text and images) that excels at advanced reasoning, problem-solving, and knowledge retrieval.
• Key Features:
  • Multimodal: Handles both text and images. Video generation is expected in October.
  • Large Context Window: Supports a context window of 256K tokens.
  • Advanced Reasoning: Excels at solving multi-step logic problems, mathematical proofs, and other problems in science, coding, and STEM.
  • Gro 4 Heavy: A high-performance version that uses a team of AI agents to solve complex tasks.
• Benchmark Performance:
  • Outperforms Claude for Opus and Gemini 2.5 Pro on GPQA (graduate-level research questions).
  • Achieved a perfect score on AIM (competitive math benchmark).
  • Significantly outperforms other models on Humanity's Last Exam (obscure knowledge test).
  • Scores highest on the ARC AGI benchmark, which evaluates the ability to solve new problems with minimal prior information.
• Accessibility: Closed source and requires a paid subscription ($30/month for the basic version, $300/month for Gro 4 Heavy).

StreamDI: Real-Time Video Generation

• Main Function: StreamDI generates videos in real time from text prompts.
• Key Features:
  • Real-Time Generation: Generates videos at 16 frames per second on a single H100 GPU.
  • Prompt Injection: Allows users to inject another prompt during video generation to modify the scene.
  • Variable Video Length: Can generate videos ranging from one minute to five minutes long.
  • Moving Buffer with Flow Matching: Uses a moving buffer and flow matching to ensure smooth transitions between frames.
• Model Variants: A 4 billion parameter version (real-time) and a 30 billion parameter version (higher quality, not real-time).
• Availability: Developed by Meta, but no code or models have been released.

4K Agent: AI Deblurring and Upscaling

• Main Function: 4K Agent deblurs and upscales images, significantly enhancing their quality and detail.
• Key Features:
  • Deblurring: Removes blur from images, including those with motion blur.
  • Upscaling: Increases the resolution of images while preserving detail.
  • Versatile Application: Can be used on various types of images, including satellite imagery, microscopic images, and X-rays.
• Availability: A GitHub repository has been released, but no code or models are currently available.

Conclusion

This week in AI has seen significant advancements across various domains, including video and audio generation, 3D modeling, robotics, and large language models. Tools like Omni V, Think Sound, 4D Slowmo, Jarvis Art, Omniart, StreamDI, and 4K Agent showcase the increasing capabilities of AI in creative and practical applications. The release of Gro 4 by XAI marks a new milestone in AI performance, particularly in reasoning and problem-solving. The open-source robotics initiatives from Hugging Face and the agentic AI coding platform Emergent highlight the growing accessibility and potential of AI to empower individuals and automate complex tasks. While some tools are still under development or have limited availability, the overall trend indicates a rapid pace of innovation and a promising future for AI technology.