AI videos with audio, new Gemini 2.5, 3D faces, Sora is FREE, AI for video games, open-source TTS

Key Concepts

• Multimodal AI: AI models that can process and generate different types of data (text, images, 3D models).
• Video Generation: AI models that create videos from text prompts, images, or other inputs.
• 3D Generation: AI models that create 3D models from text prompts, images, or other inputs.
• Text-to-Speech (TTS): AI models that convert text into spoken audio.
• Open Source AI: AI models and code that are publicly available for use and modification.
• Context Window: The amount of information an AI model can process at once.
• Lip Sync: Synchronizing the movement of a character's lips with audio.
• VRAM: Video Random Access Memory, the memory on a graphics card.
• Diffusion Models: A type of generative model used for image and video creation.
• AI Agents: AI systems that can perform complex tasks autonomously.

3D Generation with Shape LLM Omni

• Main Topic: Shape LLM Omni, a multimodal model for 3D generation and interaction.
• Key Points:
  • Can generate 3D models from text prompts or images.
  • Allows editing of existing 3D models through chatbot-like prompting.
  • Can answer questions about 3D models.
  • Models and code are available on Hugging Face and GitHub.
• Examples:
  • Generating a 3D model of a handgun from a mesh and answering questions about it.
  • Creating a 3D model of a drone from a text prompt.
  • Editing a 3D model to add storage bags or change a spout into a chainsaw.
• Technical Terms:
  • Multimodal Model: An AI model that can process and generate different types of data (text, images, 3D models).
  • Hugging Face: A platform for sharing and discovering AI models and datasets.
  • GitHub: A platform for software development and version control.
  • VRAM: Video Random Access Memory, the memory on a graphics card.
• Logical Connections: Introduces Shape LLM Omni as a versatile tool for 3D creation and manipulation, highlighting its multimodal capabilities and accessibility.

Video Enhancement with Flow Mo

• Main Topic: Flow Mo, a plug-in to improve the quality and coherence of video generations.
• Key Points:
  • Smooths and enhances video generated by other AI models.
  • Reduces motion variance between frames for more realistic results.
  • Model agnostic, works with various video generators like one by Alibaba and Cog Videoo X.
  • Code available on GitHub.
• Examples:
  • Improving a video of a boy flying a kite, removing an extra kite that appears.
  • Making the motion of a person more coherent, preventing limb warping.
  • Enhancing videos of a dolphin jumping and a child jumping on a trampoline.
• Step-by-Step Processes:
  • Analyzes the video as it's being generated.
  • Measures the patch-wise variance of the video.
  • Adjusts the motions to reduce variance and create smoother motion.
• Technical Terms:
  • Model Agnostic: Can be used with different AI models.
  • Patch-wise Variance: A measure of how much the motion changes from one frame to the next.
• Logical Connections: Presents Flow Mo as a valuable addition to existing video generation workflows, improving the visual quality and realism of the output.

Native Resolution Image Synthesis with NIT

• Main Topic: Native Resolution Image Synthesis (NIT), an AI that generates images in any size and aspect ratio.
• Key Points:
  • Creates images without being specifically trained on those sizes.
  • Maintains image quality regardless of aspect ratio or size.
  • Data set and models are available on Hugging Face, with code on GitHub.
• Examples:
  • Generating images of a sea turtle, parrot, and arctic fox in various aspect ratios and sizes.
  • Creating super wide and super tall images that other models struggle with.
• Technical Terms:
  • Aspect Ratio: The ratio of the width to the height of an image.
  • Diffusion Transformer: A type of neural network architecture used for image generation.
• Logical Connections: Introduces NIT as a solution to the limitations of other image generators, which often struggle with non-standard image sizes and aspect ratios.

Car Crash Simulation with Control Crash

• Main Topic: Control Crash, an AI that generates realistic videos of car crashes from a single image.
• Key Points:
  • Simulates different types of crashes and predicts how they might happen.
  • Generates multiple scenarios from a single initial frame.
  • Can extrapolate and predict crash outcomes from initial frames and bounding box data.
  • Generates counterfactual scenarios to explore hypothetical crash outcomes.
  • Code available on GitHub.
• Examples:
  • Generating scenarios of no crash, ego-only crash, ego and vehicle crash, and vehicle and vehicle crash from the same initial image.
  • Predicting how a car crash would occur based on initial frames and bounding box data.
• Real-World Applications:
  • Safety analysis and planning.
  • Training algorithms for self-driving cars.
• Technical Terms:
  • Bounding Box: A box that delineates moving objects in a video.
  • Counterfactual Scenarios: Hypothetical scenarios that could have occurred under different conditions.
• Logical Connections: Highlights the practical applications of Control Crash in improving safety and training AI systems for autonomous vehicles.

Video Game Scene Generation with Deep First

• Main Topic: Deep First, an AI that generates gameplay scenes of any video game from a single image.
• Key Points:
  • Generates realistic and accurate gameplay scenes.
  • Understands physics, lighting, and character movements.
  • Can be controlled using key presses or a game controller.
  • Can generate any game, unlike other AI gameplay generators that are fixed to specific games.
• Examples:
  • Generating a scene of a character running into a wall and reacting realistically.
  • Creating scenes of a car driving on an empty road and a character walking along a dirt path.
  • Generating a scene of a character running with a flashlight, accurately simulating the lighting.
• Logical Connections: Positions Deep First as a versatile tool for generating video game content, offering more flexibility than existing game-specific AI generators.

Free Video Generation with Microsoft Bing

• Main Topic: Microsoft offering free and unlimited video generations powered by OpenAI's Sora through the Bing mobile app.
• Key Points:
  • Generates 5-second videos in a vertical 9:6 format.
  • Users get 10 fast generations, followed by unlimited generations at standard speed.
  • Quality is good but has been surpassed by newer models.
• Real-World Applications:
  • Creating social media clips.
• Logical Connections: Presents a readily available option for free video generation, albeit with some limitations in terms of quality and format.

Figure 2 Robot Package Sorting Demo

• Main Topic: A new demo of the Figure 2 robot autonomously sorting and scanning packages.
• Key Points:
  • Improved speed and dexterity compared to previous demos.
  • Can sort and scan packages of different shapes and sizes.
  • Flattens packages for more efficient scanning.
• Logical Connections: Showcases advancements in humanoid robot capabilities, particularly in tasks requiring dexterity and adaptability.

Abacus AI's Chat LLM and Deep Agent

• Main Topic: Abacus AI's Chat LLM and Deep Agent, AI tools for various tasks.
• Key Points:
  • Chat LLM: An all-in-one platform for using the best AI models, including image and video generators.
  • Deep Agent: An AI agent that can perform complex tasks autonomously, such as creating PowerPoint presentations, browsing the web, and making reservations.
• Examples:
  • Deep Agent creating a PowerPoint presentation with content, images, and charts.
  • Deep Agent browsing the web to find cheap flights.
  • Deep Agent making a dinner reservation.
• Real-World Applications:
  • Automating workflows.
  • Creating interactive dashboards.
• Logical Connections: Introduces a suite of AI tools designed to enhance productivity and automate complex tasks.

Open-Source Video with Audio Alternatives to VO3

• Main Topic: Open-source alternatives to Google's VO3 for generating video with audio.
• Key Points:
  • Skyre's Audio: Generates videos with people talking based on input audio, lip-syncing to the character.
  • Hunyen Custom: Can input an audio clip and lip-sync it to any character in the video, edit or replace anything in a video.
• Examples:
  • Skyre's Audio: Lip-syncing audio to a character, animating the character's body and background.
  • Hunyen Custom: Replacing a teddy bear with a husky in a video, replacing a clown fish with a white fish.
• Technical Terms:
  • Lip-Sync: Synchronizing the movement of a character's lips with audio.
• Logical Connections: Presents open-source options for generating video with audio, offering alternatives to Google's VO3 with varying levels of control and accessibility.

Realistic 3D Face Modeling with Pixel 3DMM

• Main Topic: Pixel 3DMM, an AI that creates accurate 3D models of a person's face from a single image.
• Key Points:
  • Generates highly accurate 3D face models.
  • Outperforms other 3D face generators in terms of accuracy.
  • Can generate a neutral expression from a face with different expressions and angles.
  • Code available on GitHub.
• Technical Terms:
  • Normal Estimation: The orientation of the surface of a 3D model.
• Logical Connections: Highlights the advancements in 3D face modeling, with Pixel 3DMM offering improved accuracy compared to existing methods.

Google Gemini 2.5 Pro Update

• Main Topic: Google's release of Gemini 2.5 Pro Preview 0605, an upgrade to their Gemini model.
• Key Points:
  • Improved coding abilities and performance across various benchmarks.
  • Ranked number one on the LM Arena leaderboard.
  • Large context window of 1 million tokens.
  • Performs well in understanding the context of long prompts.
  • Available for free in Google's AI Studio platform and the Gemini app.
• Data, Research Findings, or Statistics:
  • Ranked number one on LM Arena with an ELO score of 1470.
  • Correct over 90% of the time on the fiction livebench with 192,000-word prompts.
• Logical Connections: Showcases Google's continued advancements in AI model performance, with Gemini 2.5 Pro setting a new standard for capabilities and accessibility.

Text-to-Speech with Emotion Control: 11Labs and Fish Audio

• Main Topic: Advanced text-to-speech models that allow control over emotion and tone.
• Key Points:
  • 11Labs 11v3: A super realistic TTS model that allows adding tags within the transcript to control emotion and tone.
  • Fish Audio Open Audio S1: An open-source model that also allows adding tags to control emotion and tone.
  • Fish Audio S1 mini: A smaller, distilled version of the S1 model that is open-source.
• Examples:
  • Using 11Labs 11v3 to create a conversation with different emotions and tones.
  • Using Fish Audio Open Audio S1 to add emotional markers like "angry" or "sad" to a transcript.
• Technical Terms:
  • Text-to-Speech (TTS): AI models that convert text into spoken audio.
• Logical Connections: Presents two options for advanced text-to-speech, one proprietary (11Labs) and one open-source (Fish Audio), both offering control over emotion and tone.

Synthesis/Conclusion

This week in AI has seen significant advancements across various domains, including 3D generation, video enhancement, car crash simulation, video game scene generation, 3D face modeling, and text-to-speech. Notable releases include Shape LLM Omni for 3D creation, Flow Mo for video enhancement, Control Crash for car crash simulation, Deep First for video game scene generation, Pixel 3DMM for 3D face modeling, and Google's Gemini 2.5 Pro for general AI capabilities. Additionally, Microsoft is offering free video generation through the Bing app, and open-source alternatives to VO3 and 11Labs' TTS are emerging. These developments highlight the rapid pace of innovation in AI and the increasing accessibility of advanced AI tools.