The hidden information in your photos | Nikhil Behari | TEDxMIT

Key Concepts

• Shadows as Information: Shadows contain data about object positions, time of day, and even the presence of unseen objects.
• Reflections as Secondary Cameras: Reflections provide information about areas not directly visible to the camera.
• LiDAR (Light Detection and Ranging): A technology using laser pulses to measure distance and create 3D maps.
• Single Photon Detection: Analyzing individual photons of light to reveal hidden information about a scene, including multiple reflections and scattering.
• AI for Visual Perception: Developing AI systems to interpret shadows, reflections, and photon data for enhanced understanding of the environment.
• Depth Maps: Visual representations of distance information in a scene, used for autofocus and 3D modeling.

A Tour of Boston & The Hidden Potential of Phone Cameras

This presentation by Nikl Bahari, a graduate student at MIT and NASA fellow, explores the surprisingly rich information contained within everyday photos taken with smartphone cameras. Bahari argues that our phones are capable of “seeing” far more than we typically realize, and his research focuses on developing AI systems to unlock this hidden visual data. The presentation utilizes photos from Bahari’s personal camera roll taken over five years in Boston as illustrative examples.

I. Shadows: Revealing the Unseen

Bahari begins by highlighting the often-overlooked information contained within shadows. He demonstrates this using photos taken around MIT and Harvard campuses. He points out that shadows reveal details about the photographer (position, height), surrounding objects, and even the time of day.

He explains that while humans intuitively understand shadow information, AI systems struggle with this type of reasoning. His research at MIT aims to bridge this gap. Specifically, he describes work creating AI systems that can analyze shadows across an entire city (using satellite imagery) to generate city-scale 3D models.

Example: A photo of Bahari and a friend shows how the length of their shadows can be used to infer the time of day and approximate their height. The presence of shadows confirms their existence even when they aren’t directly visible in the frame.

Quote: “Shadows can reveal so much about the hidden world around us that our cameras would otherwise never be able to see.” – Nikl Bahari

II. Reflections: A Second Perspective

The presentation then shifts to reflections, demonstrating how they act as secondary cameras. Photos from Boston’s North End, the Public Garden, and the MIT Media Lab illustrate this point. Reflections capture information about areas outside the camera’s direct field of view, such as the underside of a duck’s beak or the sky behind the photographer.

Bahari explains the challenges AI faces in interpreting reflections: the warping of 3D space onto a 2D surface and the mixing of colors. He details his work in developing AI systems that can separate the reflected image from the object’s surface color, effectively isolating the reflection and using it as a source of visual data. This allows the AI to reconstruct the scene as seen by the reflection.

Example: A reflection in a building’s glass facade reveals the sky and surrounding structures that are not directly visible in the primary view. A reflection in water shows details of a duck’s beak that would be difficult to capture otherwise.

III. LiDAR & Single Photon Detection: The Invisible Light

The presentation delves into the technology hidden within smartphone cameras – LiDAR (Light Detection and Ranging). Bahari explains that the small black module on the camera bump houses a LiDAR sensor that emits hundreds of lasers with each photo.

He clarifies that the purpose of LiDAR isn’t simply to take pictures, but to measure distance. The phone times how long it takes for each laser pulse (and its constituent photons) to travel to an object and return. By counting individual photons and analyzing their return times, the phone creates a depth map of the scene.

Technical Detail: LiDAR utilizes the principle of time-of-flight measurement. The time delay between emission and detection of a photon is directly proportional to the distance to the object.

Data Point: Smartphones emit hundreds of laser pulses with each photo.

Bahari then introduces the concept of visualizing light “in flight” – tracking the path of individual photons. Using a lab recreation of LiDAR technology, he demonstrates how photons bounce, scatter, and reflect throughout a scene. This allows for the visualization of multiple reflections and photons that travel through translucent objects, revealing hidden details.

Technical Term: Photons – The fundamental particles of light.

IV. AI for Photon Analysis & Unveiling Hidden Objects

Bahari’s research focuses on developing AI systems capable of interpreting this single-photon data. He has designed systems that can analyze the data from LiDAR and reconstruct the 3D world around the phone with greater accuracy.

Crucially, by tracking photons that bounce multiple times, his AI can even “unveil” objects that are not directly visible from the camera’s perspective. This opens up possibilities for seeing around corners or through partially obscured objects.

Example: A demonstration shows a laser pulse hitting a wall and the subsequent propagation of photons throughout the scene, revealing hidden details and multiple reflections.

Quote: “By detecting those photons that bounce around multiple times in a scene, we can even uncover, unveil objects that our phone could have never seen from our point of view.” – Nikl Bahari

Conclusion

Bahari concludes by emphasizing the untapped potential of smartphone cameras. He encourages the audience to re-examine their own camera rolls, recognizing that even seemingly ordinary photos contain a wealth of hidden information. His research demonstrates that by leveraging AI to interpret shadows, reflections, and photon data, we can unlock a new level of visual understanding and “see the invisible.” The presentation highlights the power of combining everyday technology with advanced AI to reveal a more complete and nuanced picture of the world around us.