PlatoNeRF, created by MIT and Meta, employs multibounce lidar and machine learning to enable autonomous vehicles to detect hidden obstacles. This innovative technique, which also assists in AR/VR and robotics, uses shadows to generate precise 3D reconstructions of environments.
Researchers leverage shadows to model 3D scenes, including objects blocked from view.
This technique could lead to safer autonomous vehicles, more efficient AR/VR headsets, or faster warehouse robots.
Imagine driving through a tunnel in an autonomous vehicle, but unbeknownst to you, a crash has stopped traffic up ahead. Normally, you’d need to rely on the car in front of you to know you should start braking. But what if your vehicle could see around the car ahead and apply the brakes even sooner?
Researchers from 
Plato-NeRF is a computer vision system that combines lidar measurements with machine learning to reconstruct a 3D scene, including hidden objects, from only one camera view by exploiting shadows. Here, the system accurately models the rabbit in the chair, even though that rabbit is blocked from view. Credit: Courtesy of the researchers, edited by MIT News
They call their approach PlatoNeRF, based on Plato’s allegory of the cave, a passage from the Greek philosopher’s “Republic” in which prisoners chained in a cave discern the reality of the outside world based on shadows cast on the cave wall.
By combining lidar (light detection and ranging) technology with PlatoNeRF: 3D Reconstruction in Plato’s Cave via Single-View Two-Bounce Lidar” by Tzofi Klinghoffer, Xiaoyu Xiang, Siddharth Somasundaram, Yuchen Fan, Christian Richardt, Ramesh Raskar, Rakesh Ranjan, 2024, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Discussion about this post