How to See Around Corners with a Digital Camera

(Inside Science) – Shadows are everywhere. Some may look like formless blobs, but researchers are now showing that even indistinct shapes could reveal what’s hidden around the corner, no fancy equipment required.

By using an ordinary digital camera and a computer to analyze a shadow that’s reflected off a normal white wall, the researchers can reconstruct a scene that’s hidden from a camera’s line of sight. The new experiment, published today in Nature , is the latest in a growing effort to develop ways to see around corners -- technology that could someday be useful for everything from self-driving cars and medical imaging to monitoring hazardous environments and performing military reconnaissance.

This ability to see around corners isn’t new. But while other methods require expensive lasers and light-detection systems, the new technique uses equipment that ordinary consumers could buy.

“You don’t need thousands of dollars of equipment to achieve non-line-of-sight imaging,” said Vivek Goyal, an electrical engineer at Boston University and one of the three authors of the new paper.

The way researchers are trying to see around corners is similar to how submarine crews use periscopes to see above water. The difference is that instead of the periscope’s mirrors, which reflect a sharp image, there’s a wall that scatters light, which usually produces a diffuse glow. Ideally, you would take that scattered light, figure out where each beam came from, and reconstruct the reflected image.

But like Humpty Dumpty, it’s nearly impossible to put such a scattered image back together again.

One trick, it turns out, is to examine the shadows the hidden object might cast. “We see shadows around us all the time,” Goyal said. “But we don’t think about how we can extract information from those shadows.”

Goyal, and engineers Charles Saunders and John Murray-Bruce, also at Boston University, used an LCD screen to display several illuminated cartoon images. They then placed a small rectangular sheet in front of the screen, casting a shadow onto a matte white wall. Off to the side was a digital camera, which couldn’t see the LCD screen but could take pictures of the shadow on the wall.

“The coolest thing is a wall really can be used as a mirror,” Goyal said.

What’s crucial is that the shadow isn’t a complete shadow. It’s a partial shadow, called a penumbra, in which some of the light from the screen reaches the wall while the rest is blocked by the rectangle.

Every spot on the penumbra contains light that originated from a different part of the image. For example, one of the images in the experiment was of a cartoon face wearing a baseball cap. On one part of the penumbra, the light may have originated from the tip of the cap’s brim, while on another spot the light may have come from the face’s ear.

By using a computer algorithm to analyze an image of the penumbra, the researchers could examine every pixel to see which light was blocked, which made it through, and where that light came from. The algorithm combines all that information to reconstruct the cartoon image, turning a fuzzy dark patch into a colorful picture close to the original.

“The results look incredible,” said Gordon Wetzstein, an electrical engineer at Stanford University who was not involved in the study. “I haven’t really seen full color results at this quality.”

Wetzstein was part of a previous approach that produced a monochrome reconstructed image of a hidden object. As part of the experiment , researchers bounced laser beams off a wall and toward a three-dimensional rabbit figurine. The light scattered, and some of it reflected back off the wall and toward the laser. There, a detector captured the reflected light, which could be pieced together to form an image of the rabbit. Instead of trying to make sense of diffuse, scattered light, this method tracks the laser beams’ entire trajectories and travel times, producing a sharper image -- albeit in only a single hue.

While both techniques are still far from being applied commercially, the fact that the new experiment uses ordinary instruments pushes research closer in that direction, Wetzstein said. “That’s a big step toward making techniques like imaging hidden objects more practical with conventional hardware rather than relying on specialized equipment.”

Researchers still have much to work out before this technique can, for instance, help tomorrow’s driverless cars anticipate hazards lurking in the shadows. The method still requires a laboratory setting, and it only works if the contrast between the penumbra and the background is high enough. The researchers are now working on a hybrid approach that combines their shadow-based technique with other laser-based methods, Goyal said.

But the fact that any of these methods work at all and that we can see around corners is amazing, Wetzstein said. “I wouldn’t have thought it was even possible.”