By pairing a retina-like sensor with a spiking neural network, a new technique with potential applications for self-driving cars, search-and-rescue drones or medical imaging can reveal objects invisible to standard cameras.
A new imaging system is able to distinguish shapes through fog and other scattering media. Characters that were obscured by fog were imaged by a dynamic vision sensor (left) and reconstructed clearly (left) by a spiking neural network. Image credit: Brown University
Researchers from the Brown University School of Engineering have developed a new imaging system that can see and track moving objects in partially opaque environments like dense fog or muddy water.
The technique, described in a study published in Advanced Science, combines a dynamic vision sensor — a camera with pixels that report only changes in brightness, rather than capturing full frames — with a spiking neural network computer model that reconstructs the moving objects spied by the sensor. In laboratory tests, the system’s reconstructed images matched the hidden targets with structural similarity scores of up to 96% in turbid water and drifting fog that would hide the objects from a traditional camera, while they simultaneously tracked the targets’ positions as they moved.
The brain-inspired system is the first end-to-end technique that can both track and image randomly moving objects hidden by such dense scattering media, the researchers say.
“The reason it’s hard to see objects in fog is that the light bouncing off of objects gets scattered by the fog before it reaches our eyes or the camera lens,” said Ning Zhang, a postdoctoral researcher in engineering at Brown. “This research project is about using novel imaging cameras and a brain-inspired model that can filter out that scattering process and reconstruct what’s behind the fog.”
The system could eventually have a variety of real-world applications, including for self-driving cars and other autonomous vehicles, drone-based search and rescue efforts during storms or wildfires, or even medical imaging, the researchers say.
Inspired by the brain
To develop the technique, the researchers took cues from the human visual system, which can process images with remarkable speed and efficiency. Rather than collecting a full picture all at once like a video camera, the retina responds mainly to changes in the visual field — edges, motion and flicker —and feeds that information to the brain, which builds a reconstruction of the scene over time. When it senses a change, it sends neural spikes to the brain, which are used to update the brain’s reconstruction of the world.
The researchers were able to harness a similar architecture to create a system that processes images quickly and efficiently, but that outperforms human vision in foggy or turbid conditions. The system starts with a dynamic vision sensor that can detect light-intensity changes in individual pixels within the visual field. Because each pixel fires only when the brightness it sees changes by more than a set threshold, the sensor registers a moving object even when much of the light has been scattered by fog or some other scattering medium. And because the scattering background changes far more slowly than the moving target, the sensor largely ignores the fog and responds mainly to light that has interacted with the object. What it records is still a shapeless cloud of spikes, which is where the spiking neural network comes in.
“The human eye is quite good at motion detection, but there are ways in which a camera, specifically a dynamic vision sensor, can be better, including that it’s sensitive to near-infrared light, which the human eye cannot see,” said Arto Nurmikko, a professor of engineering at Brown and the study’s senior author. “It captures information only about a moving object, and in doing so effectively filters out some of the fogginess while generating an output where relevant information is coded as trains of asynchronous spikes, much as the biological retina acquires data in a form the brain’s visual cortex understands.”
The key innovation in the work is a neuromorphic computer model mimicking the visual cortex: a deep spiking neural network that takes data from the vision sensor and reconstructs images of randomly moving, normally unrecognizable objects while tracking their trajectories in fractions of a second. The network runs on spikes as the fundamental unit of information.
To test the system, the researchers projected images of moving alpha-numeric characters and silhouettes of different species of flying birds through an obscuring fog chamber or a tank of turbid water. The experiments showed that the system was able to image and track the randomly moving objects unrecognized by a standard camera system or the human eye.
The system is also highly energy efficient. The sensor itself draws only tens of milliwatts. The spiking network uses a fraction of the energy used by a conventional neural network because it computes only on sparse spikes rather than dense frames.
The approach has limits, the researchers note. Because the sensor responds only to change, an object has to be moving relative to the camera to be seen, and the current system recovers an object’s silhouette rather than a full grayscale image. The sensor also loses sensitivity in very dim light. The team is now exploring a light-intensifier front end for low-light conditions, as well as depth-resolved approaches that can extend the method to three-dimensional targets — for example, by timing light’s travel time, or by pairing two event cameras like a pair of eyes.
“This could be useful anywhere where light-scattering by the surrounding medium is a serious problem — self-driving cars, search-and-rescue drones and underwater navigation are a few examples,” Zhang said. “The goal is to push the boundary of what we can perceive, to improve safety, healthcare and beyond.”
The research was supported by the Office of Naval Research (N00014-25-1-2061), Intel Laboratories (CG70982727 2021) and the National Science Foundation (232600).
Source: Brown University
