Maryland Today | Researchers' new camera takes inspiration from human vision

A team led by computer scientists at the University of Maryland has invented a camera mechanism that improves the way robots see and react to the world around them. Inspired by the workings of the human eye, its innovative camera system mimics the tiny involuntary movements the eye uses to maintain clear, steady vision.

The team’s prototyping and testing of the camera, called the Artificial Microsaccade-Enhanced Event Camera (AMI-EV), was detailed in a paper recently published in Scientific robotics.

Event cameras are a relatively new technology that are more effective than traditional cameras at tracking moving objects, but today’s event cameras struggle to capture sharp, blur-free images when there’s a lot of motion, said the paper’s lead author, Botao He, a doctoral student in computer science at UMD.

“This is a major problem because robots and many other technologies, such as self-driving cars, need accurate and up-to-date images to respond properly to a changing environment,” he said. “So we asked ourselves: How do humans and animals ensure that their vision stays focused on a moving object?”

For He’s team, the answer was microsaccades, those tiny, rapid eye movements that occur without conscious thought to help the human eye focus on objects and visual textures, such as color, depth and shadow.

“We thought that just as our eyes need these small movements to stay focused, a camera could use a similar principle to capture clear, sharp images without motion blur,” he said.

Camera system diagram

Schematic of the new camera system (AMI-EV). (Image courtesy of UMIACS Computer Vision Lab)

The team was able to reproduce microsaccades by inserting a rotating prism inside the AMI-EV to redirect the light beams captured by the lens. The continuous rotating motion of the prism simulated the movements that naturally occur in a human eye, allowing the camera to stabilize the textures of a recorded object as a human would. The team then developed software to compensate for the prism movement inside the AMI-EV to consolidate stable images from changing lights.

Yiannis Aloimonos, a co-author of the study and a professor of computer science at UMD, called the team’s invention a major step forward in the field of robotic vision.

“Our eyes take pictures of the world around us, and those pictures are sent to our brains, where they are analyzed. Perception happens through this process, and that’s how we understand the world,” says Aloimonos, who is also director of the Computer Vision Lab at the University of Maryland Institute for Advanced Study in Computer Science (UMIACS). “When you’re working with robots, replace the eyes with a camera and the brain with a computer. Better cameras mean better perception and better reactions for the robots.”

The researchers also believe their innovation could have significant implications beyond robotics. Scientists working in industries that rely on precise image capture and pattern detection are constantly looking for ways to improve their cameras. AMI-EV could be the key solution to many of the problems they face.

Thanks to their unique features, event sensors and AMI-EV are poised to take center stage in the field of connected objects, said researcher Cornelia Fermüller, lead author of the paper.

“They have clear advantages over traditional cameras, such as superior performance in extreme lighting conditions, low latency and low power consumption,” she said. “These features are ideal for virtual reality applications, for example, where a smooth experience and fast calculations of head and body movements are required.”

In early tests, AMI-EV was able to accurately capture and display motion in a variety of settings, including detecting human pulses and identifying fast-moving shapes. The researchers also found that AMI-EV could capture motion at tens of thousands of frames per second, outperforming most commercial cameras on the market, which average 30 to 1,000 frames per second. This smoother, more realistic representation of motion could prove critical for creating more immersive augmented reality experiences and better security monitoring, as well as improving how astronomers capture images in space.

“Our new camera system can solve a lot of specific problems, like helping a self-driving car determine what’s human and what’s not on the road,” Aloimonos said. “So it has a lot of applications that a lot of the general public already interacts with, like autonomous driving systems or even smartphone cameras. We think our new camera system paves the way for more advanced and capable systems to come.”