Bio-inspired cameras and AI help drivers detect pedestrians and obstacles faster

Artificial intelligence (AI) combined with a new bio-inspired camera enables 100 times faster detection of pedestrians and obstacles than current automotive cameras. This milestone in computer vision and AI by researchers at the University of Zurich can greatly improve the safety of automotive systems and self-driving cars.

It’s every driver’s nightmare: a pedestrian coming out of nowhere in front of the car, leaving only a fraction of a second to brake or steer and avoid the worst. Some cars now have camera systems capable of alerting the driver or activating emergency braking. But these systems are not yet fast or reliable enough, and they will need to be significantly improved if they are to be used in autonomous vehicles where no humans are behind the wheel.

Faster detection using less computing power

Now, Daniel Gehrig and Davide Scaramuzza from the Department of Computer Science at the University of Zurich (UZH) have combined a new bio-inspired camera with AI to develop a system capable of detecting obstacles around a car much faster than current systems and using less calculations. power. The study is published in Nature.

Most current cameras are image-based, meaning they take snapshots at regular intervals. Those currently used for driver assistance on cars typically capture 30 to 50 frames per second and an artificial neural network can be trained to recognize objects in their images: pedestrians, bicycles and other cars.

“But if something happens in the 20 or 30 milliseconds between two snapshots, the camera may see it too late. The solution would be to increase the frame rate, but that results in more data to process in time. real and by more calculations power,” says Gehrig, first author of the article.

Combining the best of two camera types with AI

Event cameras are a recent innovation based on a different principle. Instead of a constant frame rate, they have smart pixels that record information whenever they detect rapid movement.

“This way, they have no blind spots between images, which allows them to detect obstacles more quickly. They are also called neuromorphic cameras because they mimic the way human eyes perceive images,” explains Scaramuzza, head of the Robotics and perception group. But they have their own flaws: They can miss slow-moving things, and their images aren’t easily converted into data used to train the AI ​​algorithm.

Gehrig and Scaramuzza have developed a hybrid system that combines the best of both worlds: It includes a standard camera that collects 20 frames per second, a relatively low frame rate compared to those currently in use. Its images are processed by an AI system, called a convolutional neural network, trained to recognize cars or pedestrians.

Data from the event camera is coupled with another type of AI system, called an asynchronous graphical neural network, which is particularly suited to analyzing 3D data that changes over time. The detections of the event camera make it possible to anticipate the detections of the standard camera and also to increase its performance.

“The result is a visual detector that can detect objects as quickly as a standard camera shooting 5,000 frames per second, but requiring the same bandwidth as a standard 50 frames per second camera,” says Gehrig.

Hundred times faster detections using less data

The team tested their system against the best cameras and visual algorithms currently available in the automotive market, finding that it enabled detections one hundred times faster while reducing the amount of data that must be transmitted between the camera and the on-board computer as well as the calculation time. power needed to process images without affecting accuracy.

Above all, the system can effectively detect cars and pedestrians entering the field of view between two successive images from the standard camera, providing additional safety to the driver and road users, which can make a huge difference, especially at high speed.

According to the scientists, the method could be made even more powerful in the future by integrating cameras with LiDAR sensors, like those used on self-driving cars.

“Hybrid systems like this could be crucial to enabling autonomous driving, ensuring safety without leading to substantial growth in data and computing power,” says Scaramuzza.