Smart detection of road damage with AI

Innovations on Cologne's roads with AI

Potholes or blocked manhole covers can cause trouble on the roads. In Cologne, this could previously be reported via hotlines and online forms. Or staff walking the streets noted the road damage. There are many ways of doing this, but so far they have all been manual. This is set to change: An innovative pilot project by Cologne's waste management company in cooperation with msg detects and reports unnoticed road damage fully automatically with the help of artificial intelligence.

Initial situation

Road damage in the city of Cologne is recorded manually, which involves a great deal of work and time for the city. The work is also made more difficult in part by the fact that sections of the road can be obscured by parked vehicles, which means that the assessment of the roads has to be repeated.

The AWB therefore looked into the question of whether the process could be made more efficient and sustainable through the use of artificial intelligence. The idea was born to use some of AWB Cologne's city cleaning vehicles, which are deployed throughout the city, to record videos. The image material collected in this way was to be used to train an AI to automatically record road damage and its location. For the implementation of this pilot project, AWB worked together with msg.

Solution

Some of the city cleaning vehicles were equipped with smartphones in the driver's cab, which were attached behind the windshield. Using an Android app, the smartphones recorded and stored video footage of the road surface and the GPS coordinates associated with the images. All data was uploaded to the Azure Cloud via a secure Wi-Fi connection in the vehicle fleet. In the first step, the project team used the data collected from some road sections to train the machine learning model to detect road damage. In addition to road damage, the model was also trained to detect manhole covers. Compared to road damage, however, manhole covers have a known position and therefore serve as a benchmark for the quality of the algorithm. Due to their fixed structure, they are also easier to recognize, which is why a higher hit rate was expected. The custom vision web portal in the Azure Cloud was used to train the machine learning model.

Once the models had been trained, the msg team used the Computer Vision API of Azure Cognitive Services in the second step to identify road damage and manhole covers in the individual images of the videos using the trained model. As expected, the hit rate was higher for the manhole covers than for the road damage. The corresponding GPS coordinates of the individual hits were marked on a map in a portal. For each marker, the employees responsible for maintenance at the City of Cologne can take a closer look at the corresponding image with the road damage found. They can then check the report and initiate the next steps for repair if necessary.

Benefits

The results of the pilot have shown that automated detection of road damage with a high hit rate is possible through the use of artificial intelligence. As cleaning vehicles drive the same routes at periodic intervals, road damage that is temporarily concealed by parked vehicles, for example, can also be detected. The information obtained can be forwarded promptly to the responsible parties, who can thus ensure that roads are safe and maintained more quickly and efficiently.

Outlook

The use of “Smart Fleet” can also be extended to other fields of application in the future - for example, to detect bystanders at containers or waste bins or to analyze narrow lanes or defective traffic lights. This data can then also be reported to the responsible parties, such as the AWB or the City of Cologne and its departments, within a few minutes. This means that faults can be rectified more quickly and repairs or cleaning can be arranged promptly, making a valuable contribution to improving the quality of life in Cologne.