The CSIR has developed a visual surveying platform (VSP) to assist road engineers and municipalities with the maintenance of road infrastructure by collecting and analysing road data in order to detect and highlight defects. An efficient road network is vital to any economy. In order to ensure the high availability of the road network, it is important to regularly maintain the road infrastructure. 

The VSP consists of a vehicle-mounted sensor system, a desktop client application, and a server backend. The sensor system consists of a high-resolution camera, a global positioning system (GPS) receiver and a computer with peripherals for wireless connectivity. It is used to collect synchronised GPS and video data from the environment. The sensor is mounted onto a vehicle and collects data while it is driven through the area that is to be inspected. 

The synchronised data is uploaded to the client application where the video is analysed. The output of the analysis is a list of detections, for example a list of defects such as potholes or faded road markings. Each detection is stored in a local database along with its properties such as type of defect, estimated size and the location. The local desktop client application allows the user to filter and visualise the detections found in videos that the user has collected. At the local level, the analysed data is used by a client to support operational decision-making, for example, maintenance schedules.

The user of a local client system can upload the collected detections to a server hosted at the CSIR. The server uses a technology that was developed at the CSIR, called CMORE. This web-based backend system can accumulate data from multiple client applications in different regions. The accumulated data can then be analysed to identify trends over time and across larger regions. At this level the data analysis can support decision-making at municipal level. The backend system supports a feature that would allow community members to submit reports of road defects directly into the backend system via a mobile application.

Current imported state-of-the-art systems that are used for road measurements are very accurate but can cost more than R15-million each, which limits the number of deployments of such systems. This limitation translates into a lower update frequency and lower coverage. The VSP can be produced at an estimated R500 000. The lower cost allows more units to be deployed which would result in more area coverage and a higher frequency of updates. The VSP provides decision-makers with timeous first-level estimates of the state of the road infrastructure to enable effective planning of further maintenance operations. In this way, the VSP is not seen as a replacement of existing systems, but it is used to better plan the deployment of limited and expensive resources.

The CSIR issued technology evaluation licences to two local repair companies to trial the platform and to investigate the integration of the system into their current workflows.

The analysis module is a key component of the system and it uses a combination of image processing and machine learning algorithms to detect, classify and quantify road defects.

Three modes of operation were investigated during the development of the platform and these are a manual mode, an autonomous mode and a hybrid mode. In the manual mode an operator can watch the video stream and manually detect and tag defects. The size estimation and the geo-tagging of the defect are performed automatically and the detection is added to the database.

In the autonomous mode, the system automatically analyses the video stream using the machine- learning algorithms to detect specific defects before calculating the size and location of the defect.

During the pre-processing stages of the automated algorithm, the frames are extracted from the video stream and the full colour red, green and blue input frames are then converted into the hue, saturation and value (HSV) colour space and the independent channels are extracted. The saturation channel is used by the road detection and feature extraction stages.

During the detection stage, a Gaussian kernel filter is used to create a mask for the road which is then used to extract the road texture from the frame. The detection stage serves to reduce the search area for identifying defects and increases the accuracy by introducing prior information.

A Local Binary Pattern (LBP) feature extractor was applied to the saturation channel of the HSV image of the road. The LBP feature extractor makes use of textural relationships within the image. These features were chosen to make use of the fact that potholes are irregularly defined objects of interest that occur in a uniform background.

The LBP features are then used to train a cascade of classifiers. The method involves training a group of statistical models such that they are each able to weakly classify the data. The group of classifiers is then combined using the method of boosting to produce a single stronger classifier.

During testing, the video data was first processed manually and then automatically by the algorithm. The manual and automatic detections were then compared. The results indicate that the automated detection alone had an accuracy of approximately 89%. The hybrid mode validation would increase the overall accuracy of the system further.

In the hybrid mode the system automatically analyses the video stream and suggests possible defects to an operator who then validates the detection.

Based on feedback from potential clients, the development team decided that the manual mode and hybrid mode would be suitable for early integration into the existing workflows of companies and municipalities that are involved in the maintenance of road infrastructure. The collection of manually detected defects as well as those detections that are positively validated in the hybrid mode would be used as a training dataset to improve the autonomous detector which would be used in future versions of the hybrid mode detector thus allowing it to improve the quality of the suggestions made.

Although defects of different types can be extracted from the video stream, the current version of the visual surveying platform focuses on detecting and estimating the size of potholes. During one investigation the team developed an algorithm to detect road signs. The team has received requests from a road defect repair company for further developing the road sign detector, as they have a need to geo-locate road signs to determine if they are placed correctly or if they are missing.

Future development plans involve continuing to refine the system by incorporating feedback received from companies that the team is currently collaborating with and to meet with potential clients and investors. The team will also investigate the extraction of additional useful information from the video stream that could support decision-making.