Pothole Detection Using Android Smartphone with a Video Camera 1 Youngtae Jo *, 2 Seungki Ryu 1 Korea Institute of Civil Engineering and Building Technology, Korea E-mail: 1 ytjoe@kict.re.kr, 2 skryu@kict.re.kr ABSTRACT Potholes have caused serious problems such as flat tire, wheel damage, and impact on the lower part of a vehicle. To address the problems, diverse sensors have been employed to automatically detect potholes. An accelerometer detects potholes by recognizing certain signal patterns. It can be implemented at low cost with simple detection algorithms, but it shows wrong detection results with certain objects on the roads such as manholes and speed bumps. Laser scanners can detect correct pothole size and location. However, the sensor is not suitable for pothole detection owing to their high cost and low scalability. Recently, a pothole detection system using a video camera have been proposed. It can detect potholes at low cost and with high accuracy. In this paper, we introduce a pothole detection system based on video data using Android smartphone. A smartphone is suitable device for pothole detection because it has video cameras, processing units, network systems, and GPS. Experimental results show that our system has high detection accuracy. Keywords: Pothole detection, smartphone, video camera, image processing. 1. INTRODUCTION Damaged road such as cracks and potholes may cause serious problems such as flat tire, wheel damage, and traffic accident. In Korea, the number of potholes has increased because of poor pavement materials and maintenance system. The number of potholes was estimated to be approximately 90,000 and 180,000 in 2008 and 2013, respectively. Figure 1 shows total cost for pothole repair in Korea. National government and local government spent most the expense with \7,943 million and \9,628 million in 2013, and \8,116 million and \9,398 million in 2014. In addition, 4,223 car accidents caused by potholes occurred between 2008 and 2013. To address the problems, diverse pothole detection systems have been introduced such as vibration-based methods, 3D reconstruction-based methods, and vision-based methods [1-6]. Vibration-based methods can be implemented at low cost with simple detection algorithm [1, 2], but it shows wrong detection results with certain objects on the roads such as manholes and speed bumps. Laser scanners can detect correct pothole size and location [3, 4], but it is not suitable for pothole detection system because of their high cost and poor scalability. Vision-based methods can detect potholes at low cost with high accuracy. Many studies have been attempted to develop accurate pothole detector using road surface video data [5, 6]. In a previous study, we developed and introduced pothole detection algorithms using video data [7, 8]. At first, we proposed a feature-based pothole detection algorithm [7] for two-dimensional images that uses various pothole features such as ordered histogram intersection (OHI), standard deviation, compactness, and size. In order to improve the detection accuracy, we introduced a motion-based pothole detection algorithm [8]. The algorithm showed better detection accuracy than previous one. However, the two algorithm was not suitable for a smartphone because algorithm complexity was too high. Figure 1: Total cost for pothole repair in Korea. In this paper, we introduce a pothole detection system using an Android smartphone. We show a pothole detection algorithm for the smartphone computing environment. 2. RELATED WORKS Vibration-based methods generally use gradient variation from accelerometer data [1, 2]. Accelerometers have been employed for pothole detection, due to their low cost and relatively simple algorithms. However, the accuracy of detection is lower than other sensors such as cameras and lasers, because potholes are detected only when vehicle s wheels pass a pothole. Moreover, frequent false detections can occur with vehicles passing over manhole covers and speed bumps. Nevertheless, vibration-based methods are advantageous given its low cost and simple methodology. Many studies have been performed in an effort to increase the accuracy of vibration-based detection by designing advanced algorithms and combining other sensor data. Laser scanning has outstanding detection performance, compared to other methods. This approach is 25
able to collect extremely detailed road-surface information using a technique that employs reflected laser pulses to create precise digital models [3, 4]. Accurate 3D point clouds measure elevation in the surface, and this information is captured with the laser and then extracted by filtering the data for specific distress features by means of a grid-based processing approach. However, whereas laser scanning is highly precise, the equipment needed is expensive. Furthermore, this method cannot be applied over a wide area for fast pothole detection. Vision-based methods, however, are appropriate for accurately detecting potholes over a wide area at low costs [5, 6]. Many approaches have been studied using 2D images and video data. Pothole detection using 2D images was originally introduced by Koch and Brilakis [5]. Their method involved searching for specific pothole features and determining pothole regions. They used a remote-controlled robot vehicle prototype equipped with a webcam (HP Elite Autofocus) installed at approximately 60 cm above the ground. Buza et al. introduced a new unsupervised vision-based method that does not require expensive equipment, additional filtering algorithms, or a training phase [6]. 3. POTHOLE DETECITON USING ANDROID SMARTPHONE Previously, we developed pothole detection algorithm using video data [7, 8], which showed great detection performance. In particular, the motion-based algorithm [8] showed the best detection performance. However, the previous algorithms were not suitable for a smartphone because the complexity of the two algorithms was too high. Thus, we redesigned the pothole detection algorithms. As shown in Figure 2, our new algorithm consists of two steps: candidate extraction and decision. The candidate extraction step extracts candidate regions of potholes. Input images are converted into grayscale images, and noise filtering is performed. In the first noise filtering step, we used Gaussianblur to remove high frequency of pixels. Filtered images are converted into binary values; 1 and 0. The thresholding step is to extract pothole candidate regions. The extract regions are filtered by noise filters such as a median filter and a morphology operation. In the decision step, position, size and standard deviation values are used to make final decision of potholes. Through the proposed algorithm, we can produce pothole location and size data. Geographical location is collected by GPS and size data is extracted by detected pothole regions such as the number of pixels. Figure 2: Pothole detection algorithm. Figure 3: Pothole detection software overview. In order to implement our pothole detection system, we developed three different software; pothole detector, pothole client, and pothole server. As illustrated in Figure 3, the pothole detector and pothole client are embedded on a smartphone, and the pothole server is installed on a server system. Table 1 shows the overview of the three programs. The pothole detector is to detect pothole regions by the proposed pothole detection algorithm. The pothole client transmits collected pothole information such as location and size. The pothole server receives pothole information and builds a pothole database. Data transmitting is periodically performed such as 30 seconds and 60 seconds. Table 1: Explanation of pothole detection programs. Software Pothole detector Pothole client Explanation Pothole detection program by the proposed algorithm Transmitting program that collects pothole information such as location and size Pothole server Pothole database program 26
Figure 4: Pothole client. Figure 6: Pothole server. Figure 5: Pothole detector. Figure 4 depicts the pothole client software where users can input server IP address and port number. The program transmits pothole information to pothole server program every certain seconds. Figure 5 shows pothole detector software that should receive parameter information such as thresholding values of minimum and maximum pothole size, standard deviation, and position. Figure 6 shows pothole server program that receive pothole information from pothole client and save the data into pothole database. Users can see data transmitting and saving operation. The received pothole data is used by pothole monitoring system as shown in Figure 7. The monitoring system is developed in JavaScript where users can see pothole images and geographical location on a digital map. The detailed information such as vehicle driver names, vehicle number plate, certification number, detection time, address, XY coordinates, size, direction, and speed are provided in real-time. Figure 7: Pothole monitoring system. 4. EXPERIMENT RESULTS In this study, we tested our system on two different roads where 20 and 24 pothole images are collected in national roads in Seoul city and in national highway in Gyeonngi-do, respectively. Figure 8 and 9 show the pothole examples on the two different roads. We used Galaxy Note 7 to test our pothole detection system because currently it provides the greatest computing environment and camera performance. The captured video size is 1920 x 1080 and the ROI (Region Of Interest) is located at (420, 800) and is 1000 x 200 in size. The pothole detection algorithm is implemented in C++ and OpenCV. 27
Table 2 shows the evaluation results. In national roads, our system reached 85% of accuracy and in national highway it showed 83% of accuracy, which means our algorithm can detect potholes with over 80% accuracy. In object detection using computer vision, over 80% accuracy is pretty high value. Table 2: Performance evaluation. Tested Roads Detection Accuracy National roads in Seoul city National highway in Gyeonggi-do 85% 83% 5. CONCLUSION In this paper, we analyzed our previous pothole detection algorithm and introduced new pothole detection algorithm for Android smartphone. The proposed algorithm was designed for the computing environment of a smartphone. We reduced the algorithm complexity with proper detection accuracy. Figure 8: Examples of potholes on national roads in Seoul city. We installed the smartphone on the front windshield, and tested on the roads in Seoul city and Gyeonggi-do. The detection accuracy was about 85% and 83%, respectively. The hardware performance of smartphones will increase continuously. Thus, powerful machine learning algorithm such as deep learning can be used for real time pothole detection. We will develop machine learning algorithms such as support vector machine, logistic regression, and deep learning for pothole detection algorithm. We expect that employing machine learning algorithms will lead to further improvement of detection accuracy. ACKNOWLEDGEMENTS This research was supported by a grant from a Strategic Research Project (Development of Pothole-Free Smart Quality Terminal, 2016-0164) funded by the Korea Institute of Civil Engineering and Building Technology. Figure 9: Examples of potholes on national highway in Gyeonggi-do REFERENCES [1] B. X. Yu, and X. Yu, Vibration-based system for pavement condition evaluation, In Proceedings of the 9th International Conference on Applications of Advanced Technology in Transportation (2006), 183-189. [2] K. De Zoysa, C. Keppitiyagama, G. P. Seneviratne, and W.W.A.T. Shihan, A public transport system based sensor network for road surface condition monitoring, In Proceedings of Workshop on Networked Systems for Developing Regions (2007), 1-6. [3] K. C. P. Wang, Challenges and feasibility for comprehensive automated survey of pavement conditions, In Proceedings of 8th International Conference on Applications of Advanced Technologies in Transportation 28
Engineering (2004), 531-536. [4] K.T. Chang, J. R. Chang, and J. K. Liu, Detection of pavement distress using 3D laser scanning technology, In Proceedings of the ASCE International Conference on Computing in Civil [5] C. Koch, and I. Brilakis, Pothole detection in asphalt pavement images, Advanced Engineering Informatics, Vol. 25 (2011), 507-515. Engineering (2005), 1-11. [6] Buza, E., S. Omanovic, and A. Huseinnovic, Stereo vision techniques in the road pavement evaluation, In Proceedings of the 2nd International Conference on Information Technology and Computer Networks, (2013), 48-53. [7] S. K. Ryu, T. Kim, Y. R. Kim, Feature-Based Pothole Detection in Two-Dimensional Images, Transportation Research Record(2015), 9-17. [8] Y. T. Jo, S. K. Ryu, Y. R. Kim, Pothole Detection Based on the Features of Intensity and Motion, Transportation Research Record(2016), 18-28. AUTHOR PROFILES Youngtae Jo received the degree in computer information and communication engineering at Kangwon National University in Korea. He is a senior researcher at Korea Institute of Civil Engineering and Building Technology. His research interest covers intelligent transportation system, embedded system, wireless sensor networks, and robotics. Seungki Ryu received the degree in electrical engineering at ChungBuk National University in Korea. Currently, he is a research fellow at Korea Institute of Civil Engineering and Building Technology. His research interest covers intelligent transportation systems, information technology, ubiquitous city, construction IT convergence and logistics. 29