Thermal Imaging-Based Vehicle Classification in Nighttime Traffic Apiwat Sangnoree King Mongkut s University of Technology Thonburi Kosin Chamnongthai

Transcription

1 Thermal Imaging-Based Vehicle Classification in Nighttime Traffic Apiwat Sangnoree King Mongkut s University of Technology Thonburi Kosin Chamnongthai King Mongkut s University of Technology Thonburi Figure 1: Conceptual Design of the Research Abstract This research proposes a novel method for classifying vehicles in nighttime traffic by utilizing thermal imaging-based as the source of analysis. Using the proposed classifying method, light occlusions from vehicles and surrounding environment are able to be eliminated. Naturally, various thermal features from a vehicle, comprising windscreen, engine heat are spontaneously occurred and illustrate as different intensities in thermal images. In the research, the mentioned features are utilized for categorizing vehicle types. For classifying steps, initially, the proposed model will search for engine heat of a vehicle and select a suspected area which cover windscreen feature of the vehicle. Secondly, the area is thresholded for categorizing the intensity of windscreen, engine and environment heat, respectively. As the result, the vehicle front is able to be extracted from the mentioned area. For the classification which is the final step, the extracted vehicle front will be scaled, reformed and compared to assigned templates in the database for categorizing the type of vehicle. Experimentally, with the four types of vehicle, consisting of motorcycle, car, van and truck, respectively, the accuracy of classifications are over 86 %. Keywords: thermal imaging, vehicle classification, nighttime traffic 1 Introduction This research has been done to aid ITS (Intelligent Transportation Systems), especially for nighttime traffic, in vehicle classification affair. So far, to classify vehicles in the nighttime, the light occlusions from both vehicles and peripheral light sources have distorted an analysis. Nowadays, many researches attempt to solve the mentioned constraint in different ways. Unfortunately, because of unre- sangnoree-bank@yahoo.com kosin.chamnongthai@gmail.com liable results, the constraint remains an open problem. As the reason, the research is introduced for being an alternative for solving such trouble with the following purposes: 1. Visibility Analysis. This research classifies types of vehicles by using thermal images as the source of analysis. The analysis is done by image processing method, including additional techniques. Therefore, the classifying results of the research can be compared with human s eyes inspection. 2. Portability and Non-Destructive Technique. The research uses a thermal camera to record thermal images from the road side and makes the analysis by a portable computer. As the reason, the advantage of such method is portability as well as non-destructive process, comparing to those methods which need the installation of devices in the road surface. 3. Alternative Method for Vehicle Classification. Because the classifying method is based on thermal features of vehicles, meanwhile traditional image processing methods need visible lights of objects to analyze, this research is an alternative for vehicle classification in the present and future. 2 Literature Review Nowadays, for ITS, various methods have been utilized for classifying vehicles i.e. 1) Radio frequency sensing technology. Such technology, for example, RFID and GPS, works well with registered vehicles. However, because the installation of communicating devices is fundamentally needed, it is impossible to classify vehicles if the mentioned devices are not available. 2) Inductive loop detection can be placed in a roadbed to detect vehicles as they pass over the loop by measuring the vehicle s magnetic field. The simplest detectors simply count the number of vehicles during a unit of time that pass over the loop, while more sophisticated sensors estimate the speed, length and weight of vehicles and the distance between them. Loops can be placed in a single lane or across multiple lanes, and they work with very slow or stopped vehicles as well as vehicles moving at high-speed. Nevertheless, because we cannot see whether inspected objects are vehicles or not, the results of classification are still not reliable for the case of non-vehicles. 3) Video vehicle detection. Daytime vehicle classification. So far, there have been implemented 124

2 Figure 2: Three Thermal Features of Vehicle-Front Figure 3: Proposed Model of the Research technologies for daytime traffic monitoring. The method in [Gupte et al. 2002] uses only one video camera, assuming there are no occlusions from vehicle shadows and two categories can be decided, cars and no cars on the road. In addition, [Yoshida et al. 2002] improved the method of vehicle classification for more categories by using 3D vehicle feature. However, it is unfeasible for real-times applications. In [Hsieh et al. 2006] proposed line-based elimination technique for solving shadow occlusions from vehicles by using only one video camera but it is difficult to classify vehicles which have dark tone colors. The excellent results of vehicle tracking and classification by [Pang et al. 2007] contribute further researches for detecting focused vehicles in daytime-traffic. Unfortunately, when implementing in nighttime, according to [Thi et al. 2008], it will not be guaranteed to give the same results. Nighttime vehicle classification. As mentioned in daytime vehicle classification, those methods are suitable for high intensity of light in observing scene. Nevertheless, it is still not work with low light situation in nighttime traffic. In nighttime case, [Lee et al. 2002] proposed two-step tracking algorithm to CCD images for utilizing in nighttime traffic but headlights from vehicles made difficulties in vehicle detection. In addition, [Kanai et al. 2005] declared that when using CCD images for nighttime traffic, the neat camera adjustment is also required. In [Chen et al. 2006] detected vehicle headlights and taillights in order to inspect whether there are near vehicles or not. However, it is unreliable when the vehicles turn off the lights. From preceding researches that are mentioned previously, the vehicle s detection for ITS has utilized different image processing techniques, depending on the specific situations. However, for nighttime traffic evaluations, the efficiency of classification is required to improve continually. In this research, we apply thermograph technique for classifying vehicles. Such technique has been acceptable for vehicle tracking and classification by [Sangnoree and Chamnongthai 2008]. According to the mentioned research, their models aimed to classify types of any vehicle by considering only on the size of reflected heat underneath the vehicle. As the reason, the accuracy of classification was still relatively low because there was not enough information for categorizing running vehicles on the road into many types. To improve the classifying accuracy, [Sangnoree and Chamnongthai 2009] utilized more thermal features from a vehicle front. The mentioned features comprise windscreen, reflected heat underneath the vehicle and elsewhere. However, the classification result was ineffiecient with some vehicles. Although there are face recognition methods related in the vehicle classification, in thermal imaging-based which relates with various intensities, as mentioned in [Liao 2007], those methods are still not the best solution. As previously mentioned, a modified concept and design of our research for vehicle classification, regarding on thermal imaging-based were undertaken. The research concept and processes will be declared in the latter sections. 3 Conceptual Design of the Research For the conceptual design of this research, it is shown as in figure 1. At the first step, a thermal camera which is utilized for recording thermal image frames of the focused vehicles in the nighttime traffic will be installed in a position that the thermal features which are radiated from various types of vehicles can be clearly recorded. The mentioned features considerably comprise the underneath heat, windscreen and elsewhere, respectively. As illustrated in figure 2, those three features have different levels of intensity, and will be utilized for analyzing vehicle types. Hereafter, the mentioned features appeared on an image frame, these features will then be delivered to the vehicle classification processor in order to categorize all detected vehicles that can be seen on a frame. For the full detail of classifying process, it will be deeply identified in the following section. 4 Proposed Model of the Research Because of different theme for vehicle classification, comparing to traditional researches, in this research, levels of the intensity of thermal features will be instead used to categorize vehicle types. As shown in figure 3, the operation of proposed model of the research 125

3 Figure 4: The Sample of Free Road and Its Intensities Figure 5: The Process of Vehicle-Front Search starts by inputting thermal image frames into the classifying process. All steps of the model are as followings: 4.1 Vehicle Detection To detect suspected objects whether vehicles or not, intensity of free road is used as the key factor. Generally, all vehicles are driven on the road surface and heat features from engines spontaneously occur. As the reason, the road intensity which implies the comparator to vehicles is utilized to subtract the objects from an image. Obviously, the vehicle detection method of this research is different from the others that use the background subtraction. The sample of free road and its intensities are displayed in figure Vehicle-Front Search Hereafter, an image frame was subtracted by the road intensity, there will only be heat features of objects appear in the image. In this research, they are used as the positioning index of vehicles in the image. The process to search for vehicular thermal features illustrated in figure 5. Although the mentioned index was known, it is unable to specify whether vehicle or not. In this research, for a vehicle, we use the heat feature to define a vehicle area. The defined area covers windscreen, engine heat and elsewhere feature of the vehicle. The next step is to sort all three features by the automatic thresholding, the thresholded area will then be transformed to have three intensity levels, 0, 1 and 2, respectively. The final step is to search for vehicle-front, when the thresholded vehicle area was made, the vehicle-front area can be obviously found because the location of windscreen area implies the position of vehicle-front. The cropped area of vehicle-front from the thresholded vehicle will be utilized for classifying vehicle in the next operation. 4.3 Vehicle Classification When the vehicle-front was extracted from the vehicle area, it will be used for categorizing a vehicle type. The process to classify a vehicle illustrated in figure 6, the operation starts by scaling the vehicle-front to an assigned size. Because vehicles appear in an image frame feasibly differ from the latter frames, as the reason, it is necessary to scale them, being the same size for all vehicles. Secondly, the scaled vehicle-front will then be reformed to have the same size as designed templates in the database. The mentioned templates comprise motorcycle, car, van and truck, respectively. Eventually, the reformed vehicle-front is taken to compare with those templates in order to decide the type of vehicle. 5 Experimental Results For testing the proposed vehicle classification method, we used a NECTH 7100 thermal camera to record a sample nighttime traffic on Rangsit-Nakornayok road, Bangkok, Thailand. A stream of thermal images, 1223 frames, 30 frames/second was recorded. figure 7 (a) shows the free road. Figure 7 (b) and (c) illustrate running vehicles on the road, consisting of motorcycle, car, van and truck. The results of each operating stage are declared as the followings: 5.1 Vehicle Detection Method and Results As previously described, in this research, the detection of a vehicle is done by subtracting all intensities which appear in a recorded frame with free road intensity. As shown in figure 8 (a) is the selected area to detect suspected objects. Because of many levels of intensity of the sample free road, as illustrated in figure 8 (b), the representative of free road level is (1) where road intensity equals its average intensity plus standard deviation. Hereafter, the road intensity was computed, the heat feature of objects that appear on the frame can be detected by assigning any pixels which has lower intensity than the road intensity equals zero, and the others to one. A sample of the detection is illustrated in figure 9. R = µ + σ (1) 5.2 Vehicle-Front Search Method and Results 1) Sample results of the selected area of vehicle. The vehicle-front search starts with considering on defined area from heat feature of vehicle. As earlier mentioned, when the heat feature area was detected, in this research, 2.5 times of the area is vertically selected from the original frame. Experimentally, the selection can cover engine heat, windscreen and elsewhere of a vehicle. Figure 10 showed the sample results of the process; there are four types of the selection area comprise motorcycle, car, van and truck, respectively. 2) Sample results of the automatic thresholding of vehicle. The selected area of vehicle will then be thresholded to sort windscreen, heat and elsewhere. The automatic thresholding is implemented for this task. Sample results of the operation are displayed in figure 11. 3) Sample results of cropped vehicle-front. From preceding method, when the area of windscreen was found, the next step is cropping for vehicle-front. Sample results of such method are shown in figure

4 Figure 6: The Vehicle Classification Process Figure 7: The Sample of Recorded Thermal Images 5.3 Vehicle Classification Method and Results Hereafter, the cropped vehicle-front was passed the previous method, it will be scaled to an assigned size. In this research, the assignment is set to 20 X 20 pixels for all vehicle-fronts. The vehiclefront will then be reformed to have its size equals vehicle templates in the database. Experimentally, the template size is set to be 10 X 4 pixels. In figure 13, the results of reformed vehicle-front of motorcycle, car, van and truck as well as designed vehicle templates are illustrated. Eventually, when the reformed vehicle-front was compared to all templates, the type of vehicle can be declared depends on the most percentage of similarity to those templates and the percentage of corrections of the classification are listed in table 1. 6 Conclusion The research presents a thermal imaging-based vehicle classification in nighttime traffic. With the proposed method, four types of vehicle are categorized. The percentage of corrections of classification is satisfyingly high. Experimentally, 89.83, 88.98, and % of accuracy are founded for motorcycle, car, van and truck, respectively. Acknowledgements According to the success of this research, I would like to sincerely admire the Thailand Institute of Scientific and Technological Research for aiding us the thermal camera which is utilized for recording nighttime thermal images. In addition, the financial support from The University of the Thai Chamber of Commerce for attending the conference is also the valuable thing to respect. References CHEN, Y. L., CHEN, Y. H., CHEN, C. J., AND WU, B. F Nighttime vehicle detection for driver assistance and autonomous vehicles. In Proceedings of 18th International Conference on Pattern Recognition, vol. 1, IEEE Computer Society, GUPTE, S., MASOUD, O., MARTIN, R. F. K., AND PA- PANIKOLOPOULOS, N. P Detection and classification of vehicles. IEEE Transactions on Intelligent Transportation Systems 3, 1 (Mar.), HSIEH, J. W., YU, S. H., CHEN, Y. S., AND HU, W. F Automatic traffic surveillance system for vehicle tracking and classification. IEEE Transactions on Intelligent Transportation Systems 7, 2 (Jun.), KANAI, H., FUJITA, K., ISHIZAWA, H., MATSUMOTO, Y., TOBA, E., AND NISHIMATSU, T Evaluation of lane indication performance using ccd image sensor. In Proceedings of 2005 IEEE conference On Instrumentation and Measurement Technology Conference, vol. 3, IEEE Computer Society, LEE, I., KO, H., AND HAN, D. K Multiple vehicle tracking based on regional estimation in nighttime ccd images. In Proceedings of 2002 IEEE conference on Acoustics, Speech, and Signal Processing, vol. 4, IEEE Computer Society, LIAO, K. J Image-based pattern recognition principles. National Taiwan University. PANG, C. C. C., LAM, W. W. L., AND YUNG, N. H. C A method for vehicle count in the presence of multiple vehicle occlusions in traffic images. IEEE Transactions on Intelligent Transportation Systems 8, 3 (Sep.), SANGNOREE, A., AND CHAMNONGTHAI, K An application of thermography to solve the problem of nighttime tracking and classification and occlusions of vehicles in its system. In Proceedings of 2008 Nicograph International, vol. 1, Nicograph International, SANGNOREE, A., AND CHAMNONGTHAI, K An efficiency increment for vehicle s classification in nighttime traffic by analyzing on thermal features of vehicles front side. In Proceedings of 2009 Nicograph International, vol. 1, Nicograph International, THI, T., ROBERT, K., LU, S., AND ZHANG, J Vehicle classification at nighttime using eigenspaces and support vector machine. In Proceedings of 2008 Congress on Image and Signal Processing, vol. 2, IEEE Computer Society, YOSHIDA, T., MOHOTTALA, S., KAGESAWA, M., AND IKEUSHI, K Vehicle classification system with local-feature based algorithm using cg model images. IEICE Transactions on Information System E85D, 12 (Dec.),

5 Figure 8: The Considered Boundary and Intensity Distribution of Free Road Figure 9: The Sample of Heat Feature Detection of Vehicles Figure 10: The Sample of Defined Area from Heat Feature of Vehicles Figure 11: Sample Results of Automatic Threshoded Vehicle Figure 12: Sample Results of Cropped Vehicle-Front 128

6 Table 1: The Percentage of Corrections of Vehicle Classification Type Times of appearance Correct Classification Incorrect Classification Percentage of Correction Motocycle % Car % Van % Truck % Figure 13: The Sample Results of Reformed Vehicle-Front and Templates in The Database 129