User talk:Amit1201

=Real time traffic signal detection using image processing=

An implementation and limited extension of a traffic sign recognition method based on the work by [1] is presented here. This implementation can be extended to target real-time detection. Yield sign, stop sign and red-bordered circular signs are considered. First, image is color segmented based on a thresholding technique. Then, corner features are detected using convolution masks (based on work in [2] called optimal corner detector in comparison to Canny’s work). Geometric constraints used for shape recognition along verification methods for each sign. Keywords: Traffic sign recognition, corner detection 1 Introduction Traffic sign recognition (TSR) can be considered part of the bigger problem of autonomous vehicles. An autonomous vehicle system relies on vision-based recognition of surrounding area in order to make driving decisions. This vision-based recognition system functions as the feedback provider for control of steering wheel, accelerator, brake, ..., and needs to: • Recognize road and lane to allow control system follow the course of own vehicle. • Detect obstacles on the road till control system avoid them. • Detect the passing vehicles (e.g. by side or back cameras) to notify the control system about probable hazards. • Detect and interpret the traffic signs to provide feedback for safe driving. Traffic signs provide important information for drivers about road condition and hazards. Their discriminating shape and colors make them easily recognizable by humans. Same factors can help development of a visionbased TSR system. Beside the application of TSR in autonomous vehicles, it can also serve as an assistant driver (e.g. when combined with speedometer output) to notify the driver about approaching a traffic sign (e.g. even before driver sees it) or his risky behavior (like driving above the speed limit). There have been several big projects in the autonomous vehicle area. PROMETHEUS (Program for European Traffic with Highest Efficiency and Unprecedented Safety) was an initiative started in 1986 in Europe to stimulate research in development of a transport system to exploit full potential of information and telecommunication technologies to improve traffic efficiency and safety [3]. Safe driving was one of the three identified main work areas and meant to employ autonomous vehicle control for safer driving with less mental load on the driver. A TSR system was developed in Daimler-Benz as part of the collision avoidance project. It employed a detection process to scan the image for possible traffic sign candidates and a tracking process to identify the sign and track it in the following images. Three sub-units (called specialists) were used: color segmentation specialist to find a sign candidate based on color of regions in the image, shape recognition specialist to classify the candidate based on its contour and pictogram recognition specialist to identify the pictogram inside the traffic sign by comparing it against a library of possible pictograms. UC Berkeley’s PATH is another example. It is a collaboration between the California Department of Transportation (Caltrans) and the University of California with the mission of applying advanced technology to increase highway capacity and safety, and to reduce traffic congestion, air pollution, and energy consumption [4]. It consists of several projects. For example its Stereo Drive project explored the feasibility of use of stereo vision (for providing range information) in conjunction with a scanning laser radar sensor to detect obstacles and also maintain a fixed distance from a lead vehicle using feedback provided by range sensors. In particular area of TSR, there have been many research in academia and also industry (e.g. as part of PROMETHEUS) especially in the mid-90’s. Different approaches have been used in different stage of the problem from color segmentation, control theory, feature extraction, learning-based neural networks, morphological based recognition, ... [5] cascades three modules: an ART2 neural network module for color segmentation, a log-polarexponential grid and Fourier transformation module to extract invariant traffic sign signatures (its first 16 Fourier coefficients), and a back-propagation neural network module to classify such signatures. Instead, [6] uses a split Figure 1: Some of European and North American signs differences. and merge concept for a color segmentation technique insensitive to noise and brightness variation. [7] discusses pictograph identification of different class of signs (which previously detected and classified): a structural approach for arrow signs, a number detector for speed limit signs, a general nearest neighbor classifier and neural network for prohibition signs. In an ideal condition (off-line indoor detection of signs with direct front view) traffic sign recognition is not very hard in principle as signs have discriminating color and 2- D shape. Some of the main issues involved are: • Variety of signs with all different colors, shape and pictographic symbols. • The complex and uncontrolled road environment (lighting, shadow, occlusion, orientation, distance, ...) which all can be categorized as noise (see Section 4 of [5] for description of variety of noise for a TSR system. • And real-time recognition. In this paper, we first overview the method in Section 2 and explain it in details in Section 3. We discuss the implementation in Section 4 and provide the test results in Section 5. Section 6 briefly overviews how this implementation can be extended. And finally Section 7 presents our conclusions.

2 Overview of the method Like other classical object recognition problems, our problem can be divided to localization and identification; first an area with probable sign is segmented, then passed for identification. Not every approach makes such a clear distinction between localization and identification. While [3] does these two steps quit separately, localization in our case also identifies the sign partially (depending on type of the sign). Our method is based on the work by De La Escalera et al. in [1]. They applied their method to European warning signs (equilateral triangles with one vertex upward) and circular prohibition signs (see Figure 1 for comparison of European and North American signs). Here are the main stages of their method: Color segmentation Based on the traffic signs to be detected, input image is passed through a color mask. The output is a thresholded binary image carrying white pixels for the points where the desired color range exist and black pixels otherwise. Corner detection A set of corner detectors (based on convolution masks) are applied to the binary image from the previous step. To reduce number of false corners, corners within a neighborhood will be combined into their center of mass. Shape recognition Using geometry information of signs of interest, corner combinations creating possible signs are looked up. Every possible sign area is normalized to a 30*30 pixels image. Sign classification A classification method based on neural network used for classifying the signs at this stage. Figure 2 describes the above stages. [1] restricts the signs of interest to warning signs (triangles with upward vertex), prohibition signs (circles) with white or blue background and red border, and rectangular informative signs. Yield sign (inverted triangle) and stop signs were excluded. The reason for choice of signs was not provided in the paper. As they experimented with two sets of training sets, one for circular signs and the other for triangular signs, we suspect they were looking for a variety of signs all with the same shape (triangular or circular) but with different pictographic content. Nine ideal signs (segmented sign without noise) were selected for training the network for each group. Beside the ideal signs, different variation of them created (5 rotated signs within ±6°, three Gaussian noise levels, 4 different color threshold levels, three pixel displacement to the left and right) ending with 1620 training patterns. From three different network size studied, the one with 30 × 30/15/5/10 responded the best with 82 − 98% success rate for detecting signs from another set of test images. The achieved recognition speed was 220ms for detection phase for a 256×256 image on a special image processing card. The neural network was taking 1.2s on a PC which the authors intended to decrease with implementing the algorithm on a DSP instead (hoping to achieve 30 − 40ms). In our project, we implemented the above method but with the following restrictions and modifications: • No pictographic sign classification implemented as implementing the neural network too was making the project beyond its intended scope. 2 Figure 2: Flow of processing in the method. • To limit the variety of signs, only yield sign (with similar detection as upward triangles of [1] and stop sign (as an extension) were targeted in the beginning. Later, support for circular red-bordered signs added too. • Similar to the original method, only limited degree of sign rotation is acceptable (increased to ±10°) in shape recognition. • Similar to the original method, image signs taken from narrow angles are not supported. Only minor skew is allowed. Basically minor sign rotation and minor change of view-angle (from the sides) are supported (along sign translation and scaling). • Similar to the original method, occlusion is not supported. All corners of a sign must be visible. • The original method didn’t need to worry about verification of detected shapes as the neural network in the classification stage would ignore unrecognized images (treat them as outliers). But here we need to have verification mechanism to be able to distinguish a real sign from just a few points which happen to satisfy our border points condition. More on this provided in Section 3.3. • Though [1] targets a real-time TSR system, we limit ourselves to still image to simplify our implementation. Each one of the steps is explained in details in the following section. 3 Description of the method As Figure 2 shows, the main stages of our implementation are color segmentation, corner detection and shape recognition. We describe each step in details here. 3.1 Color segmentation Color information in contrast to shape information is invariant to scale, rotation and view, and possesses high discriminative power. Traffic signs have distinctive colors, so using color segmentation methods are very popular in TSR systems. As the signs of interest in our implementation all have red color in their borders, we can decrease the complexity of our input image (which is in RGB format) to a binary image by applying a color thresholding algorithm for shades of red color. Then, detection of corners can be done in this binary space instead of the original image leading to less computation time. Color thresholding in RGB color space can be achieved like Equ. 1 where g(x, y) is the output of thresholding, k1 and k2 are the output binary values and fr(x, y), fg(x, y) and fb(x, y) are the red, green and blue components of input image respectively (see [1]).

Rmin ≤ fr(x, y) ≤ Rmax Gmin ≤ fg(x, y) ≤ Gmax Bmin ≤ fb(x, y) ≤ Bmax k2 otherwise (1) Rmin and Rmax define the range of red component for pixels to be considered red in traffic signs. Note that traf- 3 fic signs don’t have pure red. So green and blue components are non-zero though smaller than the red component. As the result, similar range for green and blue components need to be defined too. But there is a big problem with working in RGB color space as it is very sensitive to lighting changes (e.g due to sun angle, weather, clouds, ...). So with only minor change of lighting, a previously defined color range in Equ. 1 will not be valid anymore. A common method to attack this is to first convert the image to HSI color space (stands for hue, saturation and intensity; also called HSV color space). HSI space decouples the intensity component from colorcarrying information (hue is color purity, saturation is color dilution by white light). For more information about HSI color space see section 6.2.3 of [8] or RGB to HSI tutorial in [9]. But converting to HSI requires evaluation of some non-linear and trigonometric equations which are expensive to evaluate for every single pixel. Other variation is proposed by [10] to use ratio of pixel’s red component to its overall intensity to define the range of redness. Another simplification will use a range for red component and other ranges for ratio of blue and green components to red component for defining the threshold criteria as shown below.

3.2 Corner detection Now we have a binary image resulted from thresholding the input image for a proper shade of red color using Equ. 2 and defining proper ranges we determined by investigating some samples images. We need to find corners in this binary image (i.e. corresponding to red corners in the original image) to be used for shape recognition (a later stage). Using mask convolution for detection of different patterns is a well-studied area beside its common applications in filtering and picture enhancements. Canny’s edge detector convolves the image with masks representing first derivative of Gaussian as the first step in its detection process. Similar masks can be derived for other shapes. For example [11] generates angle-dependent convolution masks for detecting circular and radial edges in detection of circular traffic signs. Their two detectors called Detector of Circular Edges (DCE) and Detector of Radial Edges (DRE) are created by a set of convolution masks derived by aligning a base function (i.e. an edge detection mask like Sobel) with tangents to the circle (for DCE) or radials (for DRE). Of course, their method has the deficiency of relying on center of circular sign be known as a priori. Hough transform is a generalized method that can be used for detecting many shapes. But its main deficiencies are high processing time and large memory requirements. Our shape recognition uses corners as primary features to find shape information from geometrical relations of corners. Common corner detectors use edge information or eigenvectors of gradient in a pixel neighborhood. They involve many steps. [1] employed a corner detector called ”Optimal corner detector” which was developed by [2]. This corner detector has two advantages. First, since it works by convolving the image against some masks, it works faster than previously mentioned detectors (needs some simplifications as did by [1]). This makes it very suitable for our case of real-time detection were speed is a very important factor. Second, this method can classify type of detected corners (by angle and direction) which helps to reduce complexity of shape recognition phase (more on this in section 3.3).