• Automatic License Plate Recognition Using Python And Opencv

In our previous lesson, we learned how to using basic image processing techniques, such as morphological operations and contours. Figure 1: Examples of detecting license plate-like regions in images.In each of the images above, you can see that we have clearly found the license plate in the image and drawn a green bounding box surrounding it.The next step is for us to take this license plate region and apply image processing and computer vision techniques to segment the license plate characters from the license plate itself. Character segmentationIn order to perform character segmentation, we’ll need to heavily modify ourlicenseplate. Py file from the previous lesson. This file encapsulates all the methods we need to extract license plates and license plate characters from images. TheLicensePlateDetector class specifically will be doing a lot of the heavy lifting for us.

Here’s a quick breakdown of the structure of theLicensePlateDetector class. -detectCharacterCandidatesWe’ve already started to define this class in our previous lesson. Theinit method is simply our constructor. Thedetect method calls all necessary sub-methods to perform the license plate detection. The aptly nameddetectPlates function accepts an image and detects all license plate candidates. Finally, thedetectCharacterCandidates method is used to accept a license plate region and segment the license plate characters from the background.ThedetectPlates function is already 100% defined from our previous lesson, so we aren’t going to focus on that method.

However, we will need to modify the constructor to modify a few arguments, update thedetect method to return the character candidates, and define the entiredetectCharacterCandidates function.Since we’ll be building on our previous lesson and introducing a lot of new functionality, let’s just start from the top oflicenseplate. Py and start reworking our code. LicensePlate = namedtuple ( 'LicensePlateRegion'success', 'plate', 'thresh', 'candidates' )The first thing you’ll notice is that we’re importing a lot more packages than from our previous lesson, mainly image processing functions from scikit-image andimutils.We also define anamedtuple on Line 12 which is used to store information regarding the detected license plate. If you have never heard of (or used) anamedtuple before, no worries — they are simply easy-to-create, lightweight object types. If you are familiar with the C/C programming language, you may remember the keywordstruct which is a way of defining a complex data type.Thenamedtuple functionality in Python is quite similar to astruct in C. You start by creating a namespace for the tuple, in this caseLicensePlateRegion, followed by a list of attributes that thenamedtuple has. Since the Python language is loosely typed, there is no need to define the data types of each of the attributes.

In many ways, you can mimic the functionality of anamedtuple using other built-in Python datatypes, such as dictionaries and lists; however, thenamedtuple gives you a big distinct advantage — it’s dead simple to instantiate newnamedtuples. MinCharW = minCharWNot a whole lot has changed with our constructor, except that we are now accepting two more parameters:numChars, which is the number of characters our license plate has, andminCharW, which, as the name suggests, is the minimum number of pixels wide a region must be to be considered a license plate character.You might be thinking, isn’t it cheating to supply a value likenumChars to ourLicensePlateDetector? How do we know that the license plate we have detected contains seven characters? Couldn’t it just as easily contain five characters?

Or ten?Well, as I mentioned in our lesson (and is true for all computer vision projects, even ones unrelated to ANPR), we need to apply as much a priori knowledge as we possibly can to build a successful system. After examining our dataset in the previous two lessons, a clear piece of knowledge we can exploit is the number of characters present on the license plate. Figure 2: An important fact to note is that each of our license plates contains 7 characters — 3 letters followed by 4 numbers. We’ll be able to exploit this information to build a more accurate character classifier later in this module.In each of the above cases, all license plates contain seven characters. Thus, if we are to build an ANPR system for this region of the world, we can safely assume a license plate has seven characters. If a license plate does not contain seven characters, then we can flag the plate and manually investigate it to see (1) if there is a bug in our ANPR system, or (2) we need to create a separate ANPR system to recognize plates with a different number of characters.

Remember from our, ANPR systems are highly tuned to the regions of the world they are deployed to — thus it is safe (and even presumable) for us to apply these types of assumptions.Ourdetect method will also require a few minor updates. Yield ( lp, lpRegion )The changes here are quite self-explanatory. A call is made todetectPlates to find the license plate regions in the image. We loop over each of these license plate regions individually, make a call to thedetectCharacterCandidates method (to be defined in a few short sentences), to detect the characters on the license plate region itself, and finally, return a tuple of theLicensePlate object and license plate bounding box to the calling function.ThedetectPlates function is defined next, but since we have already reviewed in our, we’ll skip it in this article — please refer that lesson for the definition ofdetectPlates. As the name suggests, it simply detects license plate candidates in an input image.It’s clear that all the real work is done inside thedetectCharacterCandidates function. This method accepts a license plate region, applies image processing techniques, and then segments the foreground license plate characters from the background. Let’s go ahead and start defining this method.

Automatic License Plate Recognition Using Python And Opencv

Figure 3: Our license plate detection method returns a rotated bounding box corresponding to the location of the license plate in the image.However, looking at this license plate region, we can see that it is a bit distorted and skewed. This skewness can dramatically throw off not only our character segmentation algorithms, but also character identification algorithms when it comes to applying machine learning later in this module.Because of this, we must first apply a to apply a top-down, bird’s eye view of the license plate. Figure 4: The original license plate could be distorted or skewed which can hurt character classification performance later in the ANPR pipeline. We apply a perspective transform to obtain a top-down, 90-degree viewing angle of the license plate to help alleviate this problem.Notice how the perspective of the license plate has been adjusted, as if we had a 90-degree viewing angle, cleanly looking down on it.Note: I cover perspective transform twice on the PyImageSearch blog.

The covers the basics of performing a perspective transform, and the second post applies perspective transform to solve a real-world computer vision problem —. Once we get through the first round of this course, I plan on rewriting these perspective transform articles and bringing them inside of PyImageSearch Gurus. But for the time being, I think the explanations on the blog clearly demonstrate how a perspective transformation works. Imshow ( 'Thresh', thresh )Now that we have a top-down, bird’s eye view of the license plate, we can start to process it.

The first step is to extract the Value channel from the HSV color space on Line 109.So why did I extract the Value channel rather than use the grayscale version of the image?Well, if you remember back to our lesson on, you’ll recall that the grayscale version of an image is a weighted combination of the RGB channels. The Value channel, however, is given a dedicated dimension in the HSV color space. When performing thresholding to extract dark regions from a light background (or vice versa), better results can often be obtained by using the Value rather than grayscale.To segment the license plate characters from the background, we apply on Line 110, where thresholding is applied to each local 29 x 29 pixel region of the image.

As we know from our, basic thresholding and Otsu’s thresholding both obtain sub-par results when segmenting license plate characters. Figure 5: Applying Otsu’s thresholding method in an attempt to segment the foreground characters from the background license plate — notice how the bolt of the license plate is connected to the “0” character.

This can cause problems in character recognition later in our ANPR pipeline.At first glance, this output doesn’t look too bad. Each character appears to be neatly segmented from the background — except for that number 0 at the end. Notice how the bolt of the license plate is attached to the character.

While this artifact doesn’t seem like a big deal, failing to extract high-quality segmented representations of each license plate character can really hurt our classification performance when we go to recognize each of the characters later in this module.Due to the limitations of basic thresholding, we instead apply adaptive thresholding, which gives us much better results. Cnts = cnts 0 if imutils. Iscv2 ( ) else cnts 1 We start looping over each of thelabels on Line 125. If thelabel is 0, then we know thelabel corresponds to the background of the license plate, so we can safely ignore it.Note: In versions ofscikit - image = 0.12.X), the background label is 0. Make sure you check which version ofscikit - image you are using and update the code to use the correct background label as this can affect the output of the script.Otherwise, we allocate memory for thelabelMask on Line 132 and draw all pixels with the currentlabel value as white on a black background on Line 133.

By performing this masking, we are revealing only pixels that are part of the current connected component. An example of such alabelMask can be seen below. DrawContours ( charCandidateshull , - 1, 255, - 1 )First, a check is made on Line 138 to ensure that at least one contour was found in thelabelMask.

If so, we grab the largest contour (according to the ) and compute its bounding box.Based on the bounding box of the largest contour, we are now ready to compute a few more contour properties. The first is theaspectRatio, or simply the ratio of the bounding box width to the bounding box height. We’ll also compute thesolidity of the contour (refer to for more information on solidity). Last, we’ll also compute theheightRatio, or simply the ratio of the bounding box height to the license plate height.

Large values ofheightRatio indicate that the height of the (potential) character is similar to the license plate itself (and thus a likely character).You’ve heard me say it many times inside this course, but I’ll say it again — a clever use of contour properties can often beat out more advanced computer vision techniques.The same is true for this lesson.On Lines 151-153, we apply tests to determine if our aspect ratio, solidity, and height ratio are within acceptable bounds. We want ouraspectRatio to be at most square, ideally taller rather than wide since most characters are taller than they are wide. We want oursolidity to be reasonably large, otherwise we could be investigating “noise”, such as dirt, bolts, etc.

On the license plate. Finally, we want ourkeepHeight ratio to be just the right size — license plate characters should span the majority of the height of a license plate, hence we specify a range that will catch all characters present on the license plates.It’s important to note that these values were experimentally tuned based on our license plate dataset. For other license plate datasets these values may need to be changed — and that’s totally okay. There is no “silver bullet” for ANPR; each system is geared towards solving a very particular problem. When you go to develop your own ANPR systems, be sure to pay attention to these contour property rules. It may be the case that you need to experiment with them to determine appropriate values.Provided that our aspect ratio, solidity, and height ratio tests pass, we take the contour, compute the convex hull (to ensure the entire bounding region of the character is included in the contour), and draw the convex hull on ourcharCandidates mask.

Here are a few examples of computing license plate character regions. # clear pixels that touch the borders of the character candidates mask and detect# contours in the candidates maskcharCandidates = segmentation.clearborder(charCandidates)# TODO:# There will be times when we detect more than the desired number of characters -# it would be wise to apply a method to 'prune' the unwanted characters# return the license plate region object containing the license plate, the thresholded# license plate, and the character candidatesreturn LicensePlate(success=True, plate=plate, thresh=thresh,candidates=charCandidates). Candidates = charCandidates )On Line 164, we make a call toclearborder. Theclearborder function performs a connected component analysis, and any components that are “touching” the borders of the image are removed. This function is very useful in the oft case our contour property tests has a false-positive and accidentally marks a region as a character when it was really part of the license plate border.You’ll also notice that I have placed aTODO stub on Lines 166-168. Despite our best efforts, there will still be cases when our contour property tests are just not enough and we accidentally mark regions of the license plate characters when in reality they are not. Below are a few examples of such a false classification. Figure 9: Examples of connected-components (i.e.

Extraneous parts of the license plate) that were falsely labeled as characters.So how might we go about getting rid of these regions? The answer is to apply character pruning where we loop over the character candidates and remove the “outliers” from the group. We’ll cover the character pruning stage in our next lesson.Remember, each and every stage of your computer vision pipeline (ANPR or not) does not have to be 100% perfect. Instead, it can make a few mistakes and let them pass through — we’ll just build in traps and mechanisms to catch these mistakes later on in the pipeline when they are (ideally) easier to identify!Finally, we construct aLicensePlate object ( Line 172) consisting of the license plate, thresholded license plate, and license plate characters and return it to the calling function. Updating the driver scriptNow that we have updated ourLicensePlateDetector class, let’s also update ourrecognize. Py driver script, so we can see our results in action. DestroyAllWindows ( )Compared to our, not much has changed.

We are still detecting the license plates on Line 27. And we are still looping over each of the license plate regions on Line 30; however, this time we are looping over a 2-tuple: theLicensePlate object (i.e.namedtuple ) and thelpBox (i.e. License plate bounding box).For each of the license plate candidates, we construct an output image containing the (1) perspective transformed license plate region, (2) the binarized license plate region obtained using adaptive thresholding, and (3) the license plate character candidates after computing the convex hull ( Lines 36-39).To see our script in action, just execute the following command. Figure 12: Our ANPR pipeline is once again able to locate the license plate region in the image and segment the foreground characters from the background license plate. SummaryWe started this lesson by building on our previous knowledge of.

We extended our license plate localizer to segment license plate characters from the license plate itself using image processing techniques, including adaptive thresholding, connected component analysis, and contour properties.In most cases, our character segmentation algorithm worked quite well; however, there were some cases where our aspect ratio, solidity, and height ratio tests generated false positives, leaving us with falsely detected characters.As we’ll see in our next lesson, these false positives are not as big of a deal as they may seem. Using the geometry of the license plate (i.e. The license plate characters should appear in a single row), it will actually be fairly easy for us to prune out these false positives and be left with only the actual license plate characters.Remember, every stage of your computer vision pipeline does not have to be 100% correct every single time. Instead, it may be beneficial to pass on less-than-perfect results to the next stage of your pipeline where you can detect these problem regions and throw them away.For example, consider the lesson where we accidentally detected a few regions of an image that weren’t actually license plates. Instead of agonizing over these false positives and trying to make the morphological operations and contour properties work perfectly, it’s instead easier to pass them on to the next stage of our ANPR pipeline where we perform character detection. If we cannot detect characters on these license plate candidates, we just throw them out — problem solved!Solving computer vision problems is only partially based on your knowledge of the subject — the other half comes from your creativity, and your ability to look at a problem differently and arrive at an innovative (yet simple) solution.