OpenCV speed traffic sign detection

Question

I have a problem detecting speed traffic signs with opencv 2.4 for Android. I do the following:

"capture frame -> convert it to HSV -> extract red areas -> detect signs with ellipse detection"

So far ellipse detection works perfect as long as picture is good quality. But as you see in pictures bellow, that red extraction does not work OK, because of poor quality of picture frames, by my opinion.

Converting original image to HSV:

Imgproc.cvtColor(this.source, this.source, Imgproc.COLOR_RGB2HSV, 3);

Extracting red colors:

Core.inRange(this.source, new Scalar(this.h,this.s,this.v), new Scalar(230,180,180), this.source);

So my question is is there another way of detecting traffic sign like this or extracting red areas out of it, which by the way can be very faint like in last picture ?

This is the original image:

This is converted to HSV, as you can see red areas look the same color as nearby trees. Thats how I'm suppose to know it's red but I can't.

Converted to HSV:

This is with red colors extracted. If colors would be correct I should get almost perfect circle/ellipse around sign, but it is incomplet due to false colors.

Result after extraction:

Ellipse method:

private void findEllipses(Mat input){
Mat thresholdOutput = new Mat();
int thresh = 150;

List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
MatOfInt4 hierarchy = new MatOfInt4();

Imgproc.threshold(source, thresholdOutput, thresh, 255, Imgproc.THRESH_BINARY);
//Imgproc.Canny(source, thresholdOutput, 50, 180);
Imgproc.findContours(source, contours, hierarchy, Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
RotatedRect minEllipse[] = new RotatedRect[contours.size()];

for(int i=0; i<contours.size();i++){
    MatOfPoint2f temp=new MatOfPoint2f(contours.get(i).toArray());

    if(temp.size().height > minEllipseSize && temp.size().height < maxEllipseSize){
        double a = Imgproc.fitEllipse(temp).size.height;
        double b = Imgproc.fitEllipse(temp).size.width;
        if(Math.abs(a - b) < 10)
            minEllipse[i] = Imgproc.fitEllipse(temp);
    }
}
detectedObjects.clear();
for( int i = 0; i< contours.size(); i++ ){
    Scalar color = new Scalar(180, 255, 180);
    if(minEllipse[i] != null){
        detectedObjects.add(new DetectedObject(minEllipse[i].center));
        DetectedObject detectedObj = new DetectedObject(minEllipse[i].center);
        Core.ellipse(source, minEllipse[i], color, 2, 8);
    }
}

}

Problematic sign:

Answer 1

You can find a review of traffic signs detection methods here and here.

You'll see that there are 2 ways you can achieve this:

1. Color-based (like what you're doing now)
2. Shape-based

In my experience, I found that shape-based methods works pretty good, because the color may change a lot under different lighting conditions, camera quality, etc.

Since you need to detect speed traffic signs, which I assume are always circular, you can use an ellipse detector to find all circular objects in your image, and then apply some validation to determine if it's a traffic sign or not.

Why ellipse detection?

Well, since you're looking for perspective distorted circles, you are in fact looking for ellipses. Real-time ellipse detection is an interesting (although limited) research topic. I'll point you out to 2 papers with C++ source code available (which you can use in you app through native JNI calls):

1. L. Libuda, I. Grothues, K.-F. Kraiss, Ellipse detection in digital image data using geometric features, in: J. Braz, A. Ranchordas, H. Arajo, J. Jorge (Eds.), Advances in Computer Graphics and Computer Vision, volume 4 of Communications in Computer and Information Science, Springer Berlin Heidelberg, 2007, pp. 229-239. link, code

2. M. Fornaciari, A. Prati, R. Cucchiara, "A fast and effective ellipse detector for embedded vision applications", Pattern Recognition, 2014 link, code

---

UPDATE

I tried the method 2) without any preprocessing. You can see that at least the sign with the red border is detected very good:

Answer 2

Referencing to your text:

This is converted to HSV, as you can see red areas look the same color as nearby trees. Thats how I'm suppose to know it's red but I can't.

I want to show you my result of basically what you did (simple operations should be easily transferable to android openCV):

    // convert to HSV
    cv::Mat hsv;
    cv::cvtColor(input,hsv,CV_BGR2HSV);

    std::vector<cv::Mat> channels;
    cv::split(hsv,channels);

    // opencv = hue values are divided by 2 to fit 8 bit range
    float red1 = 25/2.0f;
    // red has one part at the beginning and one part at the end of the range (I assume 0° to 25° and 335° to 360°)
    float red2 = (360-25)/2.0f;

    // compute both thresholds
    cv::Mat thres1 = channels[0] < red1;
    cv::Mat thres2 = channels[0] > red2;

    // choose some minimum saturation
    cv::Mat saturationThres = channels[1] > 50;

    // combine the results
    cv::Mat redMask = (thres1 | thres2) & saturationThres;

    // display result
    cv::imshow("red", redMask);

These are my results:

From your result, please mind that findContours alters the input image, so maybe you extracted the ellipse but just don't see it in the image anymore, if you saved the image AFTER findContours.

Answer 3

private void findEllipses(Mat input){
    Mat thresholdOutput = new Mat();
    int thresh = 150;

    List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
    MatOfInt4 hierarchy = new MatOfInt4();

    Imgproc.threshold(source, thresholdOutput, thresh, 255, Imgproc.THRESH_BINARY);
    //Imgproc.Canny(source, thresholdOutput, 50, 180);
    Imgproc.findContours(source, contours, hierarchy, Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
//      source = thresholdOutput;
    RotatedRect minEllipse[] = new RotatedRect[contours.size()];

    for(int i=0; i<contours.size();i++){
        MatOfPoint2f temp=new MatOfPoint2f(contours.get(i).toArray());

        if(temp.size().height > minEllipseSize && temp.size().height < maxEllipseSize){
            double a = Imgproc.fitEllipse(temp).size.height;
            double b = Imgproc.fitEllipse(temp).size.width;
            if(Math.abs(a - b) < 10)
                minEllipse[i] = Imgproc.fitEllipse(temp);
        }
    }

    detectedObjects.clear();
    for( int i = 0; i< contours.size(); i++ ){
        Scalar color = new Scalar(180, 255, 180);
        if(minEllipse[i] != null){
            detectedObjects.add(new DetectedObject(minEllipse[i].center));
            DetectedObject detectedObj = new DetectedObject(minEllipse[i].center);
            Core.ellipse(source, minEllipse[i], color, 2, 8);
        }
    }
}

Answer 4

have you tried using opencv ORB? it works really well. I created a haar cascade for a traffic sign (roundabout in my case) and used opencv ORB to match features and remove any false positives. For image recognition used Google's tensorflow and results were spectacular.