OpenCV real-time stereocam image processing problems (post-calibration)

Question

I have built my own stereocamera from two USB cameras (with an annoying autofocus). I have calibrated them using /opencv/samples/cpp/stereo_calib.cpp', which produced an extrinsics/intrinsics file with an RMS Error of 0.4818 and an average reprojection error of 0.5426

I am now trying to run realtime depth mapping using Ptr<StereoBM> bm in openCV. I am getting useless images and I'm wondering if anyone can point me in the right direction of where to look. I'm also wondering if the autofocusing of the cameras every so often can change the focal length and therefore change the calibration, altering the results.

Code and pictures below.

Bonus if anyone can recommend any more robust methods than StereoBM for matching in openCV :)

Image of checkerboard calibration:

Left camera image:

Post StereoBM Processing:

Snippet of Code:

//Set up stereoBM
    Ptr<StereoBM> bm = StereoBM::create(16,9);

    //Read intrinsice parameters
    string intrinsic_filepath = "/home/will/Desktop/repos3.0/stereo-vision/Stereo-Vision/intrinsics.yml";
    FileStorage fs(intrinsic_filepath, FileStorage::READ);
    if(!fs.isOpened())
    {
        printf("Failed to open intrinsics.yml");
        return -1;
    }
    Mat M1, D1, M2, D2;
    fs["M1"] >> M1;
    fs["D1"] >> D1;
    fs["M2"] >> M2;
    fs["D2"] >> D2;

    //Read Extrinsic Parameters
    string extrinsic_filepath = "/home/will/Desktop/repos3.0/stereo-vision/Stereo-Vision/extrinsics.yml";
    fs.open(extrinsic_filepath, FileStorage::READ);
    if(!fs.isOpened())
    {
        printf("Failed to open extrinsics");
        return -1;
    }

    Mat R, T, R1, P1, R2, P2;
    fs["R"] >> R;
    fs["T"] >> T;

    Mat frame1,frame2, gray1, gray2;
    int counter = 0;

    capture >> frame1;
    capture >> frame2;

    Size img_size = frame1.size();
    Rect roi1, roi2;
    Mat Q;

    stereoRectify( M1, D1, M2, D2, img_size, R, T, R1, R2, P1, P2, Q, CALIB_ZERO_DISPARITY, -1, img_size, &roi1, &roi2 );

    Mat map11, map12, map21, map22;
    initUndistortRectifyMap(M1, D1, R1, P1, img_size, CV_16SC2, map11, map12);
    initUndistortRectifyMap(M2, D2, R2, P2, img_size, CV_16SC2, map21, map22);

    int numberOfDisparities = 16;
    int SADWindowSize = 9;

    bm->setROI1(roi1);
    bm->setROI2(roi2);
    bm->setPreFilterCap(31);
    bm->setBlockSize(SADWindowSize);
    bm->setMinDisparity(0);
    bm->setNumDisparities(numberOfDisparities);
    bm->setTextureThreshold(10);
    bm->setUniquenessRatio(15);
    bm->setSpeckleWindowSize(100);
    bm->setSpeckleRange(32);
    bm->setDisp12MaxDiff(1);

    while(1){
        capture >> frame1;
        capture2 >> frame2;
        imshow("Cam1", frame1);
        imshow("Cam2", frame2);

        /************************* STEREO ***********************/

        cvtColor(frame1, gray1, CV_RGB2GRAY);
        cvtColor(frame2, gray2, CV_RGB2GRAY);

        int64 t = getTickCount();

        Mat img1r, img2r;
        remap(gray1, img1r, map11, map12, INTER_LINEAR);
        remap(gray2, img2r, map21, map22, INTER_LINEAR);

        Mat disp, disp8;
        bm->compute(img1r, img2r, disp);
        t = getTickCount() - t;
        printf("Time elapsed: %fms\n", t*1000/getTickFrequency());

        disp.convertTo(disp8, CV_8U, 255/(numberOfDisparities*16.));
        imshow("disparity", disp8);
    }

Autofocus will change the calibration of the cameras - you need to disable it. — Chris K, Jan 14 '16 at 22:44

score 1 · Accepted Answer · edited May 23 '17 at 12:04

1

So, half a pixel of RMS error means your calibration is basically garbage.

In your calibration image I noticed that the target is not even flat. If I can see that, the camera can too, but the calibration model will still assume it's flat, and that will make the optimizer very sad.

Suggest you take a look at this answer about calibration.

edited May 23 '17 at 12:04

Community

1
1

answered Jan 15 '16 at 02:49

Francesco Callari

11,300
2
25
40

That is a fantastic post, I wish it was more visible and easy to find. I'm not sure if this is your area of expertise, but do you know of a surefire way to turn off autofocus on usb cameras? I am running ubuntu 14.04. I have not been able to find a way to turn it off through openCV, (no luck through VideoCapture::set) I have also had no luck manually through uvcdynctrl. I'm running with two of these guys: http://www.amazon.com/gp/product/B00VLXV4FU/ref=s9_simh_gw_g147_i3_r?pf_rd_m=ATVPDKIKX0DER&pf_rd_s=desktop-1&pf_rd_r=1H9FYP1AVBSBF4XQ7T8Y&pf_rd_t=36701&pf_rd_p=2079475242&pf_rd_i=desktop – wprins Jan 15 '16 at 21:10
Not specifically, but I'd start by googling the camera board serial id (or all the chip's ids), and hope a spec sheet (or better, an application programmer manual) shows up, and hope there is a focus control/on-off input, then I'd pray the Cheap Camera goddess that the designer of this particular board had the wisdom to route it to the USB controller, finally, I'd read very carefully the USB driver docs to see how that signal can be asserted. Needless to say, your chances of getting screwed along this chain increase at each step: there is a reason why CV-dedicated cameras cost money :-( – Francesco Callari Jan 15 '16 at 21:17

OpenCV real-time stereocam image processing problems (post-calibration)

1 Answers1