Train and test bag of visual word using VGG descriptors in OpenCV

Question

I had a code which is getting ORB keypoints and descriptors of images and then it trains a bag of word using "BOWKMeansTrainer". the purpose of this code is to cluster photos using k-means. I don't have any problem using ORB.

Here is my problem:

I saw that openCV provides a VGG, You can find it HERE. I want to use VGG descriptor instead of ORB. Based on the documents, VGG doesn't have any function to make keypoints, so I used ORB keypoint and then used ORB keypoints to create VGG descriptors.

Ptr<ORB> detector = ORB::create();
Ptr<cv::xfeatures2d::VGG> detectorVGG = cv::xfeatures2d::VGG::create(VGG::VGG_120, 1.4f, true, true, 0.75f, true);

detector->detect(grayFrame, kp);
detectorVGG->compute(grayFrame, kp, descriptors);

This one also works and I can easily train bag of word and cluster them similar to the way I did using ORB. However when I want to test the bag of words, I'm getting this error:

error: (-215) _queryDescriptors.type() == trainDescType in function knnMatchImpl   Aborttrap: 6

I tried to change the type of query descriptor Mat and descriptor Mat but I cannot solve it. In the best case , the error is changed to this:

(-215) (type == CV_8U && dtype == CV_32S) || dtype == CV_32F in function batchDistance

I am not sure if the problem is a small mistake in types or it is because of using ORB keypoints and VGG descriptors at the same time.

Part of my code is here:

vector<Mat> makeDescriptor(vector<Mat> frames)
    {
        int frameSize = frames.size();
        vector<Mat> characteristics;
        for (int i = 0 ; i < frameSize ; i++)
        {
            vector<KeyPoint> kp;
            Mat descriptors;
            Ptr<cv::xfeatures2d::VGG> detectorVGG = cv::xfeatures2d::VGG::create();
            Ptr<ORB> detectorORB = ORB::create();
            Mat frame = frames[i];
            if (frame.empty()) {
                cout << "Frame empty"<< "\n";
                continue;
            }
            Mat grayFrame;
            try
            {
                cvtColor(frame,grayFrame,CV_RGB2GRAY);
            }
            catch(Exception& e) 
            {
                continue;
            }

            detectorORB->detect(grayFrame,kp);
            detectorVGG->compute(grayFrame,kp,descriptors);
            characteristics.push_back(descriptors);
            }
            return characteristics;
        }

Mat makeDict(vector<Mat> characteristics, int _nCluster)
            {
                BOWKMeansTrainer bow(_nCluster);
                for (int j = 0 ; j < characteristics.size() ; j++)
                {
                    Mat descr = characteristics[j];
                    if (!descr.empty())
                    {
                        bow.add(descr);
                    }
                }
                Mat voc = bow.cluster();    
                return voc;
            }
BOWImgDescriptorExtractor makeBow(Mat dict) 
{

    Ptr<DescriptorMatcher> matcher = DescriptorMatcher::create("BruteForce-Hamming");
    Ptr<DescriptorExtractor> extractor = VGG::create();
    BOWImgDescriptorExtractor bowDE(extractor,matcher);
    Mat voc;
    dict.convertTo(voc,CV_32F);
    bowDE.setVocabulary(voc);
    return bowDE;
}

void testBow(BOWImgDescriptorExtractor bowDE, vector<Mat> frames)
{
    Mat features;
    vector< vector<KeyPoint> > keypoints = makeORBKeyPoints(frames);
    for (int i = 0 ; i < frames.size() ; i++) 
    {
        Mat bowDesc;

        if (!keypoints[i].empty())
        {
            cout << "inside if" << '\n';
            Mat frame;
            frames[i].convertTo(frame, CV_8U);
            bowDE.compute(frame, keypoints[i], bowDesc); //This Line
            bowDesc.convertTo(bowDesc,CV_32F);
        }
        features.push_back(bowDesc);
    }
}

The runtime error is happening at this line:

bowDE.compute(frame, keypoints[i], bowDesc); //This Line

I appreciate if someone helps me out to find a solution for it.

Thanks,

Train and test bag of visual word using VGG descriptors in OpenCV

0 Answers0