Calculate precision and recall on WANG database

Question

I have made an CBIR system in MATLAB and have used similarity measurement as euclidean distance. Using this for each query image I retrieve top 20 images.

I have used WANG Dataset for testing my system.
It contains 10 classes(like African people, Buses, Roses etc.) each containing 100 images.(1000 images in total).

My Method:
1. I am using Correlogram, Co-occurence Matrix(CCM) and Difference between Pixel Scan Pattern(DBPSP) for constructing my vector(64+196+28=288 dimensions respectively).

Each of the 1000 db image I have its vector constructed beforehand.
Now a query image comes and I construct it's vector too(228 dimensions again).
I use Euclidean Distance for similarity and sort db image vectors in descending order of their euclid distance.
Top 20 results are shown.
In those 20 I can have TP or FP.

For a single query image I can easily calculate Precision and Recall and plot PR-curve using this link.

How can I do the same for whole class?

My Approach: For each image belonging to class A find top 20 images and it's respective TP(true positives) and FP (False Positive).

        TP   FP

Image1  17   3  
Image2  15   5  
...  
...  
Image100  10  10  
Total   1500 500

Precision of Class A =1500/(2000) = .75 (Is it right??)
Recall of Class A ---> Stuck ??
PR-Curve ----> Stuck ?? Some links said I need a classifier for that and some not... I am really confused.

Do you have the confusion matrix or do you only have the TP/FP? — krisdestruction, Apr 22 '15 at 23:36
Only TP and FP. Out of 20 images if the image belongs to the correct class I increase TP by one and if not I increase FP by one. How do I check that a image is of correct class is simple as Class A images are from 0-99 then Class B are from 100-199 and so on. So a simple if-else loop gives me the answer. — CoderBoy, Apr 22 '15 at 23:57
how about FN or the number of false? You NEED to have either to calculate recall — krisdestruction, Apr 23 '15 at 00:03

score 2 · Accepted Answer · edited Jan 18 '21 at 12:34

2

So as you noted, you can calculate precision as follows.

P = TP ./ ( TP + FP );

However, you NEED to have either have FN or the number of total falses to calculate recall. As discussed in chat, you need to find a way to determine your FN and FP data. Then you can use the following formula to calculate recall.

R = TP ./ ( TP + FN )

If you have the confusion matrix or data, you can use my custom confusionmat2f1.m to calculate precision, recall, and f1 score. This assumes that the confusion matrix is formatted as how Matlab defines it. An explanation of each line is inline. Please let me know if you want more clarification.

function [F,P,R] = confusionmat2f1( C )
    %% confusionmat2f1( C )
    %
    % Inputs
    % C - Confusion Matrix
    %
    % Outputs
    % F - F1 score column vector
    % P - Precision column vector
    % R - Recall column vector
    
    %% 
    
    % Confusion Matrix to Probability
    M = sum( C, 3 );
    
    % Calculate Precision
    P = diag(M) ./ sum(M,1)';
    
    % Calculate Recall
    R = diag(M) ./ sum(M,2);

    % Calculate F1 Score
    F = f1( P, R );

edited Jan 18 '21 at 12:34

Community

1
1

answered Apr 22 '15 at 23:59

krisdestruction

1,950
1
10
20

How do I calculate FN and TN? The only operation that I can do is take an image and find top 20 images then out of 20 images if the image belongs to the correct class I increase TP by one and if not I increase FP by one. Can't I do this without using classifier?If no, can you suggest some inbuilt in MATLAB? – CoderBoy Apr 23 '15 at 00:04
For FN, those are the ones that are classified as no, but are actually yes. For TN, those are the ones that are classified as no, and are actually no. You need 4 of the 6 variables (TN,FN,TP,FP,total true,total false) to calculate both precision and recall. You need to either rewrite your code to account for one of the variables but I suggest you create a confusion matrix instead. It makes analysis SO much easier :) – krisdestruction Apr 23 '15 at 00:08
It's not hard to either. As long as you know what the classes are classified as, and what the ground truth (what it's supposed to be) is, you can feed it into the `confusionmat` function. – krisdestruction Apr 23 '15 at 00:09
Also having the class predicted and class expected vectors will make debugging much much easier :D – krisdestruction Apr 23 '15 at 00:12
I am not using any classification. I am just finding euclidean distance between queryImageVector and all dbImageVector. Atlast I sort them in descending order and retrieve top 20. Is it impossible for me to calculate recall then? – CoderBoy Apr 23 '15 at 00:13
It should be possible, and I'm pretty sure what you're doing constitutes as classification. You have a negative class right? Perhaps you can describe what your overall project is and what you're doing with these images for me to put it in terms that will fit your problem. – krisdestruction Apr 23 '15 at 00:15
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/75985/discussion-between-coderboy-and-krisdestruction). – CoderBoy Apr 23 '15 at 00:25

Calculate precision and recall on WANG database

1 Answers1