23

According to authors in 1, 2, and 3, Recall is the percentage of relevant items selected out of all the relevant items in the repository, while Precision is the percentage of relevant items out of those items selected by the query.

Therefore, assuming user U gets a top-k recommended list of items, they would be something like:

Recall= (Relevant_Items_Recommended in top-k) / (Relevant_Items)

Precision= (Relevant_Items_Recommended in top-k) / (k_Items_Recommended)

Until that part everything is clear but I do not understand the difference between them and Recall rate@k. How would be the formula to compute recall rate@k?

user2314737
  • 27,088
  • 20
  • 102
  • 114
Luisa Hernández
  • 596
  • 1
  • 3
  • 17
  • 1
    You directly put @K in calculating simple Precision and Recall for rating prediction what may be confusing. The rule is simple - if You try to measure only RATING prediction - use simple Precision and Recall on the whole recommended result. If You are interested in measure RANKING prediction, then You are more interested how well let say top-5 performs (first recommendation carousel screen), then top-10 (second screen) and so on. Because in second case You are more interested in how well Your solution ordered the whole response - ranked results. – Bartłomiej Twardowski Nov 15 '15 at 10:26
  • Thank you so much @Bartłomiej Twardowski. So, just I was doing that rate@k already? – Luisa Hernández Nov 16 '15 at 11:25
  • Yes, if you are truncating to top k item and only calculating p/r on it. – Bartłomiej Twardowski Nov 16 '15 at 13:03

1 Answers1

19

Finally, I received an explanation from Prof. Yuri Malheiros (paper 1). Althougth recall rate@k as cited in papers cited in the questions seemed to be the normal recall metrics but applied into a top-k, they are not the same. This metric is also used in paper 2, paper 3 and paper 3

The recall rate@k is a percentage that depends on the tests made, i.e., the number of recommendations and each recommendation is a list of items, some items will be correct and some not. If we made 50 different recommendations, let us call it R (regardless of the number of items for each recommendation), to calculate the recall rate is necessary to look at each of the 50 recommendations. If, for each recommendation, at least one recommended item is correct, you can increment a value, in this case, let us call it N. In order to calculate the recall rate@R, it is neccesary to make the N/R.

Luisa Hernández
  • 596
  • 1
  • 3
  • 17
  • 12
    I think you made a mistake there, you described the precision@k again. Recall@k means you count the relevant documents among the top-k and divide it by the total number of relevant documents in the repository. See https://ils.unc.edu/courses/2013_spring/inls509_001/lectures/10-EvaluationMetrics.pdf – Chris Jul 12 '17 at 16:58
  • 2
    Agree with Chris. What you're describing sounds like mean precision@k. – halflings Nov 08 '18 at 04:25
  • @Chris you said that "Recall@k means you count the relevant documents among the top-k and divide it by the total number of relevant documents" - not by k? If I use k=1 and there are 20 documents relevant for a query, than the maximum recall@1 will be: 1/20 if it is found, otherwise 0/20. Does this makes sense? If think it should be 1/1 or 0/1. – user3352632 Apr 13 '22 at 07:35
  • 1
    @user3352632 Yeah you are right, my mistake. It makes more sense to divide by k. – Chris Oct 01 '22 at 14:20