The following list of Rs and Ns represents relevant (R) and nonrelevant (N) retu
ID: 3753919 • Letter: T
Question
The following list of Rs and Ns represents relevant (R) and nonrelevant (N) returned documents in a ranked list of 20 documents retrieved in response to a query from a collection of 10,000 documents. The top of the ranked list is on the left of the list. This list shows 6 relevant documents. Assume that there are 8 relevant documents in total in the collection RRN N N NNNR N RNN N R N N N N R (Note that spaces above are just added to make the list easier to read) (1) What is the precision of the system on the top-20? (2) What is the Fi on the top-20? (3) What is/are the uninterpolated precision(s) of the system at 25% recall? (4) What is the interpolated precision at 33% recall? (5) Assume that these 20 documents are the complete result set of the system. What is the MAP for the query? Assume, now, instead, that the system returned the entire 10,000 documents in a ranked list, and these are the first 20 results returned (6) What is the largest possible MAP that this system could have? (7) What is the smallest possible MAP that this system could have? (8) In a set of experiments, only the top-20 results are evaluated by hand. The result in (5) is used to approximate the range (6) to (7). For this example, how large (in absolute terms) can the error for the MAP be by calculating (5) instead of (6) and (7) for this query?Explanation / Answer
Solution:
(1)Precision of the system of top 20 = 6/20 = 0.3
(2) Now, Recall = 6/8 = 0.75
And, F1=2PR/P+R = 3/7 =0.43
(3) Here 8 * 25/100=2
Uninterpolated precision could be 1, 2/3,2/4, 2/5 , 2/6, 2/7, 1/4.
(4)The highest precision found in any recall larger than 33% is 4/11.
Hence the interpolated precision at 33% recall is 4/11 = 0.364
(5)MAP for the 20 document set is = 1/6*(1+1+3/9+4/11+5/15+6/20) =0.5555
(6)MAP((largest) =1/8*(1+1+3/9+4/11+5/15+6/20+7/21+8/22)=0.503
(7) MAP (smallest) = 1/8*(1+1+3/9+4/11+5/15+6/20+7/9999+8/10000)=0.417
(8) The error is = in (6) part = 0.555-0.503=0.052
in (7) part 0.555-0.417= 0.138
Hence the errors are (0.052 and 0.138)
I hope this helps if you find any problem. Please comment below. Don't forget to give a thumbs up if you liked it. :)
Related Questions
Navigate
Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.