Needles and haystacks: Balancing precision and recall

Thursday, October 29, 2009, ZDNet

Commentary - These terms are bandied about in most every discussion about search. What do they really mean and how do we evaluate them? To those of us immersed in the thick of search lore, we have lots of answers but we certainly don't make them easy to understand to the uninitiated. There is quite a large wall around our little garden. Let's try to demystify the discussion a bit by focusing on the tradeoff that is inherent between the two concepts in context of how it applies to real world examples.

Let us begin by considering the problem of evaluating the accuracy of a spam detector for your email system. You test it on 1,000 known spam emails and it tags all but one as spam. Would you report that it is 99.9 percent accurate? No, because you still need to evaluate the performance of the detector on legitimate (non-spam) email. You perform the test on 1,000 legitimate emails and one is (incorrectly) detected as spam. The 99.9 percent figure then is accurate. But if 500 of them were falsely tagged, you certainly would not be happy with the detector's performance. What should its overall accuracy be?

Click here to read the full commentary article from Sid Probstein on ZDNet