Catching the thief – sensitivity and large numbers
Imagine, you are a shop owner and you just installed and electronic theft detection system. The system has a 99% accuracy of detecting theft. The alarm goes off, how likely is it, that the person who just passed is a thief?
It’s tempting to answer that there is a 99% chance that this person stole something. But actually, that isn’t necessarily the case.
In your store you’ll have honest customers and shoplifters. However, the honest customers outnumber the thiefs:: there are 10,000 honest customers and just 1 thief. If all of them pass in front of your alarm, the alarm will sound 101 times. 1% of the time, it will mistakenly identify a honest customer as a thief – so it will sound 100 times. 99% of the time, it will correctly recognise that a shoplifter is a shoplifter. So it will probably sound once when your thief does walk past. But of the 101 times it sounds, only 1 time will there actually be a shoplifter in your store. So the chance that a person is actually a thief when it sounds is just below 1% (0.99%, if you want to be picky).
Overestimating the probability if something is reported positive in such a scenario is called the base rate fallacy. This explains why airport searches and other methods of mass screening always will turn up lots of false positives.
