The base rate fallacy poses a significant challenge to the effectiveness of predictive algorithms, especially when these systems are deployed in contexts with extremely low prevalence rates, such as terrorism identification. This cognitive bias, where general prevalence is overlooked in favor of specific case details, can lead to an overwhelming number of false positives, rendering even highly accurate systems practically unworkable. The inherent
rarity of certain events means that the sheer volume of true negatives can easily generate more false alarms than actual detections, creating a paradox where accuracy at an individual level does not translate to effective overall identification.
The Terrorism Identification Paradox
Consider a hypothetical city of one million inhabitants, with 100 terrorists and 999,900 non-terrorists. The base rate probability of a randomly selected inhabitant being a terrorist is a mere 0.0001. Now, imagine an alarm system with a surveillance camera and facial recognition software designed to catch terrorists. This software has a 1% false negative rate (it misses a terrorist 1% of the time) and a 1% false positive rate (it falsely flags a non-terrorist 1% of the time). If an inhabitant triggers the alarm, someone committing the base rate fallacy might infer a 99% probability that the detected person is a terrorist, based on the software's high accuracy. However, this reasoning is flawed.
To illustrate, if the entire city population passes before the camera, approximately 99 of the 100 terrorists will trigger the alarm. Crucially, about 9,999 of the 999,900 non-terrorists will also trigger the alarm due to the 1% false positive rate. This means roughly 10,098 people will trigger the alarm in total, but only about 99 of them will actually be terrorists. Consequently, the probability that a person triggering the alarm is actually a terrorist is only about 99 in 10,098, which is less than 1%. This is a stark contrast to the initial guess of 99% and highlights how misleading the base rate fallacy can be when the number of non-terrorists is vastly greater than the number of terrorists.
Challenges for Data Mining and Predictive Models
Multiple experts have argued that, given the extremely low base rate of terrorism, using data mining and predictive algorithms to identify terrorists cannot feasibly work due to this false positive paradox. Estimates for the number of false positives for each accurate result vary widely, from over ten thousand to as high as one billion. Such a high ratio of false to true positives means that investigating every lead would be prohibitively expensive and time-consuming, making the system impractical.
The level of accuracy required to make these models viable is likely unachievable. Furthermore, the low base rate of terrorism also implies a scarcity of data, which hinders the development of accurate algorithms in the first place. In the context of terrorism detection, false negatives are highly undesirable and must be minimized. However, reducing false negatives typically requires increasing sensitivity, which in turn leads to an increase in false positives. This trade-off further complicates the development of effective systems. It is also questionable whether the use of such models by law enforcement would meet the necessary burden of proof, considering that over 99% of the results could be false positives.
Amplification in Multi-Attribute Screening
A distinct mechanism can amplify this effect in multi-attribute screening systems. While matching 15 specific, pre-determined attributes might have an extremely low probability (e.g., one in 30 decillion), systems that flag individuals matching any 15 out of 1,000 attributes can have per-person false alert probabilities around 10^-4. This represents a difference of 31 orders of magnitude, arising from the combinatorics of threshold rules rather than solely from low base rates. In a city of one million, such a system could produce approximately 226 false alerts, with the probability of zero false alerts being roughly 10^-99. These systems exhibit sharp phase transitions at critical population sizes, beyond which failure becomes certain and cannot be prevented through simple threshold adjustments. This underscores the profound challenges of applying predictive analytics to rare events without a thorough understanding of base rates and the potential for false positives.
