Let us assume that there is a test for a certain disease that is very reliable in two senses. First, it is very sensitive: the probability of a positive test result, given that the person tested has the disease [P(+|D)], is 0.98. Second, it is very specific: p(-|ĀD) = 0.97.
How worried should a person be if s/he tests positive? In order to know this, we need to know the prior probability of disease, otherwise known as its "base rate" within the population. For example, let us assume that only one out of every 10,000 people has the disease: p(D) = 0.0001.
Using Bayes' theorem, we can calculate the posterior probability of disease, given a positive test result:
p(D|+) = [p(+|D)p(D)]/p(+)
= [p(+|D)p(D)]/{[p(+|D)p(D)] + [p(+|ĀD)p(ĀD)]}
= [(0.98)(0.0001)]/{[(0.98)(0.0001)] + [(1 - 0.97)(1 - 0.0001)]}
= 0.0033
Thus, even after testing positive, the chance that a person in this situation has the disease is still only about one-third of one percent.
The expanded form of Bayes' theorem (Line 2 of the four-line equation above) shows that in order to calculate the posterior probability of one hypothesis (in this case, the hypothesis that a certain person has a certain disease), we need to know the likelihood and prior probability of at least one other hypothesis (in this case, the hypothesis that s/he does not have the disease).
Hitchcock, C. and E.
Sober. 2004. Prediction vs. accommodation and the
risk of overfitting. British
Journal for the Philosophy of Science 55:1-34.
Mikkelson, G. M. 2006. Realism vs. instrumentalism in a new statistical
framework. Philosophy of
Science 73:440–447.