Test your Statistical Sense

Think you're smarter than a doctor? If you get this right, you are way ahead of the curve when it comes to medically relevant statistics.

Toxicologist Dave Cragin sent me this question after we had been discussing risk, probabilities and the controversy over PSA testing for prostate cancer.

It's a question he poses to his students. I was relieved to find I'd gotten it right, since it's part of my job to communicate health information. According to an article in Science, 23 out of 24 doctors get it wrong:

For the Hemoccult-Test for colon cancer:  The probability that a 50-year-old man without symptoms has colon cancer is 0.3%. If he has cancer, the probability of him having a positive test is 50%. If he does not have cancer, the probability of him receiving a positive test result anyway is 3%.

What is the probability that a patient has colon cancer based on a positive Hemoccult test?

Dr. Cragin got this question from the journal Science. Here's the top part of the article, which explains how to figure out the problem. The article also proposes a better way to talk about statistics so that doctors have an easier time grasping them.

Science, Vol 290, Issue 5500, 2261-2262 , 22 December 2000

Communicating Statistical Information

Ulrich Hoffrage,* Samuel Lindsey, Ralph Hertwig, Gerd Gigerenzer

Decisions based on statistical information can mean the difference between life and death--for instance, when a cancer patient has to decide whether to undergo a painful medical procedure based on the likelihood that it will succeed, or when a jury has to decide whether to convict someone based on DNA evidence. Unfortunately, most of us, experts included, have difficulty understanding and combining statistical information effectively.

For example, faculty, staff, and students at Harvard Medical School were asked to estimate the probability of a disease given the following information (1): "If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5 per cent, what is the chance that a person found to have a positive result actually has the disease, assuming that you know nothing about the person's symptoms or signs?" The estimates varied wildly, ranging from the most frequent estimate, 95% (given by 27 out of 60 participants), to the correct answer, 2% (given by 11 out of 60 participants) (2). In a study requiring interpretation of mammography outcomes (3), almost all physicians confused the sensitivity of the test (the proportion of positive test results among people with the disease) with its positive predictive value (the proportion of people with the disease among those who receive a positive test result). This is a common confusion that even crops up in scholarly articles (3) and statistical textbooks (4) and certainly affects the ability of lay people (5) to understand the statistical information. Recent discussions of genetic testing have indicated that genetic counselors are experiencing the same difficulty (6).

It makes little mathematical difference whether statistics are expressed as probabilities, percentages, or absolute frequencies. It does, however, make a psychological difference. More specifically, statistics expressed as natural frequencies improve the statistical thinking of experts and nonexperts alike.

Natural Frequencies

To illustrate how natural frequencies differ from probabilities, we use the example of a cancer screening test. The probability of colorectal cancer can be given as 0.3% [base rate]. If a person has colorectal cancer, the probability that the hemoccult test is positive is 50% [sensitivity]. If a person does not have colorectal cancer, the probability that he still tests positive is 3% [false-positive rate]. What is the probability that a person who tests positive actually has colorectal cancer? A restatement of the same problem in terms of natural frequencies would be that out of every 10,000 people, 30 have colorectal cancer. Of these, 15 will have a positive hemoccult test. Out of the remaining 9970 people without colorectal cancer, 300 will still test positive. How many of those who test positive actually have colorectal cancer?

Only 1 out of 24 physicians gave the correct answer when the statistical information was expressed in probabilities (7). When it was presented in natural frequencies, 16 out of 24 other physicians gave the correct answer: 15 out of 315 (i.e., 5%). Whereas natural frequencies seem to help people make statistical inferences, probabilities apparently hinder them. Unfortunately, in contexts in which the positive predictive value of a test is at issue, statistics are typically expressed and communicated in the form of probabilities, although they can easily be translated into natural frequencies, as follows: