11 <strong>Topic X. Types of Errors and their Costs</strong>
Topic X. Types of Errors and their Costs
- Considering the relative costs of each possible mistake helps us make better decisions under conditions of uncertainty, when we cannot eliminate the possibility of a mistake either way.
- It is often necessary to make decisions or judgments under conditions of uncertainty. When this happens, two kinds of errors are possible; we might think that something is present when it is not (a Type 1, or false positive, error), or we might think that something is absent when it is present (a Type 2, or false negative error). In different contexts, these two types of errors may come with different costs. When one kind of error is worse than the other, it is prudent to err on the side of making the less bad error. Sometimes it even makes sense to make one kind of error quite a lot in order to avoid making the other kind of error. For example, even though the large majority of tumors are benign, it makes sense to get tumors biopsied because if you do have a cancerous tumor and assume it is benign (a false negative), it can kill you.
- Addressing the Question: How confident should we be?
- False Positive Errors
- False Negative Errors
- Balancing risks under conditions of uncertainty
- Exemplary Quotes
- "The oncoming asteroid has only a 1% chance of hitting Earth. But if it does, life on Earth will be destroyed. It'll be expensive to stop the asteroid, but the risk is bad enough it's worth it."
- "It's true that sometimes seatbelts cause deaths, when people get stuck in them and can't get out. But they more often save lives, so it is prudent to wear your seatbelt whenever you drive."
- "I asked them to imagine that they faced a choice between two types of radiation therapy for early-stage breast cancer. The first treatment would leave them with a 15% chance of local recurrence and a 10% chance of moderate or severe breast fibrosis. The second treatment would leave them with only an 8% chance of local recurrence but a 30% chance of moderate or severe fibrosis. The radiation oncologists raised their hands in almost equal numbers for the two treatments. Some believed the higher risk of fibrosis was unacceptable, given the treatability of most local recurrences, whereas others believed the trauma of recurrence outweighed the discomfort of fibrosis. But sometimes physicians' values differ in important ways from those of many patients. When such value judgments are incorporated into professional treatment guidelines, without any explicit acknowledgment that a reasonable patient might choose an alternative course of treatment, they take potential choices away from patients." https://www.nejm.org/doi/full/10.1056/NEJMp1504245
- Cautionary Quotes: Mistakes, Misconceptions, & Misunderstandings
- The commonest error for these concepts is simply failing to notice that they are relevant considerations. When cued, students tend to be good at applying them.
- "We don't have to prepare for the hurricane they're forecasting might hit, because most of the time they say a hurricane might come, it turns out to be not that bad."
- A. ATTITUDES
- Given some degree of uncertainty, appreciate that different kinds of errors come with different costs, such that in some cases it is worthwhile to presume the less likely alternative because the error you risk is less costly.
- B. CONCEPT ACQUISITION
- False Positive/Type I Errors: A test yields a positive result, but in fact the condition is not present.
- False Negative/Type II Errors: A test yields a negative result, but in fact the condition is present.
- There is always the possibility of a trade-off—for a given test, one can reduce the risk of false positives by increasing the risk of false negatives, and vice-versa.
- Good decision-making under uncertainty involves having sufficient signal (an adequate test) and setting your threshold appropriately for the relative costs of false positives and false negatives.
- In some classification cases like pornography identification or graduate school admissions, there may not be a “truth of the matter” so there aren’t true “false” positives or “true” negatives, although a threshold must still be set.
- C. CONCEPT APPLICATION
- Identify false positive (type I) and false negative (type II) errors in scientific and everyday situations.
- Weigh the costs associated with false positives/negatives with the benefits associated with true positives/negatives when making a decision under uncertain conditions.
- Explain how people could come to different decisions or policies as a result of different utilities/values associated with different types of errors, even if they agree about the relevant facts.
- Suggested Readings & Reading Questions
- Clicker Questions
- Discussion Questions
- In the Stanley Kubrick film Dr Strangelove, which takes place during the Cold War, a rogue American general sends the code to send planes to drop nuclear bombs on Russia because he wants to start a nuclear war. One of the pilots believes that this could only have happened if Russia had struck first. Under these conditions (leaving aside the Doomsday Machine), what are the costs of a false positive (believing Russia struck first when it didn't) and what are the costs of a false positive (believing it is some kind of mistake, when in fact Russia did strike first)?
- Class Exercises
- Practice Problems
- Suppose you have very bad migraines. Your doctors tell you migraines can be sparked by a lot of different things, which vary for different people: most kinds of food, coffee, stress, and/or bright sunlight. You want to find out what is causing your migraines, so you cut your diet to potatoes and then slowly add one food back at a time, waiting a few days between each addition to see if you get a migraine. The whole thing is stressful, and you’re so hungry and grumpy you lose your sunglasses on Monday. On Thursday, you eat chocolate again for the first time. That afternoon, you get a relatively small migraine. You are very sad, because chocolate is your very favorite food.
- a.) Should you wait a few days and try eating a small amount of chocolate again, or should you ban it from your cupboards? Why?
- b.)What if your migraines came with sudden, intense, lasting vertigo, and you work in skyscraper construction and can’t take off work?
- Recount an instance in which you thought you made a right diagnosis of a situation, but then found out that you were wrong. Explain whether it was an instance of a false positive or a false negative. What makes it a false positive/negative as compared to a false negative/positive? Which type of error do you think would have been better to make in this situation, and why?
- Data Science Applications
- Some kind of risk analysis task