# 4 <strong>Topic IV. Systematic and Statistical Uncertainty<…

**Topic IV. Systematic and Statistical Uncertainty**

- This topic explores the sources of error and uncertainty in data.
- We examine sources of statistical uncertainty/error (which can be averaged down) and systematic uncertainty (which can’t). We also connect these concepts to related terminology (jargon) from other fields: precision vs. accuracy, variance vs. bias, and reproducibility vs. validity.
- Addressing the Question:
*How confident should we be?**Statistical Uncertainty**Systematic Uncertainty*

## TOPIC RESOURCES

## EXAMPLES

- Exemplary Quotes
- "It won't do us any good to average lots and lots of test subjects' heights together if our tape measure got shrunk in the wash!"
- "If we estimate the effect of a drug on weight by randomly assigning people to take the drug vs. not take it and then measuring their weight after a year, we could subtract the average weight loss of drug-takers vs. non-drug-takers to get the effect size of the drug on weight loss. But the people know if they're taking a drug for weight loss, so there could be a placebo effect creating a systematic bias. So the better way to do the experiment is to give the control group sugar pills. Then we can be more confident that any weight loss is due to the drug, and not a systematic bias created by the placebo effect."

- Cautionary Quotes: Mistakes, Misconceptions, & Misunderstandings
*Difficulty distinguishing degrees of statistical error or uncertainty*- "We can't say one instrument has more statistical uncertainty than the other, because everything has statistical uncertainty."

*Failing to Notice Possibility of Statistical Error*- "I know that everyone likes chocolate, because I asked three of my friends and all of them said they like chocolate."
- "Scientists ran a randomized controlled study with 30 people and four conditions, and the results were statistically significant, so we know the drug works."

*Failing to Notice Possibility of Systematic Error*- “There’s no way that X [fill in with name of candidate] could win the election! Everyone that I know is voting for Y [fill in with name of other candidate].”
- "We surveyed over a hundred thousand people, so our study is definitely an accurate picture of how Americans think about science."
- "We asked students to share if they were sexually active with a show of hands, and almost no one raised their hand. So we know that it's rare at our school."

## LEARNING GOALS

- A.
*ATTITUDES*- Often our contact with reality is mediated by measurement and quantification. We need to be aware that every measurement comes with some degree of uncertainty.

- B.
*CONCEPT ACQUISITION**Statistical Uncertainty/Error:*Differences between reality and our measurement on the basis of random imprecisions.- a. All measurements have a certain amount of variance, which are just differences between multiple measurements due to error and/or genuine variation in the sample. These differences will not all go in the same direction.
- b. Statistical uncertainty can be reduced by averaging a larger amount of data.

*Systematic Uncertainty/Error:*Differences between reality and our measurement that skew our results in one direction.- a. Such measurements will show a consistent bias, that is, a consistent deviation from reality in one direction.
- b. Systematic uncertainty cannot be reduced by averaging a larger amount of data.

- C.
*CONCEPT APPLICATION*- Identify sources of measurement uncertainty/error that introduce statistical uncertainty/error, that introduce systematic uncertainty/error, and that introduce both.
- Suggest approaches to reducing or constraining measurement uncertainty (both statistical and systematic).
- Recognize that the process of science involves creativity in identifying sources of systematic uncertainty and inventing strategies to reduce or eliminate them.

## CLASS ELEMENTS

- Suggested Readings
- Clicker Questions
- Sara is running a mile, which is a four-lap race around a track. Her coach wants to time how fast she runs on the third lap of the four-lap race. As Sara goes past the line on the track that marks the the start of the laps, her coach presses a button on a stopwatch to start a timer. When Sara goes past that line again, her coach presses the button again to stop the stopwatch. Which of the following will be a noticeable source of uncertainty in the coach’s measurement of how long it took Sara to run the lap?
- A. Primarily statistical uncertainty
- B. Primarily systematic uncertainty
- C. Both statistical and systematic uncertainty
- D. Neither statistical nor systematic uncertainty

- You have heard that a cat can run up to 30 miles per hour and alligators can move at 11 miles per hour on land, and you want to know how fast you can move. Your friend has the fully automatic electronic timing system that is used for the Olympics, and you use it to time how fast you can run 100 meters at full speed. Your friend fires a starter’s pistol in the air to start your sprint, and the timer automatically starts at the sound of the pistol. The timer automatically stops when you cross a beam of light shining across the finish line. Which of the following will be a noticeable source of uncertainty when you compare your speed to the Olympic record?
- A. Primarily statistical uncertainty
- B. Primarily systematic uncertainty
- C. Both statistical and systematic uncertainty
- D. Neither statistical nor systematic uncertainty

- Vote for a candidate for the Berkeley School Board Director
- A. Judy Appel
- B. Norma Harrison
- C. Tracy Hollander
- D. Jane Shelton
- E. Beatrice Cutler

- Discussion Questions
- How could we find out if there were a systematic bias in our senses?
- Consider the problem of measuring your class' understanding of types of uncertainty. What is a possible source of systematic uncertainty in measuring each individual's true understanding? What is a possible source of statistical uncertainty? How could both be reduced?
*Hint: A larger number of and more varied problems on quizzes*What would be the cost of reducing these uncertainties in this case?*Hint: Too much time spent on quizzes*

- Class Exercises
- Students line up in “human histograms,” demonstrating statistical dispersion and systematic bias.
- Small group discussions, clicker-questions, and exercises to have students identify the statistical versus the systematic uncertainties in a number of scenarios.

- Practice Problems
- What are plausible sources of systematic error for this graph?
- What are plausible sources of statistical error?
- Why do you think the four different lines vary as much as they do?
- Why do the four lines correspond as tightly as they do? (
*Hint: Connect back to Shared Reality*

- Homework Questions
- It’s a very close race between two presidential candidates. You have access to two polls, both of which show the race to be very close, but predict a different winner. Poll A consists of 50,000 college students. Poll B is comprised of a sample of 50 people drawn from the U.S. census. How much do you trust the result of each poll and why? Based just on the results of these two polls, which candidate do you think is more likely to become president, the one winning Poll A or the one winning Poll B, or is there not enough information to decide?
- Suppose that, after going through all of the processes described in his article to estimate the uncertainty in the election, Nate Silver's election forecasting model had indicated that there was a 99.9% chance that Obama would win the election (instead of an 83.7% chance). Would you be 99.9% sure Obama would win? Why or why not? Relate your answer to the Pengra & Dillman reading on the two types of uncertainty.
- One way to gather data about how people might respond to a product or idea is to conduct a “focus group,” in which several participants share their impressions in a group conversation. Based on the reading, why is or isn’t this a good way to obtain reliable information?