4 <strong>Topic IV. Systematic and Statistical Uncertainty<…

"It won't do us any good to average lots and lots of test subjects' heights together if our tape measure got shrunk in the wash!"

"If we estimate the effect of a drug on weight by randomly assigning people to take the drug vs. not take it and then measuring their weight after a year, we could subtract the average weight loss of drug-takers vs. non-drug-takers to get the effect size of the drug on weight loss. But the people know if they're taking a drug for weight loss, so there could be a placebo effect creating a systematic bias. So the better way to do the experiment is to give the control group sugar pills. Then we can be more confident that any weight loss is due to the drug, and not a systematic bias created by the placebo effect."

Often our contact with reality is mediated by measurement and quantification. We need to be aware that every measurement comes with some degree of uncertainty. 

Statistical Uncertainty/Error: Differences between reality and our measurement on the basis of random imprecisions.

Systematic Uncertainty/Error: Differences between reality and our measurement that skew our results in one direction.

Identify sources of measurement uncertainty/error that introduce statistical uncertainty/error, that introduce systematic uncertainty/error, and that introduce both.  

Suggest approaches to reducing or constraining measurement uncertainty (both statistical and systematic).  

Recognize that the process of science involves creativity in identifying sources of systematic uncertainty and inventing strategies to reduce or eliminate them.

Sara is running a mile, which is a four-lap race around a track. Her coach wants to time how fast she runs on the third lap of the four-lap race. As Sara goes past the line on the track that marks the the start of the laps, her coach presses a button on a stopwatch to start a timer. When Sara goes past that line again, her coach presses the button again to stop the stopwatch. Which of the following will be a noticeable source of uncertainty in the coach’s measurement of how long it took Sara to run the lap?

You have heard that a cat can run up to 30 miles per hour and alligators can move at 11 miles per hour on land, and you want to know how fast you can move. Your friend has the fully automatic electronic timing system that is used for the Olympics, and you use it to time how fast you can run 100 meters at full speed. Your friend fires a starter’s pistol in the air to start your sprint, and the timer automatically starts at the sound of the pistol. The timer automatically stops when you cross a beam of light shining across the finish line. Which of the following will be a noticeable source of uncertainty when you compare your speed to the Olympic record?

Vote for a candidate for the Berkeley School Board Director

How could we find out if there were a systematic bias in our senses?

Consider the problem of measuring your class' understanding of types of uncertainty. What is a possible source of systematic uncertainty in measuring each individual's true understanding? What is a possible source of statistical uncertainty? How could both be reduced? Hint: A larger number of and more varied problems on quizzes What would be the cost of reducing these uncertainties in this case? Hint: Too much time spent on quizzes

Students line up in “human histograms,” demonstrating statistical dispersion and systematic bias.

Small group discussions, clicker-questions, and exercises to have students identify the statistical versus the systematic uncertainties in a number of scenarios. 

Consider this graph](https://earthobservatory.nasa.gov/Features/WorldOfChange/images/decadaltemp/annualtemperatureanomalies_2014.png) (see this [NASA site for context):

What are plausible sources of systematic error for this graph?

What are plausible sources of statistical error?

Why do you think the four different lines vary as much as they do?

Why do the four lines correspond as tightly as they do? (Hint: Connect back to Shared Reality

It’s a very close race between two presidential candidates. You have access to two polls, both of which show the race to be very close, but predict a different winner. Poll A consists of 50,000 college students. Poll B is comprised of a sample of 50 people drawn from the U.S. census. How much do you trust the result of each poll and why? Based just on the results of these two polls, which candidate do you think is more likely to become president, the one winning Poll A or the one winning Poll B, or is there not enough information to decide?

Suppose that, after going through all of the processes described in his article to estimate the uncertainty in the election, Nate Silver's election forecasting model had indicated that there was a 99.9% chance that Obama would win the election (instead of an 83.7% chance). Would you be 99.9% sure Obama would win? Why or why not? Relate your answer to the Pengra & Dillman reading on the two types of uncertainty.

One way to gather data about how people might respond to a product or idea is to conduct a “focus group,” in which several participants share their impressions in a group conversation. Based on the reading, why is or isn’t this a good way to obtain reliable information?