Topic IX. Seeing Patterns in Random Noise
Filtering on
Context for this filter:
-
LEARNING GOALS
- B. CONCEPT ACQUISITION
- Look Elsewhere Effect: Even if there is a low probability of pure noise passing a given threshold for signal, if we look at enough noise some of it will pass that threshold by chance. I.e., if there is a low probability of obtaining a false positive in any given instance, the more times you try (the more questions you ask, measures you take, or studies you run without statistical correction), the more you increase the probability of getting a false positive. This occurs when one:
-
EXAMPLES
- Exemplary Quotes
- "I know it seems super meaningful that we ran into each other in Australia, when neither of us live in Australia, but I guess the chances of running into someone you know at some point, if you travel a lot and know a lot of people, are pretty high."
- "There are many many cases of people making insanely correct predictions, so many that some people are convinced clairvoyance is real. But there are many more cases of people making totally wrong predictions. So it's probably just noise; with enough predictions, someone will be correct by luck."
- Cautionary Quotes: Mistakes, Misconceptions, & Misunderstandings
- Students had great difficulty recognizing the Look Elsewhere Effect in sets of studies, in part because they struggled with the conceptual underpinnings of statistical significance.
LEARNING GOALS
- B. CONCEPT ACQUISITION
- Look Elsewhere Effect: Even if there is a low probability of pure noise passing a given threshold for signal, if we look at enough noise some of it will pass that threshold by chance. I.e., if there is a low probability of obtaining a false positive in any given instance, the more times you try (the more questions you ask, measures you take, or studies you run without statistical correction), the more you increase the probability of getting a false positive. This occurs when one:
- a. Asks too many questions of the same data set, reporting only statistically significant results.
- b. Asks the same question of multiple data sets, reporting only statistically significant results.
- c. Runs a test or similar tests too many times, reporting only statistically significant results.
- d. This also occurs in everyday life, e.g. when one looks at a whole lot of phenomena and only takes note of the most surprising-looking patterns, not properly taking into account the larger number of unsurprising patterns/lack of pattern.
- C. CONCEPT APPLICATION
- Describe how scientists guard against detecting a signal that does not exist.
- Recognize and explain the flaw in everyday scenarios in which people mistake noise for signal (e.g. Look Elsewhere Effect, gambler’s fallacy, hot-hand effect).
- Recognize and explain the flaw in a scenario where scientists mistake noise for signal.
-
LEARNING GOALS
- C. CONCEPT APPLICATION
- Distinguish legitimate inquiry into human groups from bad science.