20 <strong>Topic XX. Blind Analysis</strong>
Topic XX. Blind Analysis
- Blind analysis, the practice of deciding how we will analyze data before finding out if the analysis we have chosen supports our hypothesis, counteracts confirmation bias.
- Science is not a single “scientific method” (as often taught in school), but better characterized as an ever-evolving collection of tricks and techniques to compensate for our mental (and, occasionally, physical) failings and build on our strengths—and, in particular, to help us avoid fooling ourselves. These techniques must constantly be re-invented, as we develop new ways to study and explain the world. In the last few decades we have entered a period in which most scientific analyses are complicated enough to require significant debugging before a result is clear. This has exposed another way we sometimes fool ourselves: the tendency to look for bugs and problems with a measurement only when the result surprises us. Where previously we recognized the need for “double blind” experimentation for medical studies, now some fields of science have started introducing blind analysis, where the results are not seen during the development and debugging of the analysis—and there is a commitment to publish the results, however they turn out, when the analysis is “un-blinded” and the results interpreted.
- Addressing the Question: How can we avoid going wrong?
- Blind Analysis
TOPIC RESOURCES
EXAMPLES
- Exemplary Quotes
- Cautionary Quotes: Mistakes, Misconceptions, & Misunderstandings
- Confusion between blind analysis and double blind experiments is common. These concepts are related but distinct.
LEARNING GOALS
- A. ATTITUDES
- One should always be looking for ways that we get things wrong (by fooling ourselves or due to bugs in our reasoning processes) so that we can invent better procedures.
- This is important both for methods long in use and new ones (e.g. big data).
- B. CONCEPT ACQUISITION
- Blind analysis: Making all decisions regarding data analysis before the results of interest are unveiled, such that expectations about the results do not bias the analysis. Usually co-occurs with a commitment to publicize the results however they turn out.
- Examples of analysis decisions for which blind analysis could be useful:
- a. Stopping rules for when to stop looking for flaws in your experimental design or for computer/math bugs.
- b. Data selection decisions.
- c. Decisions about which analysis procedures to use (e.g. grouping decisions).
- Confirmation bias drives the need for blind analysis.
- Confirmation bias is pervasive and doesn’t necessarily indicate any fraudulent activity.
- Approaches to reducing confirmation bias other than blind analysis:
- a. Preregistration: A research group publicly commits to a specific set of methods and analyses before they conduct their research.
- b. Registered replication: [A] research group(s) commits to a specific set of methods and procedures to verify the result of an earlier work (typically with the input of the original research team). Results are publicized regardless of outcome.
- c. Adversarial collaboration: Scientists with opposing views agree to all the details of how data should be gathered and analyzed before any of the results are known.
- d. Peer review: New results are evaluated by other experts in the same field to determine whether they are valid. This only reduces confirmation bias if reviewers don’t share biases.
- Scientists are constantly looking for bugs in scientific practices in order to fix them. Blind analysis is just the latest example of scientists recognizing a bug in their practice (e.g., a way of being fooled) and adjusting practice to account for/remove the bug.
- C. CONCEPT APPLICATION
- Explain why blind analysis might be needed, by explaining the errors that might arise in its absence.
- Recognize when blind analysis is being used and explain what function it serves. Identify situations and decisions that would call for blind analysis.
- Evaluate techniques (e.g., registered replication, adversarial collaboration, peer review) - a) for ability to address confirmation bias and b) in comparison to blind analysis.
- Propose how to use blind analysis for simple studies.
- Stretch Goal: Propose new practices to solve a new (fictional) problem with current scientific practice. (This is a stretch goal for evaluation, too! Can we invent a new problem with a new scientific practice?)
CLASS ELEMENTS
- Suggested Readings & Reading Questions
- We are hiding this reading because we want students to do their demos truthfully. But if you want it anyways...Hide Results to Seek Truth, Saul Perlmutter and Rob MacCoun
- Clicker Questions
- Students are presented historical graphs of improved published measurements of a physical parameter over the decades and must identify the ones that retrospectively show evidence of biases that could have been avoided by blind analysis.
- Discussion Questions
- Why is it important for scientists to publicize results even if they don't get what they predicted?
- Class Exercises
- Students make a measurement which is somewhat tricky to perform with two-digit precision, and experimental conditions are set up to show that the part of the class that was “blinded” gets a more accurate result.
- There could be a variant of the above incorporating blind analysis into a more typical lab class, where there is an expectation about the results, half the class does the analysis blinded, half the class does the analysis unblinded.