7 <strong>Topic VII. Causal Claims in the Messy Real World<…
Topic VII. Causal Claims in the Messy Real World
- Building on Correlation and Causation, we examine how to collect evidence for causality in more difficult cases.
- In the previous class we began our discussion of causality, distinguishing it from mere association and considering ideal kinds of evidence for causality, when we can run randomized controlled trials. However, in many cases, it is not possible to run RCTs to test causal hypotheses, for ethical or practical reasons. In this class, we consider other forms of evidence for causality, which cannot individually be as conclusive as RCTs but together can still present compelling evidence for causal theories.
- Addressing the Question: How do we find out how things work?
- Non-RCT evidence for causality__
- Exemplary Quotes
- “Ok, I agree that ‘correlation doesn’t prove causation’ in general, but in a case like this where we have lots of other kinds of evidence it sure gives us a pretty strong guess about causation.”
- “There is an answer to this causal question. Just because we can’t ethically do a randomized controlled study with these patients, it doesn’t mean that we can’t make progress establishing the causal link between these treatment options and the outcome. After all, we have pretty good evidence that the energy from the sun is caused by nuclear fusion and we haven’t done any randomized controlled experiments!”
- "One hundred years after that, French chemist Antoine Lavoisier used a device called an “ice calorimeter” to gauge the energy burn from animals —like guinea pigs — in cages by watching how quickly ice or snow around the cages melted. This research suggested that the heat and gases respired by animals, including humans, related to the energy they burn."
- Cautionary Quotes: Mistakes, Misconceptions, & Misunderstandings
- Students tend to think that one flaw in a study makes the study useless as evidence, rather than merely weaker evidence than it would be without the flaw.
- Students are quick to notice small sample size, slower to notice problems with experimental design.
- Students struggle to generate non-RCT types of evidence for causality, although they are better at recognizing it.
- Identifying natural experiments is also difficult, probably because many students find the principles of RCTs slippery.
- "Since we can't run a randomized controlled trial on whether CO2 emissions cause global warming, we can't ever know whether it does."
- A. ATTITUDES
- Appreciate that we can sometimes get very good evidence for a causal hypothesis, even in the absence of decisive RCTs.
- Be wary of potential confounds in apparent evidence for causality.
- B. CONCEPT ACQUISITION
- In many cases it is not possible to conduct a true RCT to test causality, for practical or ethical reasons.
- There are non-RCT forms of evidence for causal hypotheses, which are less conclusive than RCTs but together can offer strong evidence for causation. These include:
- a. Prior plausibility: Can a plausible mechanism be constructed, or is there some other basis for interpreting the current evidence in terms of one causal structure over another, such as data from other studies?
- b. Temporality/temporal sequence: Did the hypothesized cause precede the effect?
- c. Dose-response curve: Do the quantities of the hypothesized cause correlate with the quantity, severity, or frequency of the hypothesized effect across ranges?
- d. Consistency across contexts: Does the correlation appear across diverse contexts?
- C. CONCEPT APPLICATION
- For a given causal hypothesis and imperfect study, identify the imperfections (e.g., sample size, lack of randomization, lack of control) and explain how these imperfections impact claims of causality.
- Identify potential confounds in RCT and non-RCT studies.
- Sketch out methods for eliminating potential confounds in sample RCT or non-RCT studies.
- For a given scenario in which a causal hypothesis/claim is being made, identify plausible alternative hypotheses that could be consistent with the data.
- For a given scenario in which a causal hypothesis is being made, describe an ideal experiment/set of experiments to test the hypothesis and rule out alternative hypotheses.
- Identify cases in which ‘ideal’ experiments are not possible, due to ethical or practical constraints.
- Evaluate the strength of causal claims when various sources of evidence are used to help mitigate flawed experiments, including prior plausibility, dose-response relationships, size of effect, temporal ordering, and multiple complementarily-flawed experiments.
- Identify additional sources of evidence that could be used to help mitigate flawed experiments, including prior plausibility, dose-response relationships, size of effect, temporal ordering, and multiple complementarily-flawed experiments.
- Suggested Readings & Reading Questions
- Clicker Questions
- Discussion Questions
- Brainstorm in small groups three causal links that you are pretty confident are real without RCT evidence. Why are you confident about these causal links? Hint: Think about causal links in everyday life, like allergies or breaking a glass.
- Practice Problems
- Observing that major rainstorms tend to develop on the edge of massive cold fronts, scientists conjecture that the cold fronts cause the storms. Give an alternative hypothesis that could explain the data. Then say what evidence could help rule out the alternative hypothesis, and how convincing it would be.
- Alternative Hypothesis:
- Distinguishing Experiment:
- Class Exercises
- Data Science Applications
- A complex dataset where they needed to plot time sequence to make sense of it?