Topic I. Role of Science in a Democracy
Course Overview
The focus in this course is on the errors humans tend to make, and the approaches science methodology has given us (and we are still developing) to prevent or at least minimize those errors. Learn more »
-
LEARNING GOALS
- A. ATTITUDES
- Recognize the need to distinguish facts from values in political and everyday decision-making
- B. CONCEPT ACQUISITION
- Facts vs. Values
- a. Facts: Objectively true claims about reality. Everything that is the case. What is, descriptively, including spatial relations, causal relations, attributes of objects, etc.
- b. Values: What is of value, important, of worth. Oughts, shoulds, etcs.
- Democracy vs. Epistocracy
- a. Democracy: A system of government wherein a society’s citizens have more or less equal input into policies.
- b. Epistocracy: A system of government wherein a particular subset of a society—privileged by their education or other markers of expertise—decides policies.
- Scientific expertise has utility for political decision-making.
- Social and behavioral aspects of the world can be approached scientifically and, therefore, have relevant experts.
- C. CONCEPT APPLICATION
- Facts vs. Values
- a. Distinguish between facts and values in public debates and everyday decision-making.
- b. Identify ways in which facts and values are intertwined in public debates and everyday decision-making.
- c. Recognize when values determine which facts are viewed as relevant.
- d. Use distinction between facts and values to identify appropriate source(s) of authority in decision-making.
- Justify when it is appropriate for everyone to have a vote.
- Justify when it is appropriate for science to have a role in democratic decision-making and explain what that role should be.
- Democracy vs. Epistocracy.
- a. Explain arguments for each.
- b. Describe tension between the claims of the two.
- c. Appeal to scientific expertise (including in the social and behavioral sciences) for political decision-making.
- d. Identify democratic and epistocratic aspects of real governments.
-
LEARNING GOALS
- A. ATTITUDES
- Appreciate how scientific theories can be trustworthy even though past theories have turned out to be wrong.
- Appreciate how science can be both always subject to challenge, yet (often) rightly depended on for practical decision-making.
- Optimism in capacity of science to help solve problems & improve representations, for both political & personal decision-making.
- B. CONCEPT ACQUISITION
- Assumption of Reality
- Scientists assume an external reality, which is shared by and affects all people and has enough regularity to lend itself to induction. This external reality is what scientists seek to describe accurately.
- Empirical Evidence
- Science is based on appeal to empirical evidence, which is publicly accessible on the assumption of reality (although may require special instruments and/or expertise to acquire).
- Validity of Concepts
- The extent to which a scientific concept is responding to some real external thing.
- Science vs. Priesthood
- Science gains its authority from its self-questioning character, not from the concentrated power of individuals.
- The process of science leads to self-correction through:
- a. Active experimentation & observation
- b. Peer review
- c. Rewards for better theories, even those that contradict current theories
- d. Responsiveness to new evidence/actively open-minded thinking
- e. Replicability and replication
- f. Multiple approaches to each problem allow convergent evidence
- g. Interconnected nature of science, and ongoing attempts to connect the pieces that are not yet connected (e.g. Biological Synthesis of genetics & natural selection as a successful instance, cognitive neuroscience as an instance of integration currently in progress, the challenge of connecting quantum physics to general relativity…).
- The Raft vs. the Pyramid metaphors for science
- a. The Raft: Every scientific claim is subject to question and reevaluation; we can use the rest of our scientific knowledge to question any one claim at a time, though we cannot question the entire edifice at once.
- b. The Pyramid: Science builds on fixed foundations to ever higher levels of knowledge.
- Different approaches to truth in science
- a. Operationalism
- Whatever definitions work for the time being are equally valid.
- b. Conventionalism
- There is one correct answer, fixed by society.
- c. Realism
- There is one correct answer, fixed by the world.
- Social Constructivism
- “The reality [of a scientific entity or fact] is formed as a consequence of stabilization [of a controversy].” (Latour & Woolgar, 1986)
- Badging
- The phenomenon of people using claims of fact to express their identity or group affiliation.
- e.g. Midwestern farmers denying that climate change is real, even while they buy equipment and make preparations for climate change that will affect their crops.
- C. CONCEPT APPLICATION
- Defend critiques of science based on its provisionality by appeal to its self-correcting properties
- Explain and contrast the metaphors of raft vs. pyramid for science
- Identify strengths/weaknesses in Raft & Pyramid metaphors for science
- e.g. Some scientific theories are more central than others, and harder to replace. But all scientific theories are, in principle, open to revision in light of new evidence.
- Distinguish concept validity from (1) a social-constructivist picture of scientific concepts free-floating in a world of mutual agreement among power brokers, not moored to a universally shared reality, and (2) subjective preferences.
- a. Identify cases where concept validity is expected
- e.g. What is a quark/electron/boson? What is an animal?
- b. Identify cases where social constructivism might be a good approach
- e.g. What is her name? What is the name of this city?
- c. Identify cases where preference might be sufficient
- Which chocolate is tastiest? Which color palette is prettiest?
- Use the concept of validity to assess scientific claims, contrasting cases where the validity of the concept is on stronger vs. weaker footing.
- a. In straightforward cases
- e.g. Everyone or nearly everyone can agree about which animals are cats, and consequently agree that most cats have fur, etc.
- b. In less straightforward cases
- e.g. Claims about bosons, dark energy, what sort of black hole is at the center of the Milky Way
- c. In difficult cases
- e.g. Disagreement is rife over what intelligence is, so claims about the relative intelligence of two groups of people are more questionable.
- Distinguish when operationalist, conventionalist, and realist approaches to scientific claims & definitions are being used.
- Recognize cases where apparent claims of fact may also be characterized as expressions of affiliation or identity (i.e. badging).
- e.g. Stated acceptance of creationism and rejection of evolution is only very weakly responsive to education and strongly associated with affiliative factors like religion, religiosity, and political ideology, suggesting that it may be (to some extent) a result of badging.
- Falsifiability: Scientific claims are taken more seriously if they are testable.
-
LEARNING GOALS
- A. ATTITUDES
- Place appropriate trust in instruments where direct observation is not possible (or is less precise/accurate).
- B. CONCEPT ACQUISITION
- An instrument can have greater precision and accuracy than direct observation or the instruments used to test and calibrate it.
- Validity of a Measure: Is the measure yielding information about the target entity in out in the world, given that the target is something real (i.e., concept is valid).
- Challenges in validating the use of an instrument:
- a. When there is no objective gold standard (e.g. passage of time, what fluid to use in a thermometer)
- b. When direct observation is messy or impossible (e.g. radio waves, size)
- Techniques for validating instruments:
- a. Interactive exploration: Testing an instrument by changing the thing it is measuring in ways you know through other means, and seeing if the instrument recognizes the changes appropriately (e.g. does driving increase a car's odometer; see how singing higher and lower notes affects a sound spectrograph; sprayable electrons in Hacking reading).
- b. Comparison of multiple instruments (e.g. thermometers)
- c. Comparison to direct observation (e.g. naked sight compared to sight with a magnifying glass)
- C. CONCEPT APPLICATION
- Recognize continuum from direct observation to indirect observation via instruments
- Identify when and how interactive exploration, comparison of multiple instruments, and comparison to direct observation can be used to validate instruments.
- Identify cases in which instrumentation is needed to solve problems (e.g. entities/events are too slow, too fast, too small — that is, those for which we cannot rely on our everyday senses and cognitive capacities), especially in policy contexts.
- Explain how our ability to use the techniques for validating an instrument above can lead to more/less confidence in a given measurement.
- Identify where challenges in validating an instrument (and associated lack of confidence about what it is measuring) can make policy decisions more difficult.
-
LEARNING GOALS
- A. ATTITUDES
- Often our contact with reality is mediated by measurement and quantification. We need to be aware that every measurement comes with some degree of uncertainty.
- B. CONCEPT ACQUISITION
- Statistical Uncertainty/Error: Differences between reality and our measurement on the basis of random imprecisions.
- a. All measurements have a certain amount of variance, which are just differences between multiple measurements due to error and/or genuine variation in the sample. These differences will not all go in the same direction.
- b. Statistical uncertainty can be reduced by averaging a larger amount of data.
- Systematic Uncertainty/Error: Differences between reality and our measurement that skew our results in one direction.
- a. Such measurements will show a consistent bias, that is, a consistent deviation from reality in one direction.
- b. Systematic uncertainty cannot be reduced by averaging a larger amount of data.
- C. CONCEPT APPLICATION
- Identify sources of measurement uncertainty/error that introduce statistical uncertainty/error, that introduce systematic uncertainty/error, and that introduce both.
- Suggest approaches to reducing or constraining measurement uncertainty (both statistical and systematic).
- Recognize that the process of science involves creativity in identifying sources of systematic uncertainty and inventing strategies to reduce or eliminate them.
-
LEARNING GOALS
- A. ATTITUDES
- Appreciate how the “can-do” spirit of inquiry (and inventive experimental techniques) counter-balances the difficulties of discovery/innovation.
- Appreciate that iterative work on a problem is the norm in science and the most productive approach (and in some cases the only way of being productive), even when it looks like it’s not getting anywhere.
- Persist on difficult problems (scientific and non-scientific) due to optimism that iterative work will eventually pay off with interesting insights into the problem.
- Recognise that an optimistic view of the tractability of a problem and/or one’s ability to solve it eventually can in itself affect one’s capacity to solve the problem.
- B. CONCEPT ACQUISITION
- Scientific Optimism: An attitude of optimism that persistence and iteration on a difficult scientific problem will eventually pay off with interesting insights into your problem.
- Skeptical/Gatekeeping Function: Science is in the business of rigorously testing claims against experience, rather than merely accepting them.
- Discovery/Innovation Function: Science is in the business of generating new theories for how to explain the world. This is both difficult (requires resources, uncertain success) and important (need to make decisions, wouldn’t have anything to “gatekeep” if new scientific ideas weren’t being generated).
- Omnivorous Science: Constantly learning new techniques, exposure to a variety of hypotheses & theories, interdisciplinary discussion, etc. Important to progress because there are payoffs for learning novel experimental/technological/theoretical techniques and questions/problems from many domains of science, even beyond the one that one starts from.
- Zero Sum Games vs. Enlarging the Pie: A can-do spirit goes along with optimism that problems can be solved by enlarging the pie, not just redistributing zero-sum goods.
- C. CONCEPT APPLICATION
- Explain the interplay and importance of the skeptical/gatekeeping aspect of science versus discovery/innovation. .C
- Explain how “Omnivorous science” can be important to progress.
- Critique impatience with the gradual nature of scientific progress by appealing to science’s iterative nature.
- Critique cases of policy failures due to lack of an appropriately iterative/persistent approach to the problem.
- Generate examples of iterative processes for solving non-scientific problems.
-
LEARNING GOALS
- A. ATTITUDES
- B. CONCEPT ACQUISITION
- Correlation is insufficient to demonstrate causation because there are other causal structures that lead to correlation:
- a. A causes B (direct causation);
- b. B causes A (reverse causation);
- c. A and B are both caused by C
- d. A causes B and B causes A (bidirectional or cyclic causation);
- e. There is no connection between A and B; the correlation is a coincidence.
- f. The effect of A on B depends on C
- Randomized Controlled Trial (RCT): An attempt to identify causal relations by randomly assigning subjects into two groups and then performing an experimental intervention on the subjects in one of the groups.
- a. Experimental Intervention: The act of an experimenter changing one variable in a possible causal network.
- b. Randomized Assignment: Given sufficient sample size, randomized assignment rules out confounds by distributing variation randomly between the two groups, thereby avoiding systematic differences except as the result of the intended intervention.
- c. Control Condition: Comparison of an experimental to a control condition is necessary in order to distinguish effect of intervention from changes that would have occurred without the intervention.
- d. Sampling: A study of a well-chosen sample can tell you something about the population (through induction), especially if it was selected in such a way as to avoid any systematic differences between the sample and the rest of the population. It is often difficult or even impossible to capture a perfectly representative sample, so scientists do the best they can. For example, many psychology studies are done with college students because they are accessible, but such samples differ systematically from the general population. Inferences from samples to a larger population need to take such differences into account.
- Causation: X causes Y if and only if X and Y are correlated under interventions on X.
- a. This is a technical notion, which overlaps with but is slightly different from everyday usage. For example, everyday usage of the word “cause” can be influenced by moral considerations, the complexity of a causal mechanism, and/or the nature of the mechanism. We typically don’t say that the big bang "caused" this sequence of letters, or that the presence of oxygen caused the forest fire, etc, but scientifically they are part of the causal history of those phenomena.
- b. There can be other evidence for causation, even when actually performing an intervention is not feasible. However, saying something is a cause implies that there is in principle a relationship under an intervention.
- C. CONCEPT APPLICATION
- State some of the basic problems in establishing causation.
- Recognize some of the basic problems in establishing causation and use them to identify situations in which claims of causation are and are not warranted.
- Explain why a randomized controlled trial can help rule out spurious correlations.
- Identify the flaw in an argument in which correlation is inappropriately being substituted for causation.
- Address the argument, “Science can only establish correlations; it can’t determine causality.”
- Use the definition of causation to identify situations in which claims of causation are and are not warranted.
- Design RCTs for sample problems.
- Identify flaws in experimental designs aimed at testing causality and explain how the flaws could be addressed.
-
LEARNING GOALS
- A. ATTITUDES
- Appreciate that we can sometimes get very good evidence for a causal hypothesis, even in the absence of decisive RCTs.
- Be wary of potential confounds in apparent evidence for causality.
- B. CONCEPT ACQUISITION
- In many cases it is not possible to conduct a true RCT to test causality, for practical or ethical reasons.
- There are non-RCT forms of evidence for causal hypotheses, which are less conclusive than RCTs but together can offer strong evidence for causation. These include:
- a. Prior plausibility: Can a plausible mechanism be constructed, or is there some other basis for interpreting the current evidence in terms of one causal structure over another, such as data from other studies?
- b. Temporality/temporal sequence: Did the hypothesized cause precede the effect?
- c. Dose-response curve: Do the quantities of the hypothesized cause correlate with the quantity, severity, or frequency of the hypothesized effect across ranges?
- d. Consistency across contexts: Does the correlation appear across diverse contexts?
- C. CONCEPT APPLICATION
- For a given causal hypothesis and imperfect study, identify the imperfections (e.g., sample size, lack of randomization, lack of control) and explain how these imperfections impact claims of causality.
- Identify potential confounds in RCT and non-RCT studies.
- Sketch out methods for eliminating potential confounds in sample RCT or non-RCT studies.
- For a given scenario in which a causal hypothesis/claim is being made, identify plausible alternative hypotheses that could be consistent with the data.
- For a given scenario in which a causal hypothesis is being made, describe an ideal experiment/set of experiments to test the hypothesis and rule out alternative hypotheses.
- Identify cases in which ‘ideal’ experiments are not possible, due to ethical or practical constraints.
- Evaluate the strength of causal claims when various sources of evidence are used to help mitigate flawed experiments, including prior plausibility, dose-response relationships, size of effect, temporal ordering, and multiple complementarily-flawed experiments.
- Identify additional sources of evidence that could be used to help mitigate flawed experiments, including prior plausibility, dose-response relationships, size of effect, temporal ordering, and multiple complementarily-flawed experiments.
-
LEARNING GOALS
- A. ATTITUDES
- B. CONCEPT ACQUISITION
- Multiple Causation: Any given effect may be brought about by a complex combination of many causes (which may interact with each other), with varying degrees of influence on the outcome. (see Topic XIII).
- Singular Causation: A causal relation between specific events — i.e., Event A caused Event B.
- General Causation: A causal relation between variables — i.e., X causes Y.
- Causation as Production: There is a spatiotemporally connected series of causal connections between two events or event types (i.e., the kind of causation people have in mind when they say there is no action at a distance; e.g., commission).
- Causation as Dependence: If X hadn’t happened, Y would not have happened (counterfactual dependence, e.g. omission, double prevention [prevention of a prevention]).
- Decision-making involves not only assessment of the outcome, but also the agent’s causal role in the production of the outcome (i.e., omission vs. commission, e.g. trolley dilemma).
- C. CONCEPT APPLICATION
- Distinguish between cases of singular and general causation.
- Distinguish between cases of production and dependence.
- Distinguish between the evidence needed to establish singular causation and the evidence needed to establish general causation.
- Identify different policy implications of singular vs. general causation (e.g., for policy and legal decision-making).
- Identify different policy implications of production vs. dependence (e.g., for policy and legal decision-making).
-
LEARNING GOALS
- A. ATTITUDES
- B. CONCEPT ACQUISITION
- Signal: Aspects of observations or stimuli that provide useful information about the target of interest, as opposed to noise.
- Noise: The aspects of observations or stimuli that distract from, dilute, or get confused with signal, and are not signal (i.e., do not provide useful information about the target of interest).
- Noise is frequently, but not always, the result of random measurement fluctuations.
- Observations/stimuli subject to confusion between signal and noise include communication, measurements, descriptions, etc.
- Signal-to-Noise Ratio: The relative strength of signal compared to the relative strength of noise in a given context. Obtaining meaningful information from the world requires distinguishing signal from noise. Therefore, human cognition (both scientific and otherwise) relies on techniques and tools to suppress noise and/or amplify signal (i.e., increase signal-to-noise ratio).
- It is possible to design filters to increase the signal-to-noise ratio, if you know where the noise is going to appear.
- C. CONCEPT APPLICATION
- Identify examples of “signal” and “noise,” recognizing that these examples are context-dependent.
- Roughly compare measurement techniques in terms of their resultant signal-to-noise ratios.
- Describe examples of techniques and tools to suppress noise and/or amplify signal (i.e., increase signal-to-noise ratio).
-
LEARNING GOALS
- A. ATTITUDES
- Be wary of our tendency to see patterns that do not exist (to see signal where there is in fact only noise).
- B. CONCEPT ACQUISITION
- People are (evolutionarily?) disposed to over-perceive signal (i.e., noise often gets misinterpreted as signal), perhaps because the cost of missing real signal (false negatives) is typically higher than the cost of mistaking noise for signal (false positives).
- People tend to see any regularity as a pattern (i.e., see more signal than there is), even when “patterns” occur by chance (i.e. are pure noise), e.g.: People underestimate the frequency of apparent patterns produced by randomness, leading to overperception of spurious signal much more frequently than people account for. (Events that are just coincidental are much more likely than most people expect.)
- Gambler’s fallacy: Expecting that streaks will be broken, such that future results will “average out” earlier ones, even when all trials are independent.
- Hot-hand fallacy: Expecting that streaks will continue, even when all trials are independent.
- Look Elsewhere Effect: Even if there is a low probability of pure noise passing a given threshold for signal, if we look at enough noise some of it will pass that threshold by chance. I.e., if there is a low probability of obtaining a false positive in any given instance, the more times you try (the more questions you ask, measures you take, or studies you run without statistical correction), the more you increase the probability of getting a false positive. This occurs when one:
- a. Asks too many questions of the same data set, reporting only statistically significant results.
- b. Asks the same question of multiple data sets, reporting only statistically significant results.
- c. Runs a test or similar tests too many times, reporting only statistically significant results.
- d. This also occurs in everyday life, e.g. when one looks at a whole lot of phenomena and only takes note of the most surprising-looking patterns, not properly taking into account the larger number of unsurprising patterns/lack of pattern.
- Statistical Significance: How unlikely a given set of results would be if the null hypothesis were true (i.e. if the hypothesized effect did not actually exist).
- [Technical term: P-values: the probability of getting a result as extreme or more if in fact the hypothesis is false, simply through random noise.]
- Lack of statistical significance does NOT prove the null hypothesis.
- C. CONCEPT APPLICATION
- Describe how scientists guard against detecting a signal that does not exist.
- Recognize and explain the flaw in everyday scenarios in which people mistake noise for signal (e.g. Look Elsewhere Effect, gambler’s fallacy, hot-hand effect).
- Recognize and explain the flaw in a scenario where scientists mistake noise for signal.
- Given a news article or other concrete example, correctly extract the effect size versus statistical significance of a causal factor, and explain how each affects the importance and usefulness of the results.
-
LEARNING GOALS
- A. ATTITUDES
- Given some degree of uncertainty, appreciate that different kinds of errors come with different costs, such that in some cases it is worthwhile to presume the less likely alternative because the error you risk is less costly.
- B. CONCEPT ACQUISITION
- False Positive/Type I Errors: A test yields a positive result, but in fact the condition is not present.
- False Negative/Type II Errors: A test yields a negative result, but in fact the condition is present.
- There is always the possibility of a trade-off—for a given test, one can reduce the risk of false positives by increasing the risk of false negatives, and vice-versa.
- Good decision-making under uncertainty involves having sufficient signal (an adequate test) and setting your threshold appropriately for the relative costs of false positives and false negatives.
- In some classification cases like pornography identification or graduate school admissions, there may not be a “truth of the matter” so there aren’t true “false” positives or “true” negatives, although a threshold must still be set.
- C. CONCEPT APPLICATION
- Identify false positive (type I) and false negative (type II) errors in scientific and everyday situations.
- Weigh the costs associated with false positives/negatives with the benefits associated with true positives/negatives when making a decision under uncertain conditions.
- Explain how people could come to different decisions or policies as a result of different utilities/values associated with different types of errors, even if they agree about the relevant facts.
-
LEARNING GOALS
- A. ATTITUDES
- Recognize that every proposition comes with a degree of uncertainty.
- e.g., not be impressed by statements made with 100% confidence, no error bars or confidence intervals on claimed measurements, seeking definitive answers when only probabilistic information is available, not recognizing that probabilistic information is better than no information.
- Value and defend scientific expressions of uncertainty.
- B. CONCEPT ACQUISITION
- Credence: level of confidence that a claim is true, from 0 to 1.
- Confidence: essentially a synonym for credence, as in “level of confidence,” instead of colloquial meaning, “state of having a lot of confidence.”
- Accuracy: How frequently one is correct; proximity to a true value.
- Calibration: How closely confidence and accuracy correspond; that is, how accurate a person or system is at estimating the probability that they are correct.
- Because every proposition comes with a degree of uncertainty:
- a. Partial and probabilistic information still has value.
- b. Back-up plans are important because no information is absolutely certain.
- c. It is important to invest in calibrating where you are more and less likely to be right, as opposed to being overinvested in being “right.”
- d. Scientific culture primarily uses a language of probabilities, not certain facts.
- e. Even correctly-done science will obtain incorrect results some of the time.
- B. CONCEPT APPLICATION
- Appropriately weigh uncertainty in decisions involving risk. Identify a reasonable threshold of confidence for a given decision.
- Recognize situations where confidence levels can high enough for risky action (e.g. sometimes confidence is high enough to bet your life or the lives of others, even without perfect certainty).
- Explain how the treatment of uncertainty in scientific work allows scientists to follow the truth, even when that means changing their minds.
-
LEARNING GOALS
- A. ATTITUDES
- Be wary when high degrees of confidence are claimed.
- Suspect something is amiss if no finding from a scientific community is ever retracted.
- B. CONCEPT ACQUISITION
- Confidence Interval: A range within which a true value of interest lies with a specified probability.
- Most commonly a 95% confidence interval, which means there is a 5% chance the true value lies outside the range specified.
- Error Bars: Smaller bars on a graph that show the range of likely true values around the observed value, typically a 95% confidence interval, or the observed value +/- the standard error or standard deviation.
- Scientific culture at its best reinforces the importance of uncertainty by offering respect and career advancement to people on the basis of calibration as well as accuracy. In attaching the ego to calibration as well as accuracy, this discourages scientists from being overly attached to their ideas being “right,” encouraging them to prioritize truth over having been right.
- People (including many experts) tend to over-estimate their accuracy at high confidence levels (and under-estimate it at low-confidence levels).
- People often use a source’s confidence as a cue to credibility, but appropriately discount confidence when they have evidence of poor calibration.
- C. CONCEPT APPLICATION
- Compare reliability of sources of information on the basis of their assiduousness in determining confidence levels and probabilistic ranges for results (e.g., confidence intervals, error bars).
- Recognize that for scientific findings that are presented as having 95% certainty (for example), 5% of such results should be incorrect.
- Not be fooled by criticisms of scientific communities for occasionally (~5% of the time, for example) having results later shown to be wrong.
- Identify higher/lower accuracy and better/worse calibration in concrete examples.
- Identify factors that lead to better calibration and use these factors to predict and suggest ways to improve a person’s calibration in a given scenario.
-
LEARNING GOALS
- A. ATTITUDES
- B. CONCEPT ACQUISITION
- Multiple Causation: Any given effect may be brought about by a complex combination of many causes (which may interact with each other), with varying degrees of influence on the outcome.
- Orders of Understanding: When there are multiple causes of a given outcome, it is often the case that some causes are much more impactful than others. In these cases, we draw a rough qualitative distinction between the cause(s) with the greatest impact for a given effect (first order cause/explanation), the causes with somewhat less impact (second order), and the much less influential causes (third and higher order).
- Effect Size: The size of the effect under examination. (e.g. how much being overweight affects health is the effect size of obesity on health).
- C. CONCEPT APPLICATION
- Distinguish first order, second order, & lower orders of explanation.
- Identify multiple causes for a given effect or outcome and evaluate the relative importance of each with respect to different contexts and goals.
- Given a concrete example, identify good and bad examples of evaluating and using the relative importance of causal factors.
- Given a concrete example, recognize and critique poor decisions that result from considering only the ranking - and not the relative magnitude - of causal factors.
- (For instance, differentiate a case with one huge first order factor and many tiny ones from one involving several factors of comparable and large magnitude.)
-
LEARNING GOALS
- A. ATTITUDES
- Be confident in one’s ability to make a reasonable magnitude estimate of quantities for which one has no intuitive guess or direct knowledge.
- B. CONCEPT ACQUISITION
- Fermi Estimates: A systematic estimate of a quantity based on what you know. The typical goal is to get within an order of magnitude of the right answer. (This often proves possible even for topics about which you know very little.)
- Steps:
- a. Decompose the problem into multiple components that you can estimate. (Break down unfamiliar components into familiar components).
- b. Estimate components using approximations.
- c. Combine estimated components to calculate Fermi estimate.
- d. Compute upper and lower bounds (maximum and minimum quantities above/below between which you are fairly confident the correct estimate should be).
- Order of Magnitude: factor of ten.
- C. CONCEPT APPLICATION
- Identify quantities that would and would not be appropriate to estimate with a Fermi calculation.
- Provide rough estimates for real-world quantities using “back-of-the-envelope” (Fermi) approximations.
- Evaluate the credibility of quantitative statements using “back-of-the-envelope” approximations.
- Use Fermi estimates to identify first, second, third order causes for example problems, and estimate their effect sizes.
-
LEARNING GOALS
- A. ATTITUDES
- Value processes of discovering and correcting errors, recognizing that formal statistics and logic are needed to counteract various types of (frequently unrecognized) errors, despite high confidence. Be aware of the pitfalls of human reasoning.
- B. CONCEPT ACQUISITION
- Availability heuristic: Cases in which people use how readily something comes to mind as a proxy for an estimate of its probability.
- Representativeness heuristic: Cases in which how representative something is of a category or outcome is used as a proxy /for evaluating how likely the category membership or outcome is (not taking base rates into account).
- Anchoring heuristic: Cases in which an estimate is made by anchoring on a number that’s provided and potentially irrelevant and adjusting, typically insufficiently.
- Base rate neglect: People frequently overlook the importance of base rates when calculating the probability of an event based on probabilities that seem more relevant to the specific case.
- Base rates: The base frequency of a given attribute in a whole population.
- a. Bayesian reasoning: Systematically combining information regarding new evidence with prior beliefs to determine the probability of a hypothesis.
- b. Bayes Rule: $$P(A|B) = (P(B|A)*P(A))/P(B)$$ where $$A$$ & $$B$$ are events and $$P(B)$$ ≠ $$0$$.
- The formal statistical rule for applying Bayesian reasoning. When updating beliefs, final credence level should be influenced by initial credence level and the strength of the new evidence.
- Peak-End Rule: The tendency to remember the peak, or highlight, and the very end of an experience, and to take them as more representative of the experience as a whole than they really are.
- C. CONCEPT APPLICATION
- Recognize and resist instances of the availability heuristic in everyday and scientific contexts.
- Recognize and resist instances of the representativeness heuristic in everyday and scientific contexts.
- Recognize and resist instances of the anchoring heuristic in everyday and scientific contexts.
- Recognize and resist instances of base rate neglect in everyday and scientific contexts.
- Given an example in which a person updates her belief, identify the two factors that should influence her final credence level (initial credence level and strength of the evidence), and recognize that Bayes rule provides a formal specification of how to do so.
-
LEARNING GOALS
- A. ATTITUDES
- Recognize the potential to abuse science for social and political ends.
- Show heightened caution in situations in which science involves the study of human groups and subsequent validation of societal power structures.
- Recognize that you are yourself always implicated in some social dynamic or other that may be relevant to the assessment of any particular study of human groups.
- B. CONCEPT ACQUISITION
- Historically, science has been abused for social and political ends; this is why we should show heightened caution (using all the tools of the course) whenever science is used to evaluate particular groups of people.
- The abuse of science for social and political ends is exacerbated by confirmation bias and badging.
- The abuse of science for social and political ends is particularly insidious when it involves the study of human groups and subsequent validation of societal power structures (typically by purporting to establish that some group(s) have lesser capacities than others).
- Just World Fallacy: The tendency to believe that outcomes are deserved and existing social structures are justified.
- C. CONCEPT APPLICATION
- Identify examples in which there exists the potential for abuse of science for social and political ends.
- Explain how confirmation biases and badging can exacerbate the abuse of science for social and political ends in particular cases.
- Explain why the potential to abuse science for social and political ends is particularly insidious when it involves the study of human groups and subsequent validation of societal power structures, using a real example.
- Explain how the validity of human classifications (e.g., race, gender) can be problematic and contribute to such abuses (e.g., because there isn’t any there, there are a number of things there, there is one thing there but it doesn’t have the significance you take it to).
- Provide historical examples of the abuse of science for social and political ends.
- Provide examples of cases in which the study of human groups is legitimate and beneficial.
- Distinguish legitimate inquiry into human groups from bad science.
-
LEARNING GOALS
- A. ATTITUDES
- B. CONCEPT ACQUISITION
- The boundaries demarcating science from non-science and distinguishing among the categories of pathological science, pseudo-science, fraudulent science, poorly-done science, and good science can often be difficult, with overlapping and fuzzy boundaries between categories.
- Pathological Science Indicators:
- a. The effect is produced by a barely detectable cause, and the magnitude of the effect is substantially independent of the intensity of the cause.
- b. The effect is barely detectable, or has very low statistical significance. Claims of great accuracy.
- c. Involving fantastic theories contrary to experience.
- d. Criticisms are met with ad hoc excuses.
- e. Ratio of supporters to critics rises to near 50%, then drops back to near zero.
- f. Conclusion-motivated design & analysis.
- g. Pseudo-science is characterized by using scientific vocabulary without aligning with the corresponding concepts or engaging in real scientific practices (i.e., science being “skin deep,” not scientific below the surface).
- Fraudulent science involves intentional deception, such as deliberately fabricating data or deliberately deceiving the reader about the strength of evidence.
- Poorly-done science, e.g. failure to consider confounds, failure to use best practices in terms of data collection and analysis (e.g., small sample size, look elsewhere effect).
- Unintentional self-deception can be involved in justifying poor practices and/or interpretations in pathological, pseudo-, & poorly-done science.
- Motivation to support a particular conclusion (i.e., science undertaken to support a given conclusion, rather than to discover the truth) can be a feature of poorly done or pathological science.
- Good science:
- a. Will get the wrong answer some of the time, e.g., via statistical flukes.
- b. Entails good faith engagement with the alternative hypotheses through a search for evidence that you are wrong.
- C. CONCEPT APPLICATION
- Distinguish science from enterprises such as religion or (perhaps) astrology where the attempt is not to find descriptive adequacy but meaning in ordinary life.
- Identify cases of good science that gets the wrong answer, fraudulent science, pathological science, poorly-done science, and pseudo-science based on the above characteristics.
- Identify what is wrong in cases of fraudulent, pathological science, poorly-done, and pseudo-science.
-
LEARNING GOALS
- A. ATTITUDES
- Be wary of one’s own tendency towards motivated reasoning and confirmation bias.
- B. CONCEPT ACQUISITION
- Confirmation bias: Seeking or otherwise favoring evidence consistent with what is already believed or what is being tested.
- a. Selective exposure: Selectively seeking or exposing oneself to evidence that is likely to conform to prior beliefs or working hypotheses.
- b. Biased assimilation: Systematically favoring or discounting evidence to render that evidence more compatible with pre-existing beliefs or working hypotheses.
- C. CONCEPT APPLICATION
- Recognize instances of “selective exposure” in scientific, policy, and everyday contexts.
- Recognize instances of “biased assimilation” in scientific, policy, and everyday contexts.
- Explain how different forms of confirmation bias (i.e., selective exposure, biased assimilation) can lead to errors.
- In a given instance of confirmation bias, identify the following components:
- a. Prior beliefs/working hypothesis affecting reasoning,
- b. Evidence/interpretations that would be favored,
- c. Evidence/interpretations that would be missing/discounted, and
- d. How the confirmation bias might affect the resulting conclusions.
-
LEARNING GOALS
- A. ATTITUDES
- One should always be looking for ways that we get things wrong (by fooling ourselves or due to bugs in our reasoning processes) so that we can invent better procedures.
- This is important both for methods long in use and new ones (e.g. big data).
- B. CONCEPT ACQUISITION
- Blind analysis: Making all decisions regarding data analysis before the results of interest are unveiled, such that expectations about the results do not bias the analysis. Usually co-occurs with a commitment to publicize the results however they turn out.
- Examples of analysis decisions for which blind analysis could be useful:
- a. Stopping rules for when to stop looking for flaws in your experimental design or for computer/math bugs.
- b. Data selection decisions.
- c. Decisions about which analysis procedures to use (e.g. grouping decisions).
- Confirmation bias drives the need for blind analysis.
- Confirmation bias is pervasive and doesn’t necessarily indicate any fraudulent activity.
- Approaches to reducing confirmation bias other than blind analysis:
- a. Preregistration: A research group publicly commits to a specific set of methods and analyses before they conduct their research.
- b. Registered replication: [A] research group(s) commits to a specific set of methods and procedures to verify the result of an earlier work (typically with the input of the original research team). Results are publicized regardless of outcome.
- c. Adversarial collaboration: Scientists with opposing views agree to all the details of how data should be gathered and analyzed before any of the results are known.
- d. Peer review: New results are evaluated by other experts in the same field to determine whether they are valid. This only reduces confirmation bias if reviewers don’t share biases.
- Scientists are constantly looking for bugs in scientific practices in order to fix them. Blind analysis is just the latest example of scientists recognizing a bug in their practice (e.g., a way of being fooled) and adjusting practice to account for/remove the bug.
- C. CONCEPT APPLICATION
- Explain why blind analysis might be needed, by explaining the errors that might arise in its absence.
- Recognize when blind analysis is being used and explain what function it serves. Identify situations and decisions that would call for blind analysis.
- Evaluate techniques (e.g., registered replication, adversarial collaboration, peer review) - a) for ability to address confirmation bias and b) in comparison to blind analysis.
- Propose how to use blind analysis for simple studies.
- Stretch Goal: Propose new practices to solve a new (fictional) problem with current scientific practice. (This is a stretch goal for evaluation, too! Can we invent a new problem with a new scientific practice?)
-
LEARNING GOALS
- A. ATTITUDES
- Not take for granted that consensus offers the best conclusions.
- Take seriously (but not as absolute!) the consensus of a group which has reasoned about a question in a careful, appropriate way.
- Take seriously (but not as absolute!) the average of a large group's independent estimates of a number, under appropriate conditions.
- B. CONCEPT ACQUISITION
- Wisdom of Crowds: Sometimes groups make better judgments than individuals. This happens when:
- a. Judgments are genuinely independent, preventing herd thinking.
- b. Members of the group do not share the same biases.
- c. There are enough people in the group to balance out random biases or fluctuations (analogous to the need for an adequate sample size).
- d. Works especially well when estimating a quantity, where errors may be large but are not systematic.
- Herd Thinking: Sometimes groups make worse judgments than individuals. This happens when:
- a. Judgments of individuals are influenced by the judgments of others, leading to groupthink and sometimes polarization.
- b. Members of the group share biases, which can be exaggerated by discussion and cannot be decreased by averaging judgments.
- The enterprise of science is essentially social, and advances in part because scientists look actively for what other scientists might have gotten wrong. This process, including peer review, enables science to iteratively improve.
- C. CONCEPT APPLICATION
- Recognize when groups are likely to make good decisions.
- Recognize when groups are likely to make poor decisions.
- Structure group decision-making processes so as to maximize the benefits and minimize the dangers.
- Identify features of existing group decision-making practices which we could improve.
-
LEARNING GOALS
- A. ATTITUDES
- Acknowledge the inevitability of arational elements like fears and internally conflicting desires as part of the human condition.
- Acknowledge the need for an effective decision-making scaffold to cope with the inevitable arational elements at play.
- Recognize that arational elements like fear and ambition are not just unfortunate, inherent parts of decision-making, but can be useful to motivate people to solve problems.
- Respect for value pluralism (multiple principles & preferences).
- Respect for the difficulty and importance of figuring out priorities in light of personal and group value conflicts, especially in heterogeneous groups.
- Optimism about the possibility of considering and compromising on conflicting values and other arational motivators in a principled way.
- B. CONCEPT ACQUISITION
- Desiderata for Decision Making Process: Aspects of an ideal decision-making process.
- e.g. not ignoring crucial facts about the world, fairness, good outcomes, buy-in from constituents, etc.
- Value: In the philosophical and psychology of judgment and decison-making literature, we use the term "value" to refer to all values, goals, preferences, fears, etc., not only high, principled values (as often in colloquial speech).
- Value Pluralism: People often have multiple legitimate values (including both high principles and minor personal preferences), which sometimes conflict (within individuals or within groups).
- When an individual has multiple conflicting values, they often try to avoid compromising either value by:
- a. Buck-passing: Getting someone else to make the decision, so one doesn't feel responsible for compromising one of their values.
- b. Belief Overkill: Denying the facts that lead to value conflict.
- E.g., the large majority of people who think the death penalty is morally wrong also believe it is not an effective deterrent, while those who think it is morally appropriate believe it is an effective deterrent. Someone who believed it was effective but immoral would be faced with an internal conflict between their desire to deter murderers and their dislike of the death penalty.
- c. Sacred Values: Some values, which are considered "sacred" or "protected," people say they will not compromise for anything. This is not usually followed through all the way.
- E.g., saving children's lives is often considered a sacred value, while saving money is not. However, essentially no one contributes all the money they could to institutions which save children's lives and could save more lives with more money.
- d. Tragic Trade-offs: Sometimes a trade-off must be made between two sacred values, which makes for a tragic trade-off.
- E.g., deciding which of two children will receive a single life-saving organ needed by both to survive is a tragic trade-off.
- The value conflicts that arise from multiple values make people uncomfortable, but they are inevitable and coping with such conflicts requires compromise.
- C. CONCEPT APPLICATION
- Distinguish between technical, more inclusive use of "value" vs. narrower, colloquial sense of "value," and use appropriately.
- Identify multiple relevant values for individuals, for particular problems of what to do.
- Identify multiple relevant values for groups, for particular problems of what to do.
- Identify when value trade-offs are necessary, and when they can be minimized.
- Identify one's own fears, personal desires, and ambitions which come into play in a given group decision.
- Identify the fears, personal desires, and ambitions of others which come into play in a given group decision.
-
- LEARNING GOALS
- A. ATTITUDES
- Be optimistic that a community can come together to make a decision, even when people begin with heterogeneous values and beliefs.
- B. CONCEPT ACQUISITION
- Stakeholders: The set of people who have a stake in the outcome of a decision. This can include people who will implement the decision and all the people affected by it.
- Experts: The set of people who have the most knowledge/information/expertise about the facts relevant to the decision.
- Denver Bullet Study: An experiment in group deliberation in which a community came together to share values and knowledge to make a decision about what kind of bullet the Denver Police should use, which had enough stopping power to keep cops safe but was not so harmful as to cause unnecessary damage to citizens (as did classic hollow bullets).
- C. CONCEPT APPLICATION
- Identify relevant stakeholders.
- Identify relevant questions of fact for which experts can be consulted. .C
- Identify problems for which the Denver Bullet Study approach would be helpful.
- Describe a Denver Bullet Study approach to solving a novel problem.
-
LEARNING GOALS
- A. ATTITUDES
- Be optimistic about the possibility of integrating the perspectives and insights of both experts and stakeholders in forming effective policies with community buy-in.
- B. CONCEPT ACQUISITION
- Deliberative Polling: A system of decision-making wherein a representative group of stakeholders come together, exchange and discuss information, question experts, and poll their individual opinions at various points throughout the session.
- C. CONCEPT APPLICATION
- Recognize problems which could be resolved effectively using deliberative polling.
- Plan deliberative polls, including:
- a. Identifying relevant stakeholders.
- b. Identifying relevant experts.
- c. Identifying important types of factual information & questions.
- d. Proposing appropriate questions to ask in the polls.
-
LEARNING GOALS
- A. ATTITUDES
- Avoid assuming the future will continue in one direction, and ready to consider a variety of possible futures when planning.
- B. CONCEPT ACQUISITION
- Scenario Planning: A mode of problem solving which involves considering two important and uncertain dimensions along which the future might vary, imagining what each possible quadrant might look like, and considering how decisions made now will affect the likelihood and desirability of each quadrant.
- C. CONCEPT APPLICATION
- Identify problems which could be informed by scenario planning. .C
- Use scenario planning to inform reasoning about major decisions, including:
- a. Identifying important & uncertain dimensions along which the future might vary.
- b. Identifying the pros and cons of each quadrant of possibility.
- c. Identifying how decisions made now might affect each quadrant.
-
- LEARNING GOALS