a. Facts: Objectively true claims about reality. Everything that is the case. What is, descriptively, including spatial relations, causal relations, attributes of objects, etc.

b. Values: What is of value, important, of worth. Oughts, shoulds, etcs.

a. Democracy: A system of government wherein a society’s citizens have more or less equal input into policies.

b. Epistocracy: A system of government wherein a particular subset of a society—privileged by their education or other markers of expertise—decides policies.

a. Distinguish between facts and values in public debates and everyday decision-making.

b. Identify ways in which facts and values are intertwined in public debates and everyday decision-making.

c. Recognize when values determine which facts are viewed as relevant.

d. Use distinction between facts and values to identify appropriate source(s) of authority in decision-making.

a. Explain arguments for each.

b. Describe tension between the claims of the two.

c. Appeal to scientific expertise (including in the social and behavioral sciences) for political decision-making.

d. Identify democratic and epistocratic aspects of real governments.

Scientists assume an external reality, which is shared by and affects all people and has enough regularity to lend itself to induction. This external reality is what scientists seek to describe accurately.

Science is based on appeal to empirical evidence, which is publicly accessible on the assumption of reality (although may require special instruments and/or expertise to acquire).

The extent to which a scientific concept is responding to some real external thing.

Science gains its authority from its self-questioning character, not from the concentrated power of individuals.

a. Active experimentation & observation

b. Peer review  

c. Rewards for better theories, even those that contradict current theories  

d. Responsiveness to new evidence/actively open-minded thinking  

e. Replicability and replication  

f. Multiple approaches to each problem allow convergent evidence  

g. Interconnected nature of science, and ongoing attempts to connect the pieces that are not yet connected (e.g. Biological Synthesis of genetics & natural selection as a successful instance, cognitive neuroscience as an instance of integration currently in progress, the challenge of connecting quantum physics to general relativity…). 

a. The Raft: Every scientific claim is subject to question and reevaluation; we can use the rest of our scientific knowledge to question any one claim at a time, though we cannot question the entire edifice at once. 

b. The Pyramid: Science builds on fixed foundations to ever higher levels of knowledge. 

a. Operationalism

b. Conventionalism

c. Realism

“The reality [of a scientific entity or fact] is formed as a consequence of stabilization [of a controversy].” (Latour & Woolgar, 1986) 

The phenomenon of people using claims of fact to express their identity or group affiliation. 

e.g. Some scientific theories are more central than others, and harder to replace. But all scientific theories are, in principle, open to revision in light of new evidence. 

a. Identify cases where concept validity is expected 

b. Identify cases where social constructivism might be a good approach 

c. Identify cases where preference might be sufficient 

a. In straightforward cases 

b. In less straightforward cases 

c. In difficult cases 

e.g. Stated acceptance of creationism and rejection of evolution is only very weakly responsive to education and strongly associated with affiliative factors like religion, religiosity, and political ideology, suggesting that it may be (to some extent) a result of badging.

a. When there is no objective gold standard (e.g. passage of time, what fluid to use in a thermometer) 

b. When direct observation is messy or impossible (e.g. radio waves, size) 

a. Interactive exploration: Testing an instrument by changing the thing it is measuring in ways you know through other means, and seeing if the instrument recognizes the changes appropriately (e.g. does driving increase a car's odometer; see how singing higher and lower notes affects a sound spectrograph; sprayable electrons in Hacking reading).  

b. Comparison of multiple instruments (e.g. thermometers)  

c. Comparison to direct observation (e.g. naked sight compared to sight with a magnifying glass)  

a. All measurements have a certain amount of variance, which are just differences between multiple measurements due to error and/or genuine variation in the sample. These differences will not all go in the same direction.  

b. Statistical uncertainty can be reduced by averaging a larger amount of data.  

a. Such measurements will show a consistent bias, that is, a consistent deviation from reality in one direction.  

b. Systematic uncertainty cannot be reduced by averaging a larger amount of data.  

a. A causes B (direct causation);

b. B causes A (reverse causation);

c. A and B are both caused by C

d. A causes B and B causes A (bidirectional or cyclic causation);

e. There is no connection between A and B; the correlation is a coincidence.

f. The effect of A on B depends on C

a. Experimental Intervention: The act of an experimenter changing one variable in a possible causal network. 

b. Randomized Assignment: Given sufficient sample size, randomized assignment rules out confounds by  distributing variation randomly between the two groups, thereby avoiding systematic differences except as the result of the intended intervention. 

c. Control Condition: Comparison of an experimental to a control condition is necessary in order to distinguish effect of intervention from changes that would have occurred without the intervention.  

d. Sampling: A study of a well-chosen sample can tell you something about the population (through induction), especially if it was selected in such a way as to avoid any systematic differences between the sample and the rest of the population. It is often difficult or even impossible to capture a perfectly representative sample, so scientists do the best they can. For example, many psychology studies are done with college students because they are accessible, but such samples differ systematically from the general population. Inferences from samples to a larger population need to take such differences into account.

a. This is a technical notion, which overlaps with but is slightly different from everyday usage. For example, everyday usage of the word “cause” can be influenced by moral considerations, the complexity of a causal mechanism, and/or the nature of the mechanism. We typically don’t say that the big bang "caused" this sequence of letters, or that the presence of oxygen caused the forest fire, etc, but scientifically they are part of the causal history of those phenomena. 

b. There can be other evidence for causation, even when actually performing an intervention is not feasible. However, saying something is a cause implies that there is in principle a relationship under an intervention.  

a. Prior plausibility: Can a plausible mechanism be constructed, or is there some other basis for interpreting the current evidence in terms of one causal structure over another, such as data from other studies?  

b. Temporality/temporal sequence: Did the hypothesized cause precede the effect? 

c. Dose-response curve: Do the quantities of the hypothesized cause correlate with the quantity, severity, or frequency of the hypothesized effect across ranges?  

d. Consistency across contexts: Does the correlation appear across diverse contexts? 

Noise is frequently, but not always, the result of random measurement fluctuations.   

a. Asks too many questions of the same data set, reporting only statistically significant results.  

b. Asks the same question of multiple data sets, reporting only statistically significant results. 

c. Runs a test or similar tests too many times, reporting only statistically significant results. 

d. This also occurs in everyday life, e.g. when one looks at a whole lot of phenomena and only takes note of the most surprising-looking patterns, not properly taking into account the larger number of unsurprising patterns/lack of pattern.  

Lack of statistical significance does NOT prove the null hypothesis.

e.g., not be impressed by statements made with 100% confidence, no error bars or confidence intervals on claimed measurements, seeking definitive answers when only probabilistic information is available, not recognizing that probabilistic information is better than no information.

a. Partial and probabilistic information still has value.  

b. Back-up plans are important because no information is absolutely certain.  

c. It is important to invest in calibrating where you are more and less likely to be right, as opposed to being overinvested in being “right.”  

d. Scientific culture primarily uses a language of probabilities, not certain facts.  

e. Even correctly-done science will obtain incorrect results some of the time.  

Most commonly a 95% confidence interval, which means there is a 5% chance the true value lies outside the range specified.

(For instance, differentiate a case with one huge first order factor and many tiny ones from one involving several factors of comparable and large magnitude.)

Steps:

a. Bayesian reasoning: Systematically combining information regarding new evidence with prior beliefs to determine the probability of a hypothesis.

b. Bayes Rule: $$P(A|B) = (P(B|A)*P(A))/P(B)$$ where $$A$$ & $$B$$ are events and $$P(B)$$ ≠ $$0$$.

a. The effect is produced by a barely detectable cause, and the magnitude of the effect is substantially independent of the intensity of the cause.  

b. The effect is barely detectable, or has very low statistical significance. Claims of great accuracy. 

c. Involving fantastic theories contrary to experience. 

d. Criticisms are met with ad hoc excuses. 

e. Ratio of supporters to critics rises to near 50%, then drops back to near zero.  

f. Conclusion-motivated design & analysis. 

g. Pseudo-science is characterized by using scientific vocabulary without aligning with the corresponding concepts or engaging in real scientific practices (i.e.,  science being “skin deep,” not scientific below the surface). 

a. Will get the wrong answer some of the time, e.g., via statistical flukes.  

b. Entails good faith engagement with the alternative hypotheses through a search for evidence that you are wrong.  

a. Selective exposure: Selectively seeking or exposing oneself to evidence that is likely to conform to prior beliefs or working hypotheses.

b. Biased assimilation: Systematically favoring or discounting evidence to render that evidence more compatible with pre-existing beliefs or working hypotheses.

a. Prior beliefs/working hypothesis affecting reasoning, 

b. Evidence/interpretations that would be favored, 

c. Evidence/interpretations that would be missing/discounted, and 

d. How the confirmation bias might affect the resulting conclusions.  

This is important both for methods long in use and new ones (e.g. big data).

a. Stopping rules for when to stop looking for flaws in your experimental design or for computer/math bugs.  

b. Data selection decisions. 

c. Decisions about which analysis procedures to use (e.g. grouping decisions). 

a. Preregistration: A research group publicly commits to a specific set of methods and analyses before they conduct their research. 

b. Registered replication: [A] research group(s) commits to a specific set of methods and procedures to verify the result of an earlier work (typically with the input of the original research team). Results are publicized regardless of outcome. 

c. Adversarial collaboration: Scientists with opposing views agree to all the details of how data should be gathered and analyzed before any of the results are known. 

d. Peer review: New results are evaluated by other experts in the same field to determine whether they are valid. This only reduces confirmation bias if reviewers don’t share biases. 

a. Judgments are genuinely independent, preventing herd thinking. 

b. Members of the group do not share the same biases. 

c. There are enough people in the group to balance out random biases or fluctuations (analogous to the need for an adequate sample size). 

d. Works especially well when estimating a quantity, where errors may be large but are not systematic. 

a. Judgments of individuals are influenced by the judgments of others, leading to groupthink and sometimes polarization. 

b. Members of the group share biases, which can be exaggerated by discussion and cannot be decreased by averaging judgments. 

e.g. not ignoring crucial facts about the world, fairness, good outcomes, buy-in from constituents, etc.

a. Buck-passing: Getting someone else to make the decision, so one doesn't feel responsible for compromising one of their values.

b. Belief Overkill: Denying the facts that lead to value conflict.

c. Sacred Values: Some values, which are considered "sacred" or "protected," people say they will not compromise for anything. This is not usually followed through all the way.

d. Tragic Trade-offs: Sometimes a trade-off must be made between two sacred values, which makes for a tragic trade-off.

a. Identifying relevant stakeholders.  

b. Identifying relevant experts.  

c. Identifying important types of factual information & questions.  

d. Proposing appropriate questions to ask in the polls.  

a. Identifying important & uncertain dimensions along which the future might vary. 

b. Identifying the pros and cons of each quadrant of possibility. 

c. Identifying how decisions made now might affect each quadrant.