Filtering on

When is science relevant? The many uses of a scientific approach.  

What is the role of scientific expertise in a democracy? Where does the authority of science come from? Where does the authority of democratic decision-making come from?  

Science is based in the assumption that we all share a public reality. We will contrast theories of truth as correspondence vs truth as coherence, as well as underdetermination and social factors in science. Despite many limitations, science is effective primarily because it is self-correcting; that is, it involves a constant critique of the reliability and validity of our measures and the reality of the entities they seek to measure. Science assumes (or has come to believe in) an objective reality that we all share, which is at least in part knowable.   

It has been suggested that our trust in "a reality out there" is often strengthened by our actively interacting with the passively-percieved world (banging the table in front of us with our hand). Our “direct” experience with a scientist’s reality is expanding further: from the human senses to the human armed with instruments. The novel instruments that are now with us constantly (e.g. GPS, camera) allow us to interactively explore parts of the world that until recently were inaccessible or accessible only passively, through expensive technology or images made by scientists. This interactive experience of previously inaccessible aspects of the world now revealed by technology is broadening our sense of what counts as "real." We carry with us a growing range of interactive tools, these days primarily in the compact form of our smartphones. 

Class Exercises

We examine sources of statistical uncertainty/error (which can be averaged down) and systematic uncertainty (which can’t). We also connect these concepts to related terminology (jargon) from other fields: precision vs. accuracy, variance vs. bias, and reproducibility vs. validity. 

Class Exercises

“Scientific optimism” is a rarely-discussed feature of the culture of science, a kind of psychological trick/technique to keep focused on a problem much longer than the usual attention span. Scientists adopt a can-do attitude, and convince themselves that the problem is solvable. This is an antidote and a contrast to almost all of the other skeptical, self-doubting aspects of scientific culture. With this “scientific optimism,” scientists can successfully take on problems that take years or even decades to solve, with hundreds of steps and iterations involved in developing techniques, inventing technologies, collecting and analyzing data.

There is a history of problems becoming solvable once the news goes round that another group somewhere in the world has solved it. Belief that a problem is solvable makes it worth sticking to it long enough to solve it. Scientific optimism can be seen as an intentional self-delusion that a problem is solvable. (In the end, of course, this scientific optimism must be weighed against the cost of working on a problem that turns out not to be solvable given our current capabilities/knowledge, but nevertheless it has proven useful in overcoming the human tendency to give up too soon.)  

Clicker Questions

Class Exercises

An introduction to the scientific approach to determining causal relationships.

In this class, we will examine causal relationships using variables, interventions,and randomized controlled trials.

Class Exercises

Building on Correlation and Causation, we examine how to collect evidence for causality in more difficult cases.

Defining causal relationships using "variables" and "interventions." How we say that this particular thing caused that particular thing. Connections with the Trolley Problem and legal responsibility. Causal relations in observational sciences (e.g., paleontology, cosmology) where experiments are not generally possible. 

What does a scientist mean by “signal” and “noise”? We humans are always hunting for signal in noise; that is, we are looking for regularities, causal relationships and communications (the signal) amidst various distractions, both random and intentional (the noise). Scientists have developed a variety of ways to do this, including “filters" both technological and conceptual. 

Clicker Questions

Class Exercises

Humans are so good at finding signals in noise that sometimes they do so even when there is no signal. Many techniques of science and much of statistics is aimed at avoiding fooling yourself this way. A further problem is that we often aren’t aware of how much noise we have searched through, when we believe we have found a signal—the “Look Elsewhere Effect." For example, we tend to think coincidences are meaningful. Statistics was invented primarily to deal with the problem of distinguishing real signal from noise fluctuations that look like signal.

Clicker Questions

Discussion Questions

Class Exercises

An important element of the culture of science is the use of “tentative” propositions, often quantified. These can be as confident as 99.99999%—you would bet your life on it—but it would still be understood to be held as a proposition which could be wrong. This makes it psychologically easier for a scientist to be open to being wrong—and to look actively for ways they might have gotten it wrong. This cultural understanding of the importance of recognizing and reporting one's credence level leads to insistence on including error bars on graphs: a data point is completely meaningless without an error bar.  

Class Exercises

Most people’s estimates of their confidence is wrong in characteristic ways: high confidence tends to be over-estimated and low confidence tends to be under-estimated. This can be trained to be closer to an accurate calibration, most effectively with repeated, unambiguous, and immediate feedback. One problem that arises from poor calibration is that juries often use witness’ confidence to gauge the likelihood that they are correct, but this often yields poor results due to the poor calibration of the witnesses. 

Class Exercises

A useful “jargon” of the scientists is to speak of orders of understanding or orders of explanation. “A zeroth order explanation/cause” or “a first order explanation/cause,” is a major cause/factor, as opposed to “a second order” or “third order explanation/cause," which would be real causes with smaller effect sizes. This is useful because explanations for how things/actors in the world behave can often be parsed into a primary explanation, a secondary less important cause, a third order, still less significant cause, etc.

Clicker Questions

Discussion Questions

Physicists train their students in doing “Fermi problems,” back-of-the-envelope estimates of quantities that arise in physics problems and in life. This is useful as an approach to performing “sanity checks” of claims in the world and of your own ideas, beliefs, and inventions. Checking numbers with quick Fermi estimates may be even more important in a world in which it is difficult to evaluate the credibility of numbers available by Googling. 

Class Exercises

Here we will explore the most widespread heuristics and biases which psychologists of judgment and decision-making have discovered in everyday reasoning: the availability heuristic, representativeness heuristic, and anchoring heuristic, and biases like optimism bias, hindsight bias, and status quo bias. Many examples are drawn from Daniel Kahneman's book, Thinking Fast and Slow.

Class Exercises

Science has a particularly bad track record when it comes to studies of human sub-populations for the purpose of setting policy—particularly when groups in power study groups out of power. We should be aware of this, and wary of misusing science in such a way as to perpetuate injustice.

Distinguishing pathological science, pseudo-science, fraudulent science, poorly-done science, good science that happens to get the wrong answer (which should happen statistically for 1 in 20 papers that give a 0.05 confidence level result). What do practicing scientists do when they try to judge a paper in a field or sub-field outside their immediate area of expertise? 

Class Exercises

Our tendency to preserve our existing or preferred beliefs, even against the evidence.

This class explores confirmation bias in the search for and assessment of evidence. In particular, we consider the ways that people tend to seek out and think about evidence in such a way as to reinforce their existing opinions, rather than testing them against new information or alternative views.

Science is not a single “scientific method” (as often taught in school), but better characterized as an ever-evolving collection of tricks and techniques to compensate for our mental (and, occasionally, physical) failings and build on our strengths—and, in particular, to help us avoid fooling ourselves. These techniques must constantly be re-invented, as we develop new ways to study and explain the world. In the last few decades we have entered a period in which most scientific analyses are complicated enough to require significant debugging before a result is clear. This has exposed another way we sometimes fool ourselves: the tendency to look for bugs and problems with a measurement only when the result surprises us. Where previously we recognized the need for “double blind” experimentation for medical studies, now some fields of science have started introducing blind analysis, where the results are not seen during the development and debugging of the analysis—and there is a commitment to publish the results, however they turn out, when the analysis is “un-blinded” and the results interpreted. 

Clicker Questions

Class Exercises

Explore ways that groups fall short of their optimal reasoning ability. There are better and worse ways to aggregate a group’s knowledge. 

Sometimes groups of people reach better conclusions than people working independently, and sometimes they reach worse conclusions. There are features of group reasoning that can help, and features of group reasoning that can hurt. Here we explore how to avoid the pitfalls of group reasoning and to maximize the benefits.

Class Exercises

How should (and shouldn't) values/emotions/goals/desires and conflicts of interest properly be woven together with science's hyper-rational elements in decision making processes? 

Students work out a problem involving values and factual/scientific issues, using method used in Denver bullet study. 

In the two classes of this week, we try out an approach to random-sample-representative decision-making, using a panel of experts to answer questions generated by small deliberative groups, each with a moderator. We also have a presentation from the professor who developed this technique, describing its use around the world. One example topic we have used is Fracking for Natural Gas.  

Class Exercises

Adversarial vs. Inquisitorial Modes of Truth-Seeking