Filtering on

Saul's Potpourri Overview of the Course

Note: This syllabus is annotated by Saul Perlmutter for the use of teachers of this course, and includes some “spoilers” for the student exercises. Therefore, please share only with faculty, not with students, and do not post/email publicly.

    • OVERVIEW

      • When is science relevant? The many uses of a scientific approach.  
      • What is the role of scientific expertise in a democracy? Where does the authority of science come from? Where does the authority of democratic decision-making come from?  
    • OVERVIEW

      • Science is based in the assumption that we all share a public reality. We will contrast theories of truth as correspondence vs truth as coherence, as well as underdetermination and social factors in science. Despite many limitations, science is effective primarily because it is self-correcting; that is, it involves a constant critique of the reliability and validity of our measures and the reality of the entities they seek to measure. Science assumes (or has come to believe in) an objective reality that we all share, which is at least in part knowable.   
    • OVERVIEW

      • It has been suggested that our trust in "a reality out there" is often strengthened by our actively interacting with the passively-percieved world (banging the table in front of us with our hand). Our “direct” experience with a scientist’s reality is expanding further: from the human senses to the human armed with instruments. The novel instruments that are now with us constantly (e.g. GPS, camera) allow us to interactively explore parts of the world that until recently were inaccessible or accessible only passively, through expensive technology or images made by scientists. This interactive experience of previously inaccessible aspects of the world now revealed by technology is broadening our sense of what counts as "real." We carry with us a growing range of interactive tools, these days primarily in the compact form of our smartphones. 
    • CLASS ELEMENTS

      • Class Exercises
        • Diffraction-grating glasses
        • IPhone app provides interactive sound spectrogram (Spectogram Pro). Use slide whistle, stringed instrument, whistle (interactive exploration high vs. low), difference in timbre between male & female voices. What differences does the spectrograph instrument show between these sounds? How do these differences map onto differences you can hear? What does the spectrograph show that you can't know just by listening? Do you believe that what the spectrograph shows that you can't hear is real? Why or why not?
        • iPhone app (Vernier Video Physics) shows quantitative analysis of slices of time after videotaping the movement of a tossed ball (falling and bouncing).
        • A contrasting example (i.e., without much of the interactivity needed to make the "reality" evident): CO2 concentration in room is measured over the course of the class, and the resulting graph of CO2 over time is shown on the projector. You should see a significant increase. Graph should indicate increasing levels of CO2 throughout the class, as the students filling the room fill it with CO2.#h
    • OVERVIEW

      • We examine sources of statistical uncertainty/error (which can be averaged down) and systematic uncertainty (which can’t). We also connect these concepts to related terminology (jargon) from other fields: precision vs. accuracy, variance vs. bias, and reproducibility vs. validity. 
    • CLASS ELEMENTS

      • Class Exercises
        • Students line up in “human histograms,” demonstrating statistical dispersion and systematic bias.
        • Small group discussions, clicker-questions, and exercises to have students identify the statistical versus the systematic uncertainties in a number of scenarios. 
    • OVERVIEW

      • “Scientific optimism” is a rarely-discussed feature of the culture of science, a kind of psychological trick/technique to keep focused on a problem much longer than the usual attention span. Scientists adopt a can-do attitude, and convince themselves that the problem is solvable. This is an antidote and a contrast to almost all of the other skeptical, self-doubting aspects of scientific culture. With this “scientific optimism,” scientists can successfully take on problems that take years or even decades to solve, with hundreds of steps and iterations involved in developing techniques, inventing technologies, collecting and analyzing data.
      • There is a history of problems becoming solvable once the news goes round that another group somewhere in the world has solved it. Belief that a problem is solvable makes it worth sticking to it long enough to solve it. Scientific optimism can be seen as an intentional self-delusion that a problem is solvable. (In the end, of course, this scientific optimism must be weighed against the cost of working on a problem that turns out not to be solvable given our current capabilities/knowledge, but nevertheless it has proven useful in overcoming the human tendency to give up too soon.)  
    • CLASS ELEMENTS

      • Clicker Questions
        • What is the second-longest that you have ever spent trying to solve a problem/puzzle?
      • Class Exercises
        • Spinning cylinders: A challenging puzzle (involving spinning a piece of plastic tubing, with markings on it) is presented to the students. The experimental conditions end up giving experiential demonstration of the usefulness of scientific optimism.  
    • OVERVIEW

      • An introduction to the scientific approach to determining causal relationships.
      • In this class, we will examine causal relationships using variables, interventions,and randomized controlled trials.
    • CLASS ELEMENTS

      • Class Exercises
        • Online Exercise: Causality Lab   
    • OVERVIEW  

      • Building on Correlation and Causation, we examine how to collect evidence for causality in more difficult cases.
    • OVERVIEW

      • Defining causal relationships using "variables" and "interventions." How we say that this particular thing caused that particular thing. Connections with the Trolley Problem and legal responsibility. Causal relations in observational sciences (e.g., paleontology, cosmology) where experiments are not generally possible. 
    • OVERVIEW

      • What does a scientist mean by “signal” and “noise”? We humans are always hunting for signal in noise; that is, we are looking for regularities, causal relationships and communications (the signal) amidst various distractions, both random and intentional (the noise). Scientists have developed a variety of ways to do this, including “filters" both technological and conceptual. 
    • CLASS ELEMENTS

      • Clicker Questions
        • Students identify “noise” and “signal” in different situations (the same event can be “signal” or “noise”depending on context).
      • Class Exercises
        • Playing a sound with Morse code signal hidden in static. Demonstrate how our ear/brain is highly developed to find the signal.  
        • Students write down a short phrase that they proceed, by stages, to hide in more and more noise (random substituted letters). Show the concept of “signal-to-noise ratio” as away to quantify at what point they can no longer recognize the message (the signal).  
    • OVERVIEW

      • Humans are so good at finding signals in noise that sometimes they do so even when there is no signal. Many techniques of science and much of statistics is aimed at avoiding fooling yourself this way. A further problem is that we often aren’t aware of how much noise we have searched through, when we believe we have found a signal—the “Look Elsewhere Effect." For example, we tend to think coincidences are meaningful. Statistics was invented primarily to deal with the problem of distinguishing real signal from noise fluctuations that look like signal.
    • CLASS ELEMENTS

      • Clicker Questions
        • Students identify Look Elsewhere Effect mistakes in various scenarios (including medical studies). 
      • Discussion Questions
        • Students discuss Look Elsewhere Effect mistakes in various scenarios, with Clicker Questions.
      • Class Exercises
        • Professor leaves room and students write down two lists of 40 coin-toss results: “heads, tails, tails, heads...,” the first generated by students sequentially calling out “heads” or “tails,” trying to simulate random coin flips and the second by actually flipping coins. The professor returns, and has to guess which is random and which is simulated random.
    • OVERVIEW

      • An important element of the culture of science is the use of “tentative” propositions, often quantified. These can be as confident as 99.99999%—you would bet your life on it—but it would still be understood to be held as a proposition which could be wrong. This makes it psychologically easier for a scientist to be open to being wrong—and to look actively for ways they might have gotten it wrong. This cultural understanding of the importance of recognizing and reporting one's credence level leads to insistence on including error bars on graphs: a data point is completely meaningless without an error bar.  
    • CLASS ELEMENTS

      • Class Exercises
        • Many examples given from recent science presentations.  
        • A arbitrary topic is chosen for small group discussion (e.g. “Does testing in the schools help or hurt education?”), but during the discussion the students have to state their credence level (by saying a number between 0 and 100%) after every statement that they make which could have a credence level associated with it.
    • OVERVIEW

      • Most people’s estimates of their confidence is wrong in characteristic ways: high confidence tends to be over-estimated and low confidence tends to be under-estimated. This can be trained to be closer to an accurate calibration, most effectively with repeated, unambiguous, and immediate feedback. One problem that arises from poor calibration is that juries often use witness’ confidence to gauge the likelihood that they are correct, but this often yields poor results due to the poor calibration of the witnesses. 
    • CLASS ELEMENTS

      • Class Exercises
        • Credence-calibration questionnaires to show students’ calibration, and training exercises.   

    • OVERVIEW

      • A useful “jargon” of the scientists is to speak of orders of understanding or orders of explanation. “A zeroth order explanation/cause” or “a first order explanation/cause,” is a major cause/factor, as opposed to “a second order” or “third order explanation/cause," which would be real causes with smaller effect sizes. This is useful because explanations for how things/actors in the world behave can often be parsed into a primary explanation, a secondary less important cause, a third order, still less significant cause, etc.
    • CLASS ELEMENTS

      • Clicker Questions
        • Students practice (individually and in small groups) identifying first-, second-, and third-order causes/explanations in a variety of scenarios.
        • The final such question for this class concerns parsing some of the biggest drivers of government spending, specifically what are the relative “orders” of the contributions to total government spending from the costs of education, incarceration, and social security.
      • Discussion Questions
        • In small groups, students practice identifying first-, second-, and third-order causes/explanations in a variety of scenarios.
    • OVERVIEW

      • Physicists train their students in doing “Fermi problems,” back-of-the-envelope estimates of quantities that arise in physics problems and in life. This is useful as an approach to performing “sanity checks” of claims in the world and of your own ideas, beliefs, and inventions. Checking numbers with quick Fermi estimates may be even more important in a world in which it is difficult to evaluate the credibility of numbers available by Googling. 
    • CLASS ELEMENTS

      • Class Exercises
        • Together with the whole class, the professor shows how to develop Fermi-problem estimates of a given quantity, e.g. the amount per year that Americans spend on gas for personal transportation.  
        • In small groups, students work on several Fermi problems to develop facility with the approach.
        • In small groups, students use Fermi estimates to re-think the first-order, second-order, etc. parsing of US government spending on education, incarceration, and social security (following up the final activity from Topic XIIV, Orders of Understanding). The students frequently reach completely different orderings than they did in the previous class—and come within ~20% of the actual amounts spent.   
    • OVERVIEW

      • Here we will explore the most widespread heuristics and biases which psychologists of judgment and decision-making have discovered in everyday reasoning: the availability heuristic, representativeness heuristic, and anchoring heuristic, and biases like optimism bias, hindsight bias, and status quo bias. Many examples are drawn from Daniel Kahneman's book, Thinking Fast and Slow.
    • CLASS ELEMENTS

      • Class Exercises
        • Small group exercises and clicker questions to demonstrate these effects with the students. 
    • OVERVIEW

      • Science has a particularly bad track record when it comes to studies of human sub-populations for the purpose of setting policy—particularly when groups in power study groups out of power. We should be aware of this, and wary of misusing science in such a way as to perpetuate injustice.
    • OVERVIEW

      • Distinguishing pathological science, pseudo-science, fraudulent science, poorly-done science, good science that happens to get the wrong answer (which should happen statistically for 1 in 20 papers that give a 0.05 confidence level result). What do practicing scientists do when they try to judge a paper in a field or sub-field outside their immediate area of expertise? 
    • CLASS ELEMENTS

      • Class Exercises
        • Summaries of three relatively recent surprising science results (e.g. super-luminal neutrinos, bacteria with arsenic in their DNA, water with memory, cold fusion) and their follow-up in the scientific community are distributed among the groups. Each group explains the summary they read to other groups, so all have thought about each example. The groups discuss and vote with clickers on whether each article falls into the category of pathological science, poorly-done science, etc.   
    • OVERVIEW

      • Our tendency to preserve our existing or preferred beliefs, even against the evidence.
      • This class explores confirmation bias in the search for and assessment of evidence. In particular, we consider the ways that people tend to seek out and think about evidence in such a way as to reinforce their existing opinions, rather than testing them against new information or alternative views.
    • OVERVIEW

      • Science is not a single “scientific method” (as often taught in school), but better characterized as an ever-evolving collection of tricks and techniques to compensate for our mental (and, occasionally, physical) failings and build on our strengths—and, in particular, to help us avoid fooling ourselves. These techniques must constantly be re-invented, as we develop new ways to study and explain the world. In the last few decades we have entered a period in which most scientific analyses are complicated enough to require significant debugging before a result is clear. This has exposed another way we sometimes fool ourselves: the tendency to look for bugs and problems with a measurement only when the result surprises us. Where previously we recognized the need for “double blind” experimentation for medical studies, now some fields of science have started introducing blind analysis, where the results are not seen during the development and debugging of the analysis—and there is a commitment to publish the results, however they turn out, when the analysis is “un-blinded” and the results interpreted. 
    • CLASS ELEMENTS

      • Clicker Questions
        • Students are presented historical graphs of improved published measurements of a physical parameter over the decades and must identify the ones that retrospectively show evidence of biases that could have been avoided by blind analysis.   
      • Class Exercises
        • Students make a measurement which is somewhat tricky to perform with two-digit precision, and experimental conditions are set up to show that the part of the class that was “blinded” gets a more accurate result.  
    • OVERVIEW

      • Explore ways that groups fall short of their optimal reasoning ability. There are better and worse ways to aggregate a group’s knowledge. 
      • Sometimes groups of people reach better conclusions than people working independently, and sometimes they reach worse conclusions. There are features of group reasoning that can help, and features of group reasoning that can hurt. Here we explore how to avoid the pitfalls of group reasoning and to maximize the benefits.
    • CLASS ELEMENTS

      • Class Exercises
        • Students answer typical wisdom-of-crowd estimate questions using their clickers—but they can update their estimates as they see the histogram with the other students’ guesses. Afterwards, it is shown that the accuracy of the class’ mean estimate actually got worse as they continued to update their estimate—showing (if it works) that wisdom of crowd works best if the inputs are independent.    
    • OVERVIEW

      • How should (and shouldn't) values/emotions/goals/desires and conflicts of interest properly be woven together with science's hyper-rational elements in decision making processes? 
    • CLASS ELEMENTS

      • Students work out a problem involving values and factual/scientific issues, using method used in Denver bullet study. 
    • OVERVIEW

      • In the two classes of this week, we try out an approach to random-sample-representative decision-making, using a panel of experts to answer questions generated by small deliberative groups, each with a moderator. We also have a presentation from the professor who developed this technique, describing its use around the world. One example topic we have used is Fracking for Natural Gas.  
    • CLASS ELEMENTS

      • Class Exercises
        • In the two classes of this week, we try out an approach to random-sample-representative decision-making, using a panel of experts to answer questions generated by small deliberative groups, each with a moderator.   
    • OVERVIEW
      • Adversarial vs. Inquisitorial Modes of Truth-Seeking