Aired April 22, 2020
MALE VOICE: Can you tell us, how do you do it?
MALE VOICE 2: To be quite honest, we cheat.
GORDON KATIC: I’m Gordon Katic. This is Cited.
SAM FENN: And I’m Sam Fenn. And we’re here with our producer Alex.
ALEXANDER KIM: Hey, I’m Alexander Kim.
GORDON: A lot of what we’re doing this season on Cited is looking at the question of trust and whether or not we should trust scientists, and trust experts of all sorts. Our first episode - if you haven’t heard it - that was a kind of crisis in trust. A crisis sparked by post-modern scholars questioning the authority of science. And this episode is similar, except the skepticism isn’t really coming from postmodern scholars. It’s coming from science itself. It’s coming from defects in the very methodologies that scientists use.
SAM: So on the subject of trust and experts and trust in science... I know a guy who is fairly distrustful of scientists. And that’s my friend Alexander Kim here.
ALEX AND GORDON: (laugh)
SAM: But I also know another guy who was very trustful and sort of pollyannaish about scientists.
ALEX: Oh, who was that?
SAM: That was my friend Alexander Kim when I first met him!
ALEX AND GORDON: (laugh)
ALEX: Yeah. Well it’s true. Um, when I was a younger man [laughs], I was an avowed empiricist, an avowed quantitative guy. I believed in data. I was studying science during my undergrad. I thought I was going to be a scientist. And like, I believed in the goodness, the inherent goodness of science. That there was this thing that humans had figured out to come together and dispassionately ask questions about the world, and the only thing that matters is truth and the whole enterprise is just about producing truth. Figuring out what is real, what is true, and what we can learn about it. So I wanted to be part of that essentially.
GORDON: So Sam and I, we can bloviate all we want about the nature of truth or what scientists do... but in terms of actual scientific methodologies, we don’t really know that much. But you’re like, you’re the real deal. Give us your scientific credentials.
ALEX: Uh... B.Sc. (laughs) Yeah, I’m not really a scientist, but I do have some research experience. In my undergrad I studied neuroscience and I spent most of my summers in labs, learning different techniques. And I did like this thesis project in my last year. I’ve done some science, you could say.
GORDON: So you came to us, you wanted to be, at that time, a science journalist, right?
ALEX: Yeah I figured... I found out that science wasn’t exactly for me. Maybe I didn’t have the right personality type for it. And I started to ask myself, what the hell am I going to do? Maybe I’ll tell stories about science.
SAM: And so on today’s show we’re going to hear the story of your fall from grace [Alex laughs] and losing your religion, right Alex? But you want to start us in sort of a weird place to do that?
[MUSIC: Garth Mullins - Drifting Out of Consciousness]
ALEX: Yeah [sighs] this story is a bit about me... But it’s also about science, the methods of science and whether we can trust them. It’s a story about the friction between idealistic empiricism versus pragmatic economics.
And it’s a story about the long-running feud between magicians and psychics. And that story begins with Dr. Daryl Bem.
Alex: So first thing: can I get you to introduce yourself? So your name and what you do.
Daryl Bem: Okay. My name is Daryl Bem. B-E-M. And I’m retired from Cornell University where I was professor of psychology since 1978. Um. But now I’m retired.
ALEX: Professor Bem lectured at Carnegie-Mellon University, Stanford, Harvard, and Cornell. As a researcher, he studied public opinion and personality.
Bem’s work was influential, making him a prominent and respected authority in his field... not associated with anything, you know, weird.
One day in 1985, Bem receives an invitation to a convention with a strange request. It’s from a group called the Parapsychological Association.
They are - I’m quoting from their website now - “the international professional organization of scientists and scholars engaged in the study of ‘psi’ or ‘psychic’ experiences.”
Alex: What is psi?
Bem : OK. Psi is the ability to acquire information in what we call non-local ways, that is, to be able to pick up information that’s just not available to our senses in any other way.
Bem: And there are sort of three major phenomena that fall under that. The first is telepathy. The second one is clairvoyance. And then the third one is precognition, which is the ability to anticipate the future in ways that could not be done by inference or other means of knowing what’s going to happen in the future.
[MUSIC: Sam Fenn - Lydian Clouds on the Horizon]
ALEX: Bem’s work doesn’t have anything to do with this stuff. But he does have a special skill that fascinated the parapsychologists: Magic!
Bem: When I was 10, 11, 12, and 13, I would give magic shows at birthday parties where I would earn $5 if I helped clean up. And so, (laughs) by the time I was in high school I got interested in an area of magic known as mentalism, which is essentially fake psi. Fake ESP. It's magic tricks designed to look like you can read minds, for example. And so I actually developed a routine and I was a stage magician doing that kind of thing from high school on, through college and beyond. And even did my performances at the end of a semester as a treat to my students in the psychology courses I was teaching.
ALEX: The parapsychologists ask Bem, Would you come to our convention and give us a presentation? Not about your influential self-perception theory, or your research on template matching. We want you to put on a magic show.
They tell him that they want to protect themselves from fakers showing up at their labs, claiming psychic abilities. In fact, psi researchers had already been fooled, in a very embarrassing, very public ordeal just a couple years earlier.
TV PRESENTER: James Randi, conjurer, is the scourge of the psychics! He travels the world debunking claims of PK.
ALEX: James Randi - known on stage as “The Amazing Randi” - orchestrated a years-long hoax on a parapsychology research group in St. Louis, Missouri. Two young magicians working with Randi approached the lab and claimed that they could do all sorts of paranormal feats. Randi called the operation “Project Alpha”.
[TV NEWS PROGRAM]
Mark Shafer: So we have been working first of all, to establish the range of abilities Mike and Steve have. Because these have apparently included being able to move small solid objects across a table top, influencing a variety of metal objects such as keys and silverware, and metal bars and metal rods. I don’t believe they’re tricking us.
TV Presenter: But in 1983, after Edwards had amazed the audience at a New York press conference, Randi dropped his Project Alpha bombshell.
James Randi: I’m going to ask these two gentlemen a very simple direct question. Can you tell us, how do you do it?
Mike Edwards: Well to uh, be quite honest... we cheat.
[TV program fades down]
Bem: So they wanted to know from me, how I did some of the things, or at least to show them what to watch out for.
Alex: So wait. I just want to pause the story and ask you, at this time do you believe in ESP as well?
Bem: No. I held pretty much the view of most psychologists. And of all academics, the psychologists are the most skeptical!
ALEX: So, Bem travels to Medford, Massachusetts, a little city three miles north of Boston, home to Tufts University. That’s where the parapsychologists gathered.
Alex: Are you looking around this room of people and thinking, “what a bunch of fools!”? What are you thinking about these people?
Bem: Oh, absolutely not! Absolutely not! They too, have usually come to their views by looking at rigorous laboratory experiments that are well controlled. So no, I don't believe that at all.
[MUSIC: Channing Pollock performance 1956]
Alex: So Bem takes the stage and introduces his act.
And then from the audience, a man brings forth a locked box.
Bem: Someone would come prepared with a set of eight items in a box that was sealed up. And they swore that they hadn't told anyone what was in the box and that they had filled it at their home. And that even their wife or husband did not know what was in the box. And I would then ask them to concentrate on the items one at a time.
Bem: And I would describe what those items were. That was one of my favourites.
Alex: So if you don’t mind, um, revealing the nature of the trick, how are you able to do it?
Bem: I could tell you but then I'd have to kill you.
Alex: (forced laughter)
Bem: No, we do not... As a magician we do not reveal the secrets and in fact it annoys some mental-magic magicians just to tell you that it's faked.
ALEX: After the show, a man approaches Bem and introduces himself: Charles Honorton, director of a parapsychological research group based in New Jersey.
Bem: And because I had also had experience setting up and performing social psychology experiments, he invited me to his laboratory to look it over and see if it was possible to cheat in his experiment. And so I went there, and in preparation for that I started reading the literature quite, quite a bit. And that persuaded me that gee it really looks like something is here. So I went to his laboratory, looked it over, and decided it looked airtight to me. And so I said to him, “You know the other talent I have is getting published in mainstream psychological journals.” And I said if you get positive results with these experiments, I'd be willing to try to get us published.
Alex: Why did you make that offer?
Bem: Because I was very impressed by the experimental methodology and thought if he got positive results, then that was enough to persuade me!
Alex: So as a scientist, as a researcher… Do you believe in psi?
[MUSIC: 44 New Cops]
Bem: And I believe it because of the laboratory evidence for it. Psychologists tend to be very skeptical of anything other than laboratory evidence.
ALEX: And so Bem starts believing in the absurd. But ironically, this belief comes from a commitment to empiricism, not a rejection of it. Bem starts to think psi is real because he believes in the standards, procedures, and values of science.
Recruiting Bem was a major coup for the parapsychologists. They hoped that Bem would bring with him an air of legitimacy and the respect of mainstream science.
Soon, Bem begins to design his own psi experiments. And the more he does them, the more he becomes convinced in the evidence for psi. Bem wants other scientists to see it too. And so he makes these experiments watertight, ironclad, undeniable. His findings would not be easily dismissed by skeptics.
Nonetheless, these experiments are really weird... In one experiment, the participant sits down at a computer. On the screen there are two curtains - left and right. This is meant to be a test of precognition, predicting the future.
Bem: The participant is told that behind one of the curtains is a picture and behind the other one is just a blank wall. And that some of the pictures are erotic. And the challenge to the participant is to select the curtain that has the picture behind it.
[MUSIC ENDS SUDDENLY]
SAM: Let me see if I understand this, Alex -- so they’re supposed to predict where the picture, this erotic picture, is going to be-- is that right?
ALEX: Yeah. Yeah. There are two sides, they have to pick: left or right.
SAM: And you said erotic picture?
ALEX: That’s right. [laughs]
SAM: Okay, like what?
ALEX: Yeah well there’s different photosets, depending on your orientation. Um..
SAM: So you tell them when you go in, like, I’m a straight guy...
ALEX: You select your preference, whatever would be arousing to you I suppose. And then some of these images, the images that you’re supposed to be guessing are basically really old looking weird porn...
ALEX: You wanna see it? [laughs]
ALEX: I’ve got some of these on my work computer.
SAM: Okay. Oh, so Bem sent you some of the nudes?
ALEX: Yeah I’ve got his program...
SAM: Oh um. Hmm.
ALEX: That’s kind of just like a guy’s ass in bed.
SAM: Yeah you sort of see a man’s naked butt. Um. Yeah, alright. what’s that?
ALEX: That’s pretty...
SAM: That one was just porn.
SAM: So... why was he doing this? Why did he get all these nude photographs?
ALEX: Um, basically the theory is evolution.
Bem: Well you have to ask yourself, why, if psi exists, do we have that ability? And so you look to evolution. Why would evolution have favoured such an ability?
ALEX: For two reasons, Bem says. The first is avoiding danger. Call it your “spidey sense”. The second is reproductive urge. Your... “sexy sense” I suppose.
Bem: Because you have to mate! And so, that means that sexual scenes, sexual opportunities, should be something that it would be valuable to be able to anticipate.
ALEX: Okay, if you were going to do this experiment, what would your hypothesis be? What would you expect to happen?
SAM: Okay, Just to clear up: I can’t see behind the curtain at all?
ALEX: No. Not until you make your choice, left or right.
SAM: Right, Okay… you know... I don’t think that I have any kind of sense, ability to intuit where the nude pictures are. So, it would be a coin flip, right?
ALEX: Yeah. Totally random.
SAM: And if you did the experiment enough times, you’d get to 50 per cent.
ALEX: And, you’d expect there would be no difference between when there was porn hiding behind the curtain, and some other image, like a tree or whatever.
SAM: Absolutely not. It would make no difference whatsoever.
ALEX: But what if you had... psychic, time-traveling, mind-reading powers that could sense sexual, uh, materials for evolutionary reasons?
SAM: Okay, I am just going to try to just even understand the mechanics of it, okay? Is it that they travel forward in time... and read their own minds? That’s what he’s saying?
ALEX: Yes. And if that were to be possible, then you’d see that the success rate for those sexual photos would be higher than 50%.
ALEX: And it was.
Bem: In our experiment we saw 53 percent.
ALEX: In this experiment, Bem found that his participants were able to correctly predict where these sexy photos were going to be in the future 53 percent of the time. Which...
SAM: Sounds like half to me.
ALEX: Doesn’t sound like it’s different than 50/50, right?
ALEX: But according to Bem, it actually is. It’s a big deal. This means, this three percent means, that somehow… information is traveling backwards in time.
[MUSIC: Garth Mullins - Backwards in Space and Time]
ALEX: Bem writes up the findings from nine of his precognition experiments and writes it up in a manuscript. And then he submits to the Journal of Personality and Social Psychology. This is considered a top-tier journal in the field.
Bem’s paper finishes peer review in 2011 and gets published in the journal’s 100th volume.
The title of the paper is, “Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect.”
[MUSIC: Channing Pollock performance 1956]
By all standards of modern psychological science, Daryl Bem had “proven”... that precognition... is real.
[MUSIC CRESCENDO AND APPLAUSE]
You know, like 53 percent of the time…?
Alex: OK... And that's like a statistically significant difference that beats the null hypothesis there?
Bem: Yes. We can't claim it until we've actually done that. There are two numbers that are required in reporting an experiment. You mentioned one: statistically significant, and we probably should tell the listeners what that is…
ALEX: So when Bem says “statistically significant”, he’s talking about a number, and that number is usually called p, or the p-value.
ALEX: And the p-value is a number that comes out of these statistical tests that scientists will use to analyze their data. And these tests tell them whether two things are statistically the same or statistically different.
SAM: Okay. Hmm... Having made Cited, obviously I hear the term p-value a lot. At a very basic level I don’t even know if I totally understand it.
ALEX: Basically this is a probability calculation. And that’s what p stands for. P stands for probability. And if you have a p-value that is smaller than a certain amount, it represents a very small probability that this result you’re looking at is just due to random chance. It’s a false positive. So once you get a p-value that crosses that threshold... And in science, more often than not, that threshold is zero-point-zero-five.
SAM: And so basically if you run an experiment enough times and you get a certain kind of result enough times then you can start to say, “This doesn’t seem to be random, this seems to be telling us something. There is something here.”
ALEX: Yeah. Bem has mathematically shown that his participants are able to guess where those sexy pictures end up better than chance.
[MUSIC: Knocked The Door Right Off]
SAM: Let’s just...Let’s just put our cards on the table. This is outrageous, right? The experiment he’s doing, it just doesn’t pass a bullshit test, I think, for most people.
SAM: But scientists have this kind of commitment to objectivity, to having this sort of quantitative view of the world
SAM: And so I would imagine at this point there would be this real pressure from two different sides, say, if you were a publisher, to respect Bem’s findings because they satisfy your requirements, your sort of scientific requirements. But also there would be this cultural pressure to reject the findings because they seem... crazy.
Alex: And did you have trouble getting this paper published?
Bem: No. It was accepted. The editors did feel it necessary to publish a note along with it, justifying their accepting of the paper.
Alex: What did that say?
Bem: It said that these are intriguing results but of course we hope that other people will try to replicate them. And that's the absolute standard in science. You have to be able to replicate them by other people. The more extraordinary the result, the more controversial the result, the more important it is to have other people try it out.
GORDON: I’m Gordon Katic, and you’re listening to Cited. We’ll be back after this break.
This is the part of the program where I sell you meal delivery kits. But like I said last week, we’re going to spare you. I just didn’t want to subject you to ads. Instead, I asked you to share our episode with a friend. And then send me a photo of your text or email to that friend.
This was the deal: The first three people who did that would get swag. Well, you held up your end of the bargain. You shared it.
Anja is a computer science major, and she sent it to a few of her friends. Anja told them-- Cited made her think a lot about the role of science.
Patricia is in a philosophy reading group. They’re reading Deleuze-- and she shared the episode with the group. She told them, “My brain is exploding.”
Leora shared the episode with her colleagues at McGill University’s ‘expanding economics’ group. By the way, they also have a podcast.
As promised, Anja, Patricia, and Leora will all be getting swag. I’m making some mugs. As soon as I get the designs right, I’ll send them your way.
This week - no swag this time, sorry, can’t afford mugs every week. But stay tuned, there’ll be more stuff. I’m still going to ask you a favour this week. I just want to -- you know me -- I just want you to email me. I just want to know more about who you are, what do you do, and why you listen to Cited? Maybe tell me a little bit about what you think of this episode. Don’t have to give me an essay, just a short note. I really just want to know who you are. You can send that to firstname.lastname@example.org. Again, that’s email@example.com. OK - back to our regular scheduled programming.
GORDON: Welcome back to Cited. I’m Gordon Katic. Before the break we met the infamous Professor Daryl Bem. This magician-psychologist was just about to throw the entire field of psychology into chaos. Producer Alex Kim will take it from here.
Alex: So, I want you to tell me a little bit about Daryl Bem. Can you give me a sense of how respected he was, how important he was to the field, how prominent?
Jeff Galak: This person is influential. I mean he set the tone for how psychology was practiced.
Alex: What do you mean, he set the tone for how psychology was practiced?
Galak: He was one of the folks who people looked to to decide what the right research methodology is.
ALEX: This is Jeff Galak, professor of marketing at Carnegie Mellon University. He studies consumer psychology and behaviour.
Galak: Not him alone. I mean he and the cohort of other successful psychologists and researchers. But he was there doing it and everybody agreed that the methodology that he and everybody else — myself included — follows is what's appropriate. And I think his paper on psi absolutely mirrored the norms that were around and they still are around, to a large part, in research methodology and psychology.
Galak runs what’s called a “journal club” at Carnegie Mellon. Each week, researchers pick a peer-reviewed journal article and meet up to discuss. They talk about the paper’s strengths, weaknesses, and implications.
After Daryl Bem’s Feeling the Future came out, everyone wanted to talk about it.
Galak: So this was on our list really quickly to read and we had a discussion. And the discussion was mostly... positive! I mean we mostly came out of this saying, you know we're skeptical, we can't believe that this is true... But look! Here's the evidence. It's in a good journal, it's been refereed, uh, and... we should believe it.
Alex: So there was like 20 people in a room discussing this paper.
Galak: It's hard to remember. Maybe around 20.
Alex: And you know, I've sat in a journal club before. You kind of hurl everything you have at the paper and find any crack that is there, and you... and the paper mostly came away unscathed?
Galak: The paper came away with, “we still don't believe it” because it was just so incredible and yet we couldn't find any uh... critical flaw at the time. You know?
Galak: Which is unusual for our journal clubs. Usually our journal clubs, as you point out, end with us thinking that papers are terrible.
ALEX: Bem’s experiments were standing up to very close scrutiny. The paper had all the appearance of good science. It was so convincing that Galak felt that, as a scientist, he had to take this seriously.
Galak: My understanding of all the work that had been done up to this point was that it was sloppy, it was not done with any kind of scientific rigor. And here comes one of the most prominent psychologists in our field with what looks to be a rigorous, tightly controlled set of experiments that demonstrate the presence of some version of psi. That's... that's impressive.
ALEX: Galak thinks, “Could this be real? I think this might be real.” And he calls up a fellow researcher at UC Berkeley, Leif Nelson. Nelson has his own journal club at his university. Nelson’s group talked about Feeling the Future too, but Nelson came out of the discussion on the exact opposite side: There’s no way that this finding is reliable.
Galak: So he and I were chatting, and he and I bet this 20 dollars that I said it would replicate and he said there's no friggin’ way that that's going to happen.
[MUSIC: Sam Fenn - Another Clue]
ALEX: The two scientists decide to repeat Bem’s experiments themselves and see what happens. If we can replicate Bem’s findings, then psi exists and this changes everything, right? But if we can’t replicate, then maybe there’s something ever weirder going on here. Something we don’t understand yet...
Galak: So we had this huge sample of people across a variety of experiments, experimental conditions, so laboratory, different universities, online and so on. And across all of those there was absolutely no evidence for psi.
I really wanted this paper to be true. I thought the world would be just such a more interesting place if there was such a thing as psi or ESP. And I was you know... I bought it. I really did. Made a twenty dollar bet and then I lost it.
ALEX: So Galak and his collaborator, they did everything as closely as they could to match Bem’s study.
SAM: Do the exact same thing that he did right?
ALEX: Replicate. That’s the idea. But in the end, doing everything the same, these two groups come to completely different results.
SAM: Right, okay, so if Bem’s experiment actually didn’t produce real results, how do you explain that? What was actually happening with Bem’s study?
ALEX: Well, I think the first think I can say is that we can probably eliminate the possibility that time traveling mind reading is real.
SAM: Okay. So here’s my question. The p-value thing we were talking about earlier. Is the theory that it’s 100% accurate? That there will never be an experiment that reaches that p-value of point zero five and that isn’t just sort of like random noise?
ALEX: Basically it’s a marker of “good-enough-to-certain”.
SAM: Okay. But there’s an understanding that even past that threshold there will still be some stuff that is just random or whatever.
ALEX: Yeah. You can’t control for all randomness or whatever.
SAM: Okay, so maybe the Bem study is one of the ones that.. Just one of the outliers.
ALEX: Yeah and you do expect a certain rate of false positives just to happen, right?
ALEX: If everyone is using point-zero-five... What that number represents is that there is a less than five percent chance, according to the statistical model, that this result is a false positive. That it’s random, right?
SAM: Oh okay!
ALEX: We’re 95 per cent certain.
SAM: Okay so there you go, right? Case closed! End the episode. The Bem study just happened to be one of the 5 per cent that kinda sneaks through.
ALEX: Yeah. And I think a lot of people may have been thinking that. But others were looking for a sort of deeper explanation.
ALEX: They started looking under the hood of the scientific practices themselves and how they work across psychology.
SAM: Like how they collect their data.
SAM: How we determine what gets published... right.
ALEX: Exactly. And a huge impetus for this was this paper that was published in the same year - in 2011 - called False Positive Psychology. And basically, what this paper showed was that it’s possible for a psychologist, using totally normal research practices, to prove pretty much anything.
Galak: Researchers, and this includes me and everybody else, have the ability to decide on a number of factors that seemingly are inconsequential for finding things like statistical significance. And yet when you look at them in aggregate they have tremendous impact for false positive results. In other words, I as a researcher have the luxury to say, choose between multiple dependent variables, and if I fail to report the fact that I'm doing so, I'm inflating the likelihood that what I'm going to report is a false positive.
ALEX: Throughout the scientific process, scientists make choices. What to manipulate in an experiment. What to measure. How to analyze the data. And crucially, how much of this process to report when it’s time to write the paper.
All of these choices affect p-values. And if your p-value isn’t smaller than point-zero-five your results are not going to be publishable.
As a researcher, you have a couple options. You can simply abandon the study here, or... you can keep working the data.
This is called “p-hacking”.
Chris Chambers: Yeah, right. So p-hacking is basically a form of cheating.
ALEX: Chris Chambers is a professor of cognitive neuroscience at Cardiff University in Wales. He’s the author of the book The Seven Deadly Sins of Psychology.
Chambers: P-hacking is where you selectively report an analysis out of many analyses that you ran. And when it comes to writing up the paper you selectively report the analysis that quote-unquote “worked”.
ALEX: So okay. Hypothetical experiment: Let’s imagine you want to find out how much a discount will affect people’s behaviour at an all-you-can-eat buffet.
SAM: Okay. Five bucks off today?
ALEX: Yeah. Either you pay ten bucks or you can pay five bucks or whatever. So in an experiment like this you can measure whatever you want, you can measure all kinds of things. How much people eat, obviously, right? But like, what they eat, how many trips to the buffet table, do they drink alcohol, do they drink water, how much do they drink?
SAM: Do they tip better?
ALEX: All kinds of things you can look at. And then what you can do is just look for differences among all of it, right?
SAM: Okay, so you actually ask all those questions. You don’t just ask one or two of those questions. You ask all of them...
ALEX: That’s right.
SAM: You record all the answers.
SAM: You’re just gonna have this spreadsheet or something.
ALEX: You know, Test this and this... this and this... this and this...
SAM: Then we find something. Finally a hit.
ALEX: Oh that’s interesting. What does that mean?
SAM: And then we didn’t have a hypothesis about that, but maybe we cook one up now.
ALEX: And then that’s a paper.
SAM: Okay. So some of them work and some of them don’t work.
SAM: Okay… So what’s wrong with this? I mean...
ALEX: Well, basically, the only thing you’re looking for is significance. And if that’s all you're looking for, you will find it eventually. You can just keep testing, testing, testing, testing and you can find something. Now, it could be true, it could not be true. But the chances that that’s a false positive are not five percent. They are much higher.
SAM: And that’s because we just kind of observe this out of a sea of random numbers. We weren’t drawn to this result because we had a theory about how the world worked, and we tested our theory. We were drawn to it just because we looked at it and said, “Ah! That’s publishable right there!”
Chambers: If you selectively report, you cherry-pick only the significant result, then it artificially inflates the certainty in your finding and it makes it seem a lot more convincing than it really is and this helps psychologists publish in journals, because journals want convincing results. And so the whole crappy cycle just kind of goes round and round and round.
ALEX: That all-you-can-eat buffet experiment is a real one. It came out of the Food and Brand Lab at Cornell University, run by psychologist Brian Wansink. And it produced four papers, published in peer reviewed journals. Those journals later issued corrections and one paper was fully retracted.
It’s not easy to know the prevalence of this, but evidence suggests it is common. In 2012, a survey of more than 2,000 American psychologists found that every single one of them had selectively excluded data at least once.
And so, psychologists started to think a little differently about Daryl Bem’s Feeling the Future.
Chambers: And this is really interesting because if a paper like this that's doing everything normally and properly can end up producing a ridiculous conclusion then how many other papers that use those exact same methods that didn't reach ridiculous conclusions are similarly flawed?
ALEX: This was the beginning of what became known as... the replication crisis.
[MUSIC: Sam Fenn - A Darker Mystery]
NEWSY: Replication: it’s the cornerstone of science...
BRIAN NOSEK: Is there actually a reproducibility crisis...?
NEWSY: Experiments are supposed to show the same results every time...
SAM: We’re now in a very fascinating place where Bem seemed to have proved something that was unbelievable.
ALEX: Yeah, impossible. Raising the very good question, how much of what we thought was solid, was true, is in fact just wisps of dust?
NATURE: Does this mean that most published research is wrong?
AMY CUDDY: Free, no-tech life hack. All it requires of you is this: that you change your posture for two minutes.
BRIAN WANSINK: Two big eating myths...
CUDDY: Two minutes, two minutes, two minutes...
WANSINK: Some really cool research we’ve just been doing that you can take home with you tonight...
PAUL ZAK: After 10 years of experiments, I finally found it... the moral molecule.
ALEX: And so to address this question, a bunch of scientists come together from all over the world to form a group called the Open Science Collaboration, and they do a massive replication study. They pick 100 different studies - all published in the year 2008 - and they say okay, let’s organize all these labs across the world in 17 different countries, let’s repeat these 100 studies as closely as we can.
SAM: What did they find?
ALEX: Well, it wasn’t good. In 2015, the results of this replication experiment came out, and only 36 per cent of replications were successful. That’s a little bit better than one in three.
SAM: Wow! So 50 per cent would have been really bad... This wasn’t just one little outlier, perhaps, with Bem, but maybe this was the germ of something like a kind of rot right at the heart of psychology.
ALEX: Yeah. And this is not just Daryl Bem. Tons of psychologists had their whole worlds turned upside down. Let’s take just one example, the people who worked on this big idea called “ego depletion.”
Michael Inzlicht: And the idea behind it is that self-control is thought to rely on a limited resource that runs out after use.
ALEX: Michael Inzlicht is a professor of psychology at the University of Toronto.
Inzlicht: And when you use that resource, the self-control resource, to control your impulses or thoughts or behaviors at one time of the day you will have less of that resource to control yourself later on.
ALEX: Maybe you diligently eat healthy foods all day, but at night you just can’t help yourself from binging on chips. Resisting temptations or making tough decisions uses up brain power.
The idea became pretty popular. Barack Obama once said that he always wore the same suits to avoid wasting his decision-making power on trivial stuff.
Inzlicht built his career on ego depletion. He worked on this theory doing his own experiments for almost a decade. But now, doubt and uncertainty were growing in the research community.
A big meta-analysis, which is a study of many studies, came out. It said there was no statistical evidence for ego depletion.
At first, Inzlicht wasn’t convinced. To him, it seemed like funny math. Ego depletion had been empirically observed hundreds of times. You can’t just make that all disappear.
Inzlicht: But what will really solve this issue is if we have a bunch of people around the world try to replicate a paradigm - an ego depletion paradigm - that the proponents of the theory agree about, and then you know a bunch will go off and test it with thousands of people and then we can ascertain to what extent this is a valid idea or not. And that's in fact what happened in 2016. And the results were shocking. The overall meta-analytic effect from these 23 labs was zero. In other words, a finding that had been replicated - now people say about 600 times - when there was a rigorous pre registered principled attempt to replicate it, we could not. And that was... I mean disquieting...
ALEX: It wasn’t only ego depletion. All kinds of effects across psychology would not reliably replicate. Unconscious priming, stereotype threat, the facial-feedback hypothesis, “power posing” (if you remember that?)...
Inzlicht: I had trouble differentiating like, what was real, what was not real. I had grown up a scientist you know believing in the scientific method and the tools that we used, and all of a sudden, this one replication made me just question... everything. What was real? What could I trust? The things I was studying... Were they real?
ALEX: It was as if the ground had shifted. Maybe it was never there in the first place. Watching this happen changed Inzlicht. He would never be the same again. But to his shock, others in his field didn’t seem nearly as troubled.
Inzlicht: The senior people of our field, the gatekeepers, the caretakers, the people who are in charge... They were acting as if nothing was amiss. They're acting as if this is business as usual. “Nothing wrong here folks! Keep on moving! Keep on, working!” and I'm like “What??” Up until that point, you know being a scientist, working at the University of Toronto was a joy. It was a pleasure. I derived so much meaning from it. I was one of those weird guys where I told myself yeah when I'm 65-70 I'm not going to retire. Why would I want to retire? This is… This is fun! After this... Maybe walking away from this isn’t such a bad idea.
[MUSIC: Sam Fenn - Fucking Heroes]
Inzlicht: You know I'm not just doing this for myself. I'm not just doing this because I think it's fun. I'm doing this because I think it's important. I'm doing this because I think science you know advances knowledge and knowledge is important. So the fact that you know, it's been revealed that all this could have been for naught is distressing! If you're not bothered by it then... I'm not sure what's wrong with you.
ALEX: The American astronomer and author Carl Sagan wrote that science is like “a candle in the dark” in “a demon-haunted world.”
Science is supposed to be the light that leads humanity out of the darkness of ignorance, an engine of ceaseless progress towards enlightenment.
But is that really how science works? Or have we all just been fooling ourselves this whole time?
GORDON: I’m Gordon Katic, and you’re listening to Cited. We’ll be right back.
Here’s another ad break. This time, no favour; just some housekeeping. I wanted to announce a new thing we’re doing: it’s called Secondary Symptoms.
Secondary Symptoms is our new COVID-19 miniseries. We decided we needed to give you some Covid content-- so we started a quick and dirty interview program. We’re calling it Secondary Symptoms, because it’s not really about the disease itself. It’s not about what it does to your respiratory tract. No, it’s about what Covid-19 is doing to all of us: What it’s doing to our politics, our science, and our social fabric.
If you’re subscribed to our podcast, you’ll already see it in your feed. You don’t need to do anything. I’m going to post Secondary Symptoms once a week, usually Monday or Friday. And you’ll get Cited documentaries on most Wednesdays. So you’ll hear from me twice a week.
If you’re listening to Cited at citedpodcast.com, well--you’ll have to subscribe to our podcast. You can find it on iTunes, Spotify, or wherever people find podcasts.
Secondary Symptoms is an experiment. So as always, tell me what you think. You can tweet me @gordonkatic, that’s G-O-R-D-O-N K-A-T-I-C, or @citedpodcast. OK, back to our regularly scheduled programming. Sam and Alex will take it from here.
[Cited theme ends]
SAM: Welcome back to Cited. I’m Sam Fenn. I’m here with Alexander Kim.
SAM: And in this story so far, we’ve been talking mostly about psychology, right? We’re just starting to ask the question, is this really contained to this thing called psychology, or does the replication crisis maybe go further than that?
ALEX: As far as I can tell from my research and my reporting, I don’t think there is a good reason why there couldn’t be a replication crisis in the hard sciences. And there’s actually some evidence that there’s a big problem here. Bayer and Amgen - the pharmaceutical companies - they did these big replication studies where they revealed they could only replicate a very small proportion of medical findings when they were trying to develop different drugs, cancer-targeting drugs.
SAM: Oh shit.
SAM: I mean like ego-depletion not being true matters to some people, but I don’t think it’s as important as...
ALEX: As cancer drugs?
SAM: Yeah. Right. Fuck.
ALEX: There’s a recent meta analysis - which is like, a study of many studies - and it looked at findings published in some of the biggest neuroscience journals and found that maybe around 50 per cent of published findings may be false positives. Like, half! Half of it is not real! How do you like...
SAM: We’re going to spin out on some grad school level existentialism...
ALEX: Like that’s where I’m at.
SAM: There’s no truth, you know, we have no...
ALEX: How could you argue otherwise?
Harry Collins: Okay, my name is Harry Collins. I'm a professor at Cardiff University. I’m a fellow of the British Academy. My title is Distinguished Research Professor. Uh, I’ve been here for a while...
ALEX: Harry Collins is a sociologist of science at Cardiff University.
Collins: Well, a sociologist of scientific knowledge would be more accurate. That means somebody who studies the way people decide that some things are true and some things aren't true.
ALEX: Since the 1970s, Collins has written about replication in gravitational wave physics. I wanted to ask him how to think about truth in the light of the replication crisis.
Alex: So you know, people say that we are... we're living in an era of “post truth”, of “alternative facts”, and a rejection of expertise. Um, and that all sounds quite bad but you know, having taken this tour through psychology’s replication crisis and seeing kind of how structural problems can be invisible to experts themselves for so long... It kind of feels like maybe we really ought to be skeptical of expertise. What do you think about that?
Collins: No. I think it's absolutely the wrong conclusion to draw! And if we draw the conclusion that we ought to be skeptical of scientific expertise we’ll create a dystopia for ourselves in no time flat. The argument is as follows: science isn't perfect and science has never been perfect. We have to accept that what science is is a craft activity and like any other craft activity it's imperfect. Fortunately, science is generally done with integrity. So you've got a choice where you take your advice from: the powerful, celebrities, people who have a political interest in promulgating false truths; or people who are skilled craftspersons who work with integrity. What's important is that the sciences don't lose sight of their integrity. And there's huge pressure on the sciences to lose their integrity. There's huge pressure on the sciences to justify themselves by making more money for people, by producing entertainment, essentially, and by allowing themselves to become eroded in the same way as other institutions have become eroded, like the banks have become eroded. I think there is less danger in the case of science because of this underlying feature of science that it is defined by the search for truth. But scientists, scientists have got to stop selling themselves in the wrong marketplace, in the capitalist marketplace, in the political marketplace. And politicians have got to stop trying to force scientists to sell themselves in those marketplaces. That, it seems to me, is perhaps you know it may be the only hope for our societies.
[MUSIC: Sam Fenn - I See You Don’t Have a Door]
SAM: So if you listened to Cited episodes in the past or other episodes that we are going to produce this season, I think that the thing Collins is talking about, which are basically the structural problems of academia, the way it has become neoliberal, the way there are these pressures on academics...
ALEX: Everything is measured. Everything is gauged. The outputs and inputs...
SAM: The thing that’s attractive about what Collins says, to me, is that it kind of gets us out of that fog of just like “nothing’s true” and it gives us a mission. We need to reorient science so that there aren’t other undue pressures on it. So that scientists don’t feel that they need to… they need to...
ALEX: They’ve got to have the freedom to pursue truth. (laughs)
SAM: Yeah! But in order to do that, we would need to totally restructure universities and the way that they behave.
ALEX: Yes, yes.
SAM: Because right now there are all kinds of reasons why departments, individual academics, universities themselves, feel that they have to constantly prove themselves on the market as this exceptional place where major discoveries happen.
ALEX: But... if I'm going to trust the experts, I don’t think it’s unreasonable to ask for some kind of re-organization and clarification. And perhaps ironically, I think that it’s the psychologists that are leading this movement to reorient science around these values.
[MUSIC: Sam Fenn- Alex’s Detective Song]
I wanted to see that in action. I wanted to know what it’s like when a bunch of psychologists get in a room and try to fix science.
Alexa Tullett: Should we start by looking over that slide? Okay. You look for the slide...
Today, the Society for the Improvement of Psychological Science is hosting a “hackathon”. Psychologists from all over have assembled to try to figure out a new set of rules for research.
Researcher: That seems to me that there needs to be maybe something earlier on in the pre-registration flowchart that actually asks...
Researcher: Are there things you’d revise about this section, or other sections?
Researcher: What about alternative theories?
Researcher: Oh! Okay.
Researcher: But if the point is to combat publication bias, do we want to require people to tell the world about the study regardless of how it turns out?
Researcher: Yeah testing a generalization of a theory versus falsifying a theory. ‘Cause I think those two could vary...
Researcher: That’s a really good distinction.
Alex: So can I ask the three of you a collective question? I'm trying to figure out how to feel about this. Like I guess the question is like... Are the people in this room like, are they going to save science or...?
Tullett: Yeah! That's how you should feel. Right?
[MUSIC: Knocked the Door Right Off]
ALEX: Carl Sagan said science is a candle in the dark. Another writer said science is a cemetery of dead ideas.
We don’t yet know if any of the changes that psychologists are making to their science will really improve its reliability. Only time will tell. And I don’t know the future.
I don’t know if it’s fair to say that none of this would have happened without Daryl Bem, but I think that psychology, maybe all of science, owes Bem a debt of gratitude for being the impetus for change. Some even believe that this was his plan all along.
Bem: One possibility is I’m just a fraud! After all, I am a magician.
Bem: And the rumor was going around that actually what I was doing is I’m trying to play a fraud on the whole of psychology by showing how stupid they are with their methods by getting them to accept these experiments that are just fraudulent. So. (laughs) Needless to say, I don’t believe that.
Alex: Kind of, the epilogue to this paper that you published in 2011...
Alex: You have a passage in there from The Adventures of Alice in Wonderland.
Alex: And basically what’s happening is Alice is talking to the White Queen and she says, “One just can’t believe impossible things.” And do you remember the White Queen’s reply?
Bem: “Oh yes my dear. You just haven’t practiced enough. Why, I often believe at least six impossible things before breakfast.”
GORDON: This episode was produced by Alexander Kim. Edited by Sam Fenn and me, Gordon Katic.
Further production support from Polly Leger, Tom Lowe, and Emma Partridge.
“Repeat After Me” was initially made in partnership with the program Ideas, from CBC Radio. Nicola Lucsik of Ideas helped edit it, and the CBC shared production costs with Cited Media.
Our theme song and music is from our composer, Mike Barber. This episode also featured some songs from Sam Fenn and Garth Mullins. Our graphic designer is Dakota Koop is our graphic designer. And Cited’s production manager is David Tobiaz.
Cited’s executive producers are Gordon Katic, that’s me, as well as Sam Fenn.
If you like what you heard, do us a favour: Give us a rating or a review on iTunes--or wherever you found this podcast.
Cited is funded in part by the Social Science and Humanities Research Council.
This episode was made possible with a grant to discuss public distrust of science. The project was advised by Dr. Dave Ng at the University of British Columbia’s Michael Smith Labs. Dr Ng also gave some research guidance to Alex. As did Dr. Candis Callison--she helped Alex work through some of the philosophy of science in this piece. Dr. Ed Kroc was also helpful with understanding the statistics.
Cited is usually produced out of the Centre of Ethics at the University of Toronto. Today, because of this global pandemic, it’s being made in my bedroom. Both of those places are on the traditional land of Mississaugas of the Credit, the Anishnabeg, the Chippewa, the Haudenosaunee and the Wendat Peoples.
Cited is also produced out of the Michael Smith Laboratory at the University of British Columbia -- that’s on the unceded territories of the Musqueam, Squamish, and Tsleil-Waututh Nations.
SAM: You need me to do just some wild takes here, where you can just out of context, you can use whatever you want?
ALEX: Yeah, just react. Like, give me some surprise.
SAM: Woah. Shit. Really?
ALEX: Give me amazement
SAM: Oh my God.
ALEX: And then like, concern.
SAM: Are you sure about that? I mean, I don’t know.
SAM: Hmmm. Huh. That’s really.. I’m going to have to sit down and think about that.