On the “Reproducibility Crisis” in the Sciences

The field of psychology is in crisis. A recent project designed to test the veracity of many experiments in social psychology has returned disappointing results. The Reproducibility Project, an organized effort to reproduce the effects of several studies published in the psychological literature, only succeeded in replicating 36 percent of the studies that were chosen. While this does not prove that the studies were falsified, it does cast immense doubt on their veracity.

Those who do research in psychology naturally find this very distressing. The findings of the Reproducibility Project corroborate stereotypes asserting that psychology is an unrigorous “soft science.” Yet problems with the veracity of scientific studies are not limited to psychology. The findings of the Reproducibility Project could have been predicted by the landmark paper “Why Most Published Research Findings are False.” Without going into the gory mathematical details, the paper makes a statistical argument that given the small sample size and measured effects of most published scientific studies, it is probable that most of their conclusions are false. Perhaps surprisingly to the detractors of psychology, the author of this paper, John Ioannidis, is not a psychologist but a professor of medicine. The criticisms he makes were originally directed towards medical studies, not psychological studies.

So, according to this widely-cited paper, the social and life sciences are all in crisis. But I would also question the implicit definition of “crisis” I've been using up to now. The methodological foundations of the social and life sciences are in question – because of this, I say that the social and life sciences are “in crisis.” Yet it is commonplace for other academic fields to question the foundations of their own methodology. Consider, for example, philosophy. In 1979, the philosopher Richard Rorty wrote Philosophy and the Mirror of Nature, a book which argues that the problems posed by mainstream philosophy rest on an incorrect assumption about the nature of the relationship between the human mind and reality. More recently, Peter Unger published a book entitled Empty Ideas, which argues that most of contemporary American philosophy does not give us any information about concrete reality.

Yet it would not seem legitimate to say that philosophy is “in crisis” because of these methodological critiques. There is some sense that philosophy, as well as the humanities in general, are both less definite and more reflective disciplines than the sciences. In philosophy, questioning one’s own methodological assumptions is business as usual.

I suspect that part of the reason why critiques of the methodological foundations of philosophy are not taken as seriously as those of the sciences is that science is taken more seriously than philosophy (and the humanities in general) outside of academia. Many of the studies in psychology and medicine with questionable legitimacy were reported on in news articles for the general public. But you don’t see advancements in philosophy reported on in the press, at least not most of the time. Scientific studies seem concretely true in a way that philosophical arguments don’t.

But as the replication crisis has shown, there is nothing especially concrete about science as compared to other disciplines. To understand why science is nevertheless believed to be more reliably true than other disciplines, let’s take a look at the opinion of Richard Feynman, a great physicist and a famous detractor of philosophy, on the work of Baruch Spinoza, an important Dutch philosopher. In a 1979 interview with Omni magazine, Feynman said that Spinoza’s work is useless because “you can take every one of his propositions, and take the contrary propositions, and look at the world and you can’t tell which is right.”

This is an odd criticism of philosophy because nobody disputes that its subject matter includes things about which you cannot tell anything by looking at the world. In fact, that is part of why philosophy is uniquely valuable. What I suspect is at work in the Feynman quote—and in the air of certainty that science has—is a disbelief in the ability to come to correct conclusions through argument.

But as the replication crisis has shown us, even the process of looking at the world and drawing conclusions about it has a methodology that must be settled on by argument. I suspect that this 'crisis' is not so much a crisis for science as it is a crisis for the empiricists’ dream of coming to conclusions that are irrefutable because no argument is necessary to substantiate them. Psychology and medicine will be just fine—their foundations rested on argument before the replication crisis, and those foundations will continue to rest on argument after it.

William Schumacher PO '18 is a philosophy major from San Jose, Calif.

Facebook Comments

Leave a Reply