Why it’s important to look for the limitations in coronavirus studies

This week, I got to talk about vaccines and vaccine information with Natalie Dean, and The Verge’s Nicole Wetsman and Nilay Patel on The Vergecast. It was a great conversation with Dean, who is an assistant professor of biostatistics at the University of Florida and an expert in infectious disease epidemiology who has worked on designing vaccine trials.
After the podcast, I got to ask something that had been on my mind: how did she – a person who works with scientific data every day – feel about the flood of early versions of scientific studies being posted online? And how much attention should the general public be paying to them?
Before we get to her answer, let’s take a quick step back to talk about what these early studies are. Scientists call them preprints, and they are studies that get posted online before they’ve been published in a scientific journal. They haven’t been fully vetted by other experts (a process called peer review). Preprints are helpful to scientists because they quickly put data out into the world that could be useful to other researchers.
The downside? They’re essentially first drafts, and conclusions could change after the researchers get feedback. Balancing the need for speedy data with the need for reviewing work before it goes public is especially tricky during the pandemic – and both the pace of publication and attention to preprints has definitely amped up in the past several months.
“There already were these preprint servers before, but they’ve never been used to this degree,” Dean says. For a long time, prestigious journals would refuse to publish a paper if it had been posted elsewhere, which meant scientists kept their research under wraps until they had their final draft.
That’s started to change over the last few years as journals have loosened their restrictions. Now there’s a flood of coronavirus-related papers available online, and some can attract a lot of attention – and confusion. Not all of that attention is a good thing, as Kelsey Piper wrote for Vox back in May:
The newer, faster pace could mean that badly flawed pre-prints get widely shared and covered in the media, fueling the spread of misinformation and forcing other scientists to waste valuable time by publicly debunking papers that would ordinarily be rejected in the peer-review process.
Piper pointed out that there were plenty of advantages to preprints, too – they force data to move at 21st-century speeds instead of staying stuck in the 19th century, and when lots of experts are actively participating in the process, they can cut out a ton of unnecessary downtime.
That’s especially true in a pandemic. “My feeling is the benefits outweigh the risks in the situation where time is of the essence, and we’re all learning from each other,” Dean says. And the pace of academic publishing is so slow, that I just don’t think it is still viable.”
But there’s still that one big disadvantage. “Scientists know how to evaluate the limitations of preprints. But it’s challenging for the public to do so,” Dean says.
She’s got a good tip to help readers start assessing news stories about preprints: look for the limitations. A story that tells you what the study doesn’t is helpful. This doesn’t just apply to preprints, either: it’s good advice for any story about a scientific paper. All studies have limitations; good reporters tell you what they are.
In other words, look for the peer review in a news story, especially when it comes to a preprint. Look for it on Twitter, where Dean and other researchers have thrown cold water on extraordinary claims. Look for it in dedicated volunteer efforts, like Johns Hopkins’ 2019 Novel Coronavirus Research Compendium which actively assesses papers that get a lot of media attention. Look for it at Rapid Reviews: COVID-19, an open-access journal explicitly devoted to peer-reviewing COVID-19 preprints. And most of all, as we keep wading through a flood of information, take a beat to look skeptically on claims that seem incredible.
“The more extreme a result seems, the more evidence we need to support that,” Dean says.
Here’s what else is going on this week:
Around…