looking critically at sciencey claims: a layperson’s guide

The web world is full of claims about what science has supposedly proven or what a recent study has shown. Many of these claims are dubious; some are pure distilled bullshit, some actually have merit. Here I present my guide to separating the wheat of wisdom from the chaff of fraud, hyperbole, and pseudoscience.

First, start with a bold claim from the interwebz. “According to a new scientific study…” …maybe it’s a nutritional claim, maybe it relates to climate change, maybe vaccine safety, or maybe… any of dozens of topics. How do we go about verifying the veracity of this claim?

Look at the source. Inevitably this is a secondary source, rather than an actual journal article detailing scientific research. Is it from a respected purveyor of science news to laypeople? A mainstream news site? An unfamiliar website? A known purveyor of ka-ka? If it’s from a well-known manure spreader like NaturalNews, CollectiveEvolution, Mercola, or WorldNetDaily, you’ve got a big red flag already. If the site is unfamiliar to you, check it out. Look at the other articles on the site. Is it full of paranoid fantasies? Biblical prophesy? Really wacky New Age twaddle? If the site is a mainstream news site, take the conclusions with a big grain of salt; many otherwise respected news sites do a really crappy job of science reporting. If the site is a quality science site, good. But it’s still not exempt from critical analysis.

Read the source. Does it commit flagrant logical fallacies? Perhaps the most common is assuming that correlation proves causation. Grandiose extrapolations? Another red flag is a persecution complex. They mocked Galileo, they mock me, therefore I’m just like Galileo…. uhhh… no.

Lool at the source’s sources. If we’re really looking at a new scientific study, find the study. If your secondary source doesn’t link to the actual study, that’s a little red flag. If it doesn’t provide necessary clues to find the study, it’s a big red flag. If there is no peer-reviewed study to be found, you’re probably dealing with a crackpot making wild claims without foundation.

Look at the actual scientific paper. First off, most serious scientific journals don’t let everybody browse the entire journal. This sucks, but it’s the way things are at this point. Generally, you can read the abstract for free, but if you want to read the whole thing you have to fork over the big bucks. Fortunately, the abstact is usually enough for our puposes as laypeople. The abstract outlines the question, procedures, and conclusions of the study briefly. Often, when I’ve looked up the source for a claim, I’ve learned that the author of that secondary source article completely misrepresented the actual findings. When this happens, you can write off the original claim as crap. Often, the analysis of the secondary source author is shakey, but not outrageous. Perhaps a tentative call for further study has been stretched into a bold conclusion. Perhaps a petri dish study of tissue in a lab has been extrapolated grandly into (yet another) cure for cancer. Another question you should be able to answer from the abstract: is thisĀ  original research, or are the authors just crunching numbers? Meta-analyses can be useful, but often analyzing other people’s data is just a cheap way for someone to get published. Tally up the number and size of red flags accordingly.

If you haven’t completely rejected the original claim at this point, check out the journal which published the study. Is it a known, prestigious journal like Nature, Cell, JAMA, or Lancet? This brave new world we live in is full of dubious “peer-reviewed” journals which publish anything that somebody pays them to publish. It may take some Googling to assess the reputation and credibility of a journal.

Then, check out the authors of the study. Are the authors experts in the field they’re writing about? One category of bad science I’ve noticed a lot lately is experts in one field writing about a completely different field. When a computer scientist makes wild claims about biochemistry and the biochemists laugh at the study’s absurdity, I’m inclined to believe the biochemists. An astronomer doesn’t necessarily know any more about mammalian paleontology that you or I do, regardless of how many degrees that person has earned.

Please, people, follow this procedure (or a similar one) before you share that sciencey-sounding meme which just happens to reinforce what you already believe.

This entry was posted in general, science. Bookmark the permalink.

Leave a Reply