Snake Oil for Everyone!

Here in America, the term "snake oil" is used to represent misleading cure-alls. In China, we actually had to live through a real snake oil pitch after a tour and hike through Huang San (Yellow Mountain). They rubbed snake oil all over our legs and proclaimed that the reptilian extract could cure a litany of ailments. Sha-right.

The snake oil claim reminds me of how important clinical trials are in our understanding of what works to benefit our health and what does not. In the epidemiology course I took a couple of years ago, we read about the different types of studies and why researchers would choose one over another. There are a lot more subtleties to these studies than meets the eye, certainly a lot more than those marketing campaigns advertise.

The NYTimes has a good article that boils down some of the principles of medical studies and clinical trials, Searching for Clarity: A Primer on Medical Studies.

The article states that there are three important principles when designing a study that will reach the medical truth.

1. Compare like with like

"Cynthia Pearson, executive director of the National Women’s Health Network, has a favorite example of how easy it is to be fooled. Study after study found that women taking estrogen had less heart disease than women who did not. But, Ms. Pearson says, it turns out that women who faithfully take any medication for years — even a sugar pill — are different from women who don’t. The compliant pill-takers tend to be healthier, perhaps because they follow doctor’s orders."

When designing a study it is a laborious process to match outcomes appropriately so you can negate factors such as the one stated above. A good study may ensure that the healthy will be compared with the healthy, or smokers with smokers, or young with young... etc. Well, first they have to decide which of the millions of factors might actually require this kind of effort.

2. The bigger the group studied, the more reliable the conclusion.

This point involves a bit of knowledge about statistical analysis.

"... the real result of a study is not a single number, like a 20 percent reduction in risk. Instead, it’s a range of numbers that represent a so-called margin of error, like a 5 to 35 percent reduction in risk. The larger the sample size, the smaller the margin of error. Small studies have large uncertainties in results, making it difficult to know where the truth lies."

And we know that we'll never see margins of error advertised with the product claim.

3. Consider the strength of the supporting evidence separate from the study

The example given for this point is interesting. A professor shows data to his students. The data seems to indicate that the studied factor helps patients to overcome illness, though by a small margin. By raising their hands, the students show that they believe the study, but when the professor revels that the factor is prayer, hands fall.

"...there is no plausible scientific explanation of why prayer should have that effect. When no such explanation or evidence exists, the bar is higher. It takes more clinical trial evidence to make a result credible."

So the next time you read a claim, like a certain yogurt will help you lower your cholesterol, think like a skeptic. If it sounds too good to be true, then it probably is. Or you can do your own fact finding. Chances are the studies they use to make those claims are sitting in journals that may be accessible through the library. Then you can have some fun reading.