How to Spot a Good Medical Study
Page 1 of 2
Let’s say you’re trying to decide for yourself whether or not a particular alternative medicine technique has any benefit. You see on the company’s website that there are medical studies showing that the treatment was effective. Assuming they provide a reference to these studies, how do you know that you can trust it?
While the more you know about science and medicine, the more likely you’ll be able to spot fraudulent claims, you don’t need to be an expert in order to spot lies and half-truths in spurious medical claims. Let’s outline what you should look for in a medical study without going back to school for a medical degree.
(If you want to know why it’s important to have a reference to the actual study, see my episode on How Not to Be Scammed by Internet Claims.)
Sponsor: This episode is brought to you by the audiobook edition of Rogue Code by Mark Russinovich. In this intense thriller, cyber security expert Jeff Aiken discovers that the New York Stock Exchange has been hacked... and someone on the inside knows. Listen to an excerpt at www.macmillanaudio.com/RogueCodeAudio.
C Is For Cookie
One of the problems with carrying out sound medical studies is that there are various psychological phenomena that can interfere with the results.
Let’s say you’re reading about a study that says eating cookies made with some rye flour mixed in causes you to jump higher than cookies made with wheat flour alone. Our study will have two groups in it: the control group and the treatment group. In the control group, participants will be given regular cookies, while in the treatment group, participants will be given the cookies that have rye flour mixed in.
Without a control group, we have no way to know whether it's the rye cookie that is affecting jumping ability or something else. This kind of study is called a controlled study.
This Study Is So Random
Let’s say we have all of our volunteers meet at our test facility and we’ll call them back one at a time, in the order that they arrived. We do the rye cookie group first, then the plain cookie group. We perform our tests and notice that just as we predicted, the rye cookie people do jump higher after eating their cookies. Are we done? Are rye cookies the key to basketball prowess we’ve been waiting for?
Unfortunately we’ve failed to randomize our groups and so we’ve fallen into a trap called selection bias. What we didn’t realize is that all of the people who arrived first for the study, did so because they wake up early, exercise, eat a healthy breakfast, and then head out ready to seize the day. They arrived bright and early, first in line for the test. Whereas the other slovenly volunteers barely rolled out of bed in time to make it to the test because they stayed up all night playing video games and eating cheese balls.
By assigning the earliest arrivals all to a single group, we’ve inadvertently biased our results. The fix for this problem is pretty simple; we just need a “randomized trial.” The easiest way to go about this is to give everyone a number, then randomly choose numbers to be a part of either the treatment group or the control group.