Developments in Medicine
How Do We Know What We Know: The Chain of Evidence
Every day - in the newspapers, on television or online - we hear about some new development in medicine. Sometimes the evidence for this new recommendation is very strong, and sometimes it is very flimsy. How do we tell these apart? And how does it happen that at times a recommendation that seemed to be very strongly supported turns out to be quite wrong - as when the recommendation that hormone replacement therapy is good for preventing coronary heart disease, a recommendation supported by multiple articles over a period of many years, was proven entirely wrong by the Women's Health Initiative study, with that negative finding now confirmed by other good studies?
What I would like to do, without going into overly complex detail, is to describe the "chain of evidence" in science that leads to conclusions that are progressively more certain, so that you can form your own opinion about the likelihood that what you see or hear in the media is correct.
The first part of this chain is epidemiology, which is the study of disease patterns in populations. This type of study provides us with clues to the causes of disease, but not with certain evidence. For example, it is clear that the Japanese have much lower rates of coronary disease then do Americans or Western Europeans, and their diet is also much lower in saturated fat and cholesterol and their blood cholesterol levels lower, and so the epidemiology suggests that there may be a connection. But of course the Japanese differ in many ways from other populations and so this information is suggestive and no more.
The next step in the chain includes looking at animal studies and at laboratory scientific investigations. If we take a group of pigs and feed them a diet high in cholesterol and saturated fat, sure enough, their blood cholesterol levels become very high and they develop coronary artery disease. They don't smoke, they don't have high blood pressure or diabetes, and they don't seem overly stressed! Likewise, it can be shown in a laboratory setting that high saturated fat intake causes changes in the linings of the arteries that make it more likely for disease to develop. So the evidence is becoming clearer.
But the final link in the chain is often the randomized controlled trial. These are studies where a group of people are randomly divided (as by flipping a coin) into two or more groups, depending on the study, and then the groups are assigned to different conditions. Perhaps one group is assigned to eat a high saturated fat diet and the other to not eat such a diet. The advantage of this type of study is that we can control for all of the other factors that may confuse the issue. Since people in the two groups are assigned randomly, if there are enough people in the study the two groups will have similar levels of education, similar patterns of exercise, similar blood cholesterol levels at the start of the study, and will be similar in many other ways so that we can reasonably conclude that any differences that develop will be due to the one factor that we are changing. For the connection between diet, blood cholesterol levels, and coronary heart disease these types of studies have been done, as have studies looking at the reverse - taking patients who already have coronary disease, randomly dividing them into two groups, providing one group with usual care (the control group) and then providing the other group either with a very low saturated fat, low cholesterol diet, or with a medication, such as a statin, that lowers blood levels of cholesterol. All of these studies have confirmed beyond any doubt the connection between diet and blood cholesterol levels, and between cholesterol levels and atherosclerotic disease in the arteries of the heart, the brain, and elsewhere in the body.
What happened with hormone replacement therapy? All of the earlier studies were observational, and showed that women who took hormone replacement therapy had less coronary disease than women who did not. But the reality is that women who take hormone replacement therapy of their own free will are different than those who do not - perhaps they are of higher socioeconomic status, perhaps they exercise more, perhaps they are more educated, or perhaps they are different in ways that we simply do not understand. There are statistical and computer methods to help us adjust for the differences we know about, but they cannot help us adjust for the differences we do not know about. The Women's Health Initiative, a 10 year controlled trial of over 50,000 women, showed that hormone replacement therapy was harmful. But it took this enormously difficult and expensive study to make that clear.
Let me give you another example of how this works. There is a lot of evidence that people who drink alcohol in moderation have less coronary disease than those who do not drink at all. The data is largely observational, coming from studies of populations. But of course, those who drink are different than those who do not in many ways that are hard to account for. There is some additional suggestive laboratory evidence that some of the compounds in red wine may be of benefit, but the epidemiology suggests that the beneficial effect is not limited to red wine and is equally seen with any form of alcohol. There has never been a controlled clinical trial of the benefits of alcohol and it is very unlikely that there ever will be one. Can you imagine taking 10,000 people, dividing them into two groups, and telling 5000 of them that they cannot drink at all for the next five years and the other 5000 that they must drink one glass of wine every day for the next five years? Obviously impossible. So this is an example of a recommendation that may or may not be correct. As an aside, I would also point out that the toxic/therapeutic ratio of alcohol is very narrow - even the available studies all point out that one drink may be beneficial, two is probably not, and three is downright harmful!
One more example. Lots of evidence suggests that fruits and vegetables are good for health. Fruits and vegetables contain antioxidants and so it was thought that antioxidants must be good for health. For years physicians prescribed beta-carotene and vitamin E for their patients. But now there have been many well-controlled, carefully done studies of these and other antioxidants provided as supplements, and none have shown benefit, with some showing harm. Why might this happen? As was once pointed out to me by a member of the tomato industry, a tomato has approximately 200 different antioxidants, and it is an act of extraordinary faith to assume that taking one of those antioxidants in gross and unbalanced access is beneficial! It is better to eat the tomato.