Wednesday, December 24, 2008

Something fishy about "no evidence"

Searching Google for "no evidence" yields "about 21,300,000" hits. It seems we're keen to deny that there is any empirical support for countless different claims. For example, reports that there is "No Evidence For Fish Oil Benefit In Arrhythmias" based on a systematic review just published in BMJ. (Full disclosure: I have previously participated in research on omega-3 fatty acids, however I have no related financial interests.) What does the review itself say?
This is the first systematic review attempting to evaluate whether the protective mechanism of fish oil supplementation is related to a reduction of arrhythmic episodes determined either by a reduction in implantable cardiac defibrillator interventions or a reduction in sudden cardiac death. We found a neutral effect on these two outcomes. The confidence intervals for these outcomes were wide and a beneficial effect up to a 45-48% relative risk reduction cannot be excluded.
To better appreciate this, here's their Figure 2:
Note that of the three studies that looked at the proportion of implanted defibrillators that were triggered, one showed a statistically significant effect in favour of fish oil, and the other two did not show statistically significant effects (one favoured placebo and the other favoured fish oil). Six studies looked at sudden cardiac death in patients taking fish oil compared to those taking placebo. Only one was statistically significant, and it favoured fish oil. Of the five studies that did not show statistically significant effects, two favoured fish oil, and three favoured placebo.

The diamond shapes in the figure show the pooled estimates with their 95% confidence intervals: in each case the diamond overlaps an odds ratio of 1, indicating that the overall effect is not statistically significant. And when there's a non-statistically significant effect, it is common practice to say there is "no evidence". But that can be very misleading! After all, two of the individual studies did show a significant benefit of fish oil. So what's going on? Well, for starters, there's some indication of heterogeneity between the studies (particularly in the case of the defibrillator studies). But it also seems that more large studies are needed: a good deal of the variation in results between the studies may simply be due to the play of chance. Quite substantial benefits of fish oil are entirely plausible: relative risk reductions of as much as 45-48%!

What is "no evidence"?

Consider the figure below:

At the bottom there is a gray axis line with tick marks and a vertical gray line indicating the "null" value (where there is no preference one way or the other). The blue line with arrows at each end represents an infinitely wide confidence interval. This is the most straightforward representation of "no evidence": there is simply no empirical information to indicate what the true effect might be.

But suppose we have a very small sample, that is, one that provides almost no empirical evidence. The figure might become:

The only difference is the blue dot on the confidence interval just a bit to the right of the null line. It represents the point estimate based on a very small amount of empirical information. Of course it could equally well have been on the left hand side (or perhaps directly on the null line). Regardless, the confidence interval is still very wide, so very little can be said about the true effect. With such a wide confidence interval, the location of the point estimate is almost irrelevant.

Finally, suppose that a reasonably large sample is available:

I have left the point estimate at the same place. The confidence interval no longer has arrows on either end and is relatively narrow. However it still overlaps the null line. That means the estimate is not statistically significant. Sometimes this sort of situation is described as showing "no evidence of an effect". But as I noted above, that's quite misleading language. In fact, what this situation shows is indeed evidence—evidence that any effect likely has a magnitude of no more than two tick marks (whatever they represent) on the right hand side of the null line or a magnitude of no more than about a half a tick mark on the left hand side of the null line.

But here's the tricky part: what do those ticks represent? Suppose the axis represents annual cost savings that might result from implementing a certain type of federal government program. If each tick mark represents $1000, then we have estimated that the program will cost at most $500 a year and save at most $2000 a year. In other words, the program has been shown to be effectively revenue neutral: the evidence suggests that the cost/cost-savings of the program will not be important. On the other hand, if each tick mark represents one million dollars, most of us would feel that the jury's just not in yet. A possible cost of $500 a year is a drop in the bucket, but $500,000 a year is something else entirely.

In a way, money is the easiest measure to evaluate like this. Things like safety are much harder. For example, if the evidence suggests that a certain chemical may increase the rate of certain types of cancer, but the findings are not statistically significant, what can we conclude? Can the manufacturer claim that there's "no evidence" the chemical is harmful? Can health activists claim that there's "no evidence" the chemical is safe?

I would argue that the term "no evidence" is inappropriate in either case. The underlying questions remain: what is required in order to conclude that a chemical is harmful or that it is safe? Ultimately there's no getting around the issue of how large a difference (in, for example, cancer rates) has to be in order to be considered important. And that's a rather uncomfortable question.

Labels: , , , , , , , ,

Bookmark and Share


Post a Comment

<< Home