Friday, March 28, 2008

Upping the anti (depressant)

A paper on antidepressants by Kirsch and co-authors published last month in PLoS Medicine has received a lot of attention. The antidepressants studied are the six most widely prescribed approved between 1987 and 1999: Prozac, Paxil, Effexor, Serzone, Zoloft, and Celexa.

The Editors' Summary explains:
The researchers obtained data on all the clinical trials submitted to the FDA ... They then used meta-analytic techniques to investigate whether the initial severity of depression affected the HRSD [Hamilton Rating Scale for Depression] improvement scores for the drug and placebo groups in these trials. They confirmed first that the overall effect of these new generation of antidepressants was below the recommended criteria for clinical significance. Then they showed that there was virtually no difference in the improvement scores for drug and placebo in patients with moderate depression and only a small and clinically insignificant difference among patients with very severe depression. The difference in improvement between the antidepressant and placebo reached clinical significance, however, in patients with initial HRSD scores of more than 28—that is, in the most severely depressed patients. Additional analyses indicated that the apparent clinical effectiveness of the antidepressants among these most severely depressed patients reflected a decreased responsiveness to placebo rather than an increased responsiveness to antidepressants.
The press simplified it further. The MSNBC headline was "Antidepressants may not help many patients". The Guardian announced: "Prozac, used by 40m people, does not work say scientists".

Reactions, adverse and otherwise

There were reactions to the effect that "we've know all along antidepressants don't work" and at the other extreme "nothing could ever convince me that antidepressants don't work."

A lot of reaction came from people who believe they have benefited from antidepressants. See, for example, the comments following a summary of the study at depression.about.com.

The blogosphere had plenty of reactions: FuturePundit, Action Potential (the Nature Neuroscience blog), The MindFields College Blog, and on and on.

And the journal itself, PLoS Medicine, had an enormous number of responses to the paper.

Betta check the meta

The heart of the findings in this paper is the meta-analysis itself, and when I examined it, two things jumped out immediately. The figure below shows them both.
There's a lot to look at in the figure. The red triangles represent the results of the patients who received the antidepressant. The bigger the triangles, the more weight they receive in the analysis. Similarly, the circles represent the placebo results. The solid red curve is a model fit to the antidepressant results. The dashed blue curve is a model fit to the placebo results. The green region shows where there is a clinically important difference between the curves.

First, look at the vertical axis, labeled "Improvement (d)" and ranging from 0 to 2. This is the mean improvement in the Hamilton Rating Scale for Depression (HRSD), but it has been divided by the standard deviation. Why divide it by the standard deviation? Well this is what you might do if each study was using a different rating scale, in order to standardize things. But here it's not necessary. Each study used the HRSD, so it would be better not to standardize.

Second, if triangles represent antidepressant results and circles represent placebo results from the studies, how do they pair up? Each study has two "arms": an antidepressant arm and a placebo arm, but on the figure you can't tell which triangle belongs with which circle. This points to an important problem: the authors meta-analyzed the antidepressant arms separately from the placebo arms. But the studies were randomized controlled trials, which means that within each study the two arms are comparable. Ignoring this can introduce bias. The standard approach in meta-analysis is to compute a contrast between the two arms within each study, and then meta-analyze these contrasts.

But do either of these points make much of a difference? It turns out that they do. PJ Leonard took the trouble of rerunning the analyses using raw HRSD scores and the standard meta-analytic approach rather than the separate-arms analysis of Kirsch and co-authors, and obtained an effect about 50% larger than they did, and stronger evidence of clinical importance. Leonard also performed a regression analysis corresponding to the figure above.

Robert Waldmann has also done some interesting work on this.

Overcoming depression: there's no silver bullet

The evidence doesn't seem to support the notion that antidepressants "don't work". The overheated media response to this article was unfortunate. And that's a topic in itself.

Nonetheless, it seems that on average the effect of antidepressants is hardly overwhelming. So far there's no silver bullet for depression. Drugs can help, but so can other interventions. Including kindness and understanding.

Update: 11Apr2008 Thanks for a post on The Home for Wayward Statisticians, I found a couple more interesting links. One is by Mark Liberman on Language Log. The other is an editorial in Nature.

Labels: , , ,

Bookmark and Share

33 Comments:

Blogger Raywat Deonandan said...

Maybe you should write an article to this effect either for the lay media or for CMAJ?

8:14 AM, March 29, 2008  
Anonymous Wake UP AMERICA! said...

the bigger question is why is the US dollar depressed? Who is causing the value of the dollar to go down and why?

10:31 AM, March 29, 2008  
Blogger Nick Barrowman said...

Ray: good idea. There's actually more to comment on, for example the short duration of most of the trials.

1:32 PM, March 29, 2008  
Blogger Robert said...

Thanks for the link

2:15 PM, March 29, 2008  
Blogger Robert said...

Oh also it is worth pointing out that the null hypothesis that the Kirsch et al results are unbiased can be tested and that it is overwhelmingly rejected (click the link to me please to see).

2:22 PM, March 29, 2008  
Blogger Ward said...

Do you think there is a notion in the media or the public in general that since meta-analysis uses many individual studies it is somehow less biased or more credible than a single trial? It seems that way to me but I am just a layman. I suspect this bias in favor of meta-analysis is not shared but genuine experts?

8:52 PM, March 29, 2008  
Anonymous Anonymous said...

Who came up with the brilliant idea of testing severely depressed people with placebo medication?

9:02 PM, March 29, 2008  
Blogger Nick Barrowman said...

Robert: I'm really impressed by the work you and PJ have done on this. Are you planning to publish any of it?

Ward: There has always been some controversy associated with meta-analysis. Psychologist Hans Eysenck famously referred to meta-analysis as "an exercise in mega-silliness". A meta-analysis may be either more credible or less credible than a single trial. Note that meta-analyzing a bunch of biased trials gives a more precise biased estimate—hardly progress! Meta-analysis is a good thing, but it is no panacea.

Anonymous: There are definitely ethical issues with randomizing people to possibly receive a placebo. Normally the control group should receive "standard of care". But in the case of depression, I'm not sure what that is.

11:48 PM, March 29, 2008  
Blogger Ithaca said...

Amazing graphic. How did you do it?

3:12 PM, March 30, 2008  
Blogger Nick Barrowman said...

I can't claim any credit for that figure. It's from the Kirsch et al. article in PLoS Medicine. But I would make a figure like that using the R language. A very interesting blog on data visualization that you might want to check out is information aesthetics.

8:57 PM, March 30, 2008  
Anonymous Andy Alt said...

The media had something to lose by broadcasting that story. Drugs ads bring in major revenue. Open a Time or US News and World Report, you'll see pharm ads no doubt about it. And tv networks show a fine share of drug ads. I was surprised on how many media outlets carried that story. It hit a nerve because so many people have tried to get treatment and just wound up with side-effects or withdrawal symptoms. If there wasn't any real meat to the study, the whole issue would have just died. There wouldn't be a need for people to actively discredit the work; the billions of drugs and dollars should be speaking for themselves. The study may very well be flawed but there are enough cases of death by legal drugs and patients who don't improve after ten years to raise suspicion. I believe it likely the adage applies, "methinks thou dost protest too much." If only the pharms weren't making billions off people with mental illness, the demographic that are the most vulnerable and least capable of self-defense, then the story would be much easier to dismiss. Time will tell. Oh, yes, by the way, I have been treated for mental illness, by many "qualified" doctors. Overprescribing and under-diagnosing is not a good method of gaining my confidence.

12:08 AM, March 31, 2008  
Blogger Nick Barrowman said...

Thanks for your comment, Andy. Concern about whether the drug companies might be distorting the evidence about antidepressants was the motivation for accessing study data from the FDA. Although the the Kirsch et al. meta-analysis was flawed, I think P.J. Leonard's reanalysis shows that the effects of antidepressants are modest compared to placebo (at least for relatively short-term treatment periods).

Mixing money, science, and media is 21st-century alchemy. It's unlikely to produce gold.

7:58 AM, March 31, 2008  
Blogger pj said...

Although the effects of antidepressants are somewhat limited in these (fairly short) pre-licencing trials the evidence for any other interventions (e.g. CBT) are not really any better.

I think this partly represents the difficulties in treating depression (i.e. treatments truly don't work brilliantly - except maybe ECT), and the limitations in measuring the effect of these interventions (e.g. the depression scales used for the studies have their limitations, particularly, as many commenters on this paper have noted, with respect to their ordinal nature).

I do think we need to be careful in interpreting the improvement 'due to placebo' since this is more than simply a response to being giving a placebo pill and includes much more non-specific effects such as regression to the mean (i.e. we can't necessarily conclude that giving a sugar pill to people will give massive effects over and above what would happen if we completely ignored these same people).

With respect to the sins of Big Pharma - there is plenty of published evidence that effect sizes are inflated by publication bias and practice further influenced by more subtle spin, which is why compulsory trial registration and data provision is way overdue, but it is interesting to note Robert Waldmann's finding that Kirsch et al have actually introduced more bias into their meta-analysis by the methods they used than was introduced by publication bias.

Which just goes to show that bias is something we must always remain alert to. I think this study has been given a very easy ride methodologically, and when the findings are rejected it is often for very dodgy anecdotal reasons. The selective attention of the media is obvious, but the lack of scrutiny by scientists has been a deriliction of duty.

I suppose there might be a few devastating critiques sitting in peer review for publication in 2009 - but then the damage will have been well and truly done, with the 'findings' entering the medical literature and the popular consciousness.

I think andy alt's Shakespearian reference is a bit off target since neither I nor Robert Waldmann are exactly shills for Big Pharma.

1:12 PM, March 31, 2008  
Blogger Nick Barrowman said...

PJ, when you say "this study has been given a very easy ride methodologically," do you mean during peer review or after publication? It seems to me that after publication the study has had a lot of scrutiny. The media haven't scrutinized it methodologically as far as I know, but do they ever do that? Should they be expected to?

6:50 PM, March 31, 2008  
Blogger pj said...

Well I certainly think the peer review process has been deficient in this case.

But what I was getting at was that many people have accepted the methodology uncritically and then rejected the results on the basis of prejudice and 'clinical judgement'. I wouldn't say there has been much scrutiny of the paper, its methods and results, rather its message has been widely reported.

And no, I don't think it is the job of the media to dissect the methodology (although I expect science journalists to be able to detect the most blatant flaws, which often seems beyond them, and to accurately report the findings, and not just the spin in the discussion) but it is the job of other scientists to point out when the methods of a study are dodgy - something that actually happens quite rarely, and which would be facilitated by things like rapid responses to articles (as in the case of journals such as the PLoS and BMJ).

7:39 AM, April 01, 2008  
Blogger alex said...

As someone who has been prescribed many of these tested drugs in previous years, some through no choice of my own when I was a child, my concern is for those currently on the medication. I think I'd be feeling extremely confused, distressed, angry and yes, depressed right now with both the medical world and the press for its flamboyant coverage!

6:57 PM, April 01, 2008  
Blogger Nick Barrowman said...

Unfortunately, media coverage of medical studies sometimes (often?) does more harm than good. It's bit like following the stock market on a day-to-day basis: there's an awful lot of noise, and a good deal of "irrational exuberance". I wonder if the press should stick to reporting on research that's been at least somewhat time-tested and accepted by the academic community.

7:43 PM, April 01, 2008  
Blogger Ed T. said...

What did we do before anti-depressants? Maybe there is a link between our hectic modern world and a rise in depression. Or maybe people weren't sold drugs they don't need. -Ed T.

2:29 AM, April 02, 2008  
Blogger Nick Barrowman said...

Has there been a rise in depression? The evidence is not so clear. Still, I agree with you that the way a society is organized can have a profound impact on mental illness. It would be interesting to look at the history of the use of drugs to treat depression. By the way, I just noticed that the Wikipedia entry on Clinical Depression refers to the Kirsch et al. study, but without any methodological caveats.

9:08 AM, April 02, 2008  
Blogger AdamNiemczyk said...

I think that there are much better ways the help drpressed people then to just give them some meds!
There are probably more healthier ways to help depression then taking pills.

6:18 PM, April 03, 2008  
Blogger L-girl said...

"What did we do before anti-depressants?"

Many more people suffered their whole lives. Many more people's relationships were ruined. Many more people gave up.

Depression is a disease. When you don't have it, it's easy to dismiss it as "modern life".

Drugs are not the only way to treat depression, but they are an essential tool for many people.

Nick, thanks for this excellent post.

5:57 PM, April 04, 2008  
Blogger Fox Williams said...

I've read a lot on people claiming that antidepressants can cause greater levels of depression in some people. Is this true? Just curious... maybe you could post about it.

12:03 PM, April 05, 2008  
Blogger pj said...

Very difficult to prove that antidepressants cause increased depression in some people since they decrease depression on average. So you couldn't rule out the normal waxing and waning of depressive symptoms that just happens to have coincided with drug administration in a particular individual, or nocebo type effects. It is well attested that side effects are suffered within days, whereas the antidepressant effect usually takes weeks to manifest - so in the first week or so you could possibly be worse off than before you started the drugs (the agitation that SSRIs can cause is particularly problematic).

I suppose there are methods you could use to look at whether a subgroup within the drug group of a trial seems to be getting disproportionately more depressed - but that'd be difficult and probably unconvincing.

What you'd need to do is identify a subgroup of people who you think get more depressed with antidepressants and then do a placebo controlled trial on them.

1:50 PM, April 05, 2008  
Blogger Adam said...

You might want to check out some of the recent literature on Exercise effects on depression. They have been finding that exercise has better short term effects on depression than the drugs do long term. Obviously, you need some combination of both, but exercise has been implicated in the release of the VGF protein which promotes neurogenesis, repairing some of the cellular mechanisms that failed leading to depression. Just a thought. But its an interesting development that natural treatments are often just as beneficial as man-made ones. We just haven't figured them out yet.

12:39 PM, April 07, 2008  
Blogger djc said...

I've been on Effexor since 2000. I suspect that many people say "antidepressants don't work," when what they really mean is, "antidepressants don't make me happy."

To me, it's like saying, "This damned insulin won't grow new beta cells in the Islets of Langherans in my pancreas!"

I don't think any class of antidepressant will bring happiness. At best, it allows us to function, to sleep better but not perfectly, to process information better, but not on a genius level. I try to manage my disorder. I have no illusions about curing it anytime soon.

djc

www.12thandrepublic.blogspot.com

4:30 PM, April 07, 2008  
Blogger L-girl said...

To me, it's like saying, "This damned insulin won't grow new beta cells in the Islets of Langherans in my pancreas!"

Excellent analogy. I always make the analogy to diabetes or arthritis, which, like depression, are common chronic diseases. No one tells people with diabetes not to take meds.

5:10 PM, April 07, 2008  
Blogger Jeryd Smith said...

There seems to be a long, although unspoken, history of depression in my family. As a whole, my family seems to be against medication in general for no other reason than that we just don't like to take it and we all have a hidden skepticism of the real effects.
Having said that, one of my family members decided that the depression too great to ignore, or to try to fight alone. At the advice of a mental health professional, my family member started on medication to help bring a chemical balance.
I gotta say that during the prescription, there were noticable positive effects, but the counter side was a very altered personality that most of us could not relate to... I was convinced that the medicine was poison and should not be taken.
But, after nearly 1 year on medication, my family member decided to begin slowly reducing the amount taken and has been off of it for several months. The depression is not a factor, her overall attitude is positive and her personality has been restored to the intelligent, loving, and interested person that we all used to know and love.

So here is my two cents... I don't know that medication is the answer, but in this case, it did certainly give the boost in the right direction.
I do believe that it is up to the individual to try to change their own way of thinking and reactions in order to complete the process of restoring joy and a positive outlook.
Find a friend to talk to... the best ones are the ones that don't have very much to say back.
-Just my two cents.

2:51 AM, April 08, 2008  
Blogger Doonesbury said...

I have been prescribed a number of antidepressants over the past 12 years. Some did nothing, others had unpleasant side effects, some made my situation worse; only one works well for me - and it works very well indeed. I honestly believe it has saved my life.

10:56 AM, April 08, 2008  
Anonymous Anonymous said...

enjoyed your blog on line ups...and the comics...actually your whole blog is pretty cool

11:47 PM, May 08, 2008  
Blogger tonmac said...

PLoS has comments related to the 2008 Kirsch article. You should let the authors speak for themselves.
Kirsch originally had no axe to grind - he simply wanted to further study the placebo effect (he is somewhat an expert on this subject).
He has 3 studies on antidepressants (1998; 2002 & 2008). Check out my reference list at http://tonmac-noaccountingfortaste.blogspot.com/
I have posted a number of times on the Kirsch studies.
tonmac

5:46 AM, November 07, 2008  
Blogger tonmac said...

Oh I forgot to add - Turner, E., Matthews, A., Linardatos, E., Tell. R. & Rosenthal, R. (2008). Selective Publication of Antidepressant Trials and its Influence on Apparent Efficacy. The New England Journal of Medicine. 35(8):3. 252-260 Accessed 03.11.08 from http://content.nejm.org/cgi/content/full/358/3/252
This very recent journal article, freely available (like PLoS)from the internet, is worth a look. It has nothing to do with methodology. It simply looks at the impact of selective publication on the eventual results.
tonmac

5:59 AM, November 07, 2008  
Blogger Nick Barrowman said...

Thanks, Tonmac, for pointing out the NEJM article and for your reference list.

1:08 PM, November 07, 2008  
Blogger Strawn said...

As an effective medicine, paxil would surely provide you immense relief if you are in the grip of panic disorder, depression or generalized anxiety disorder but it is altogether true that the usage of Paxil occasionally facilitates certain side-effects such as headache, low blood pressure, irregular heartbeat et al. These side-effects can yield serious results if not treated immediately hence take this medicine only on proper prescription.

1:43 AM, August 10, 2009  

Post a Comment

<< Home