Done badly, meta-analysis can be a neat and scientific-sounding way of aggregating junk to create new and more convincing junk
[Note: backgrounder What is meta-analysis]
The Lancet Respiratory Medicine has been duped into publishing a ‘meta-analysis’ of e-cigarette studies authored by Professor Stanton Glantz and colleague [see Kalkhoran S, Glantz SA. E-cigarettes and smoking cessation in real-world and clinical settings: a systematic review and meta-analysis. Lancet Respir Med].
The expert reaction has been swift and devastating: “not scientific”, “grossly misleading”, “a major failure of the peer review system” are amongst the comments from Professor Robert West, Professor Ann McNeill, Professor Peter Hajek, Linda Bauld of Cancer Research UK and Roseanna O’Connor of Public Health England.
We knew this was coming because an earlier version of this thoroughly flawed exercise is available on Professor Glantz’s blog: Meta-analysis of all available population studies continues to show smokers who use e-cigs less likely to quit smoking.
I could go on at length about this exercise… but fortunately, I don’t have to. The U.S. anti-smoking agency, Truth Initiative has already provided an extensive and credible critique of this meta-analysis and underlying studies in a submission to the FDA (please note it was known as Legacy at the time of the submission). Here is its acerbic dismissal of Professor Glantz’s meta-analysis (which is ref 73):
While the majority of the studies we reviewed are marred by poor measurement of exposures and unmeasured confounders, many of them have been included in a meta-analysis that claims to show that smokers who use e-cigarettes are less likely to quit smoking compared to those who do not.73 This meta- analysis simply lumps together the errors of inference from these correlations. As described in detail above, quantitatively synthesizing heterogeneous studies is scientifically inappropriate and the findings of such meta-analyses are therefore invalid. (emphasis added)
Dismissing this and much else, the experts at Truth concluded:
Findings from the studies with the strongest methodologies suggest that e-cigarettes are effective in helping adult smokers to quit or to reduce their cigarette consumption and that rates of smoking cessation with e-cigarettes are similar to rates of cessation with nicotine replacement therapy.
This is consistent with the Cochrane Review of e-cigarette studies, which
…finds emerging evidence that smokers who use electronic cigarettes can stop or reduce their smoking.
The Cochrane Review gave a cautiously positive assessment of e-cigarettes but stressed the weakness of the evidence base. In short, there are few RCTs because there is little incentive for any manufacturer to conduct RCTs – and the value of RCTs is limited in such an uncontrolled real-world behavioural eco-system as vaping. The solution to that is not to take on lots of studies that are completely unsuitable for addressing this question and then aggregate them, it is to draw on different strands of evidence to build up a picture.
We’ll hear more from Truth later in this post. As well as the insights from this preview of Professor Glantz’s analysis, we also know this work must be very poor indeed because it was initially pitched to at least one US journal, which normally publishes anything hostile to e-cigarettes, and it was even rejected by them.
Why might a journal reject this ‘meta-analysis’? Let’s go in deeper… >> read the full post