January 13th, 2016

Who will be duped by error-strewn ‘meta-analysis’ of e-cigarette studies?


Done badly, meta-analysis can be a neat and scientific-sounding way of aggregating junk to create new and more convincing junk

[Note: backgrounder What is meta-analysis]

The Lancet Respiratory Medicine has been duped into publishing a ‘meta-analysis’ of e-cigarette studies authored by Professor Stanton Glantz and colleague [see Kalkhoran S, Glantz SA. E-cigarettes and smoking cessation in real-world and clinical settings: a systematic review and meta-analysis. Lancet Respir Med].

The expert reaction has been swift and devastating: “not scientific”, “grossly misleading”, “a major failure of the peer review system” are amongst the comments from Professor Robert West, Professor Ann McNeill, Professor Peter Hajek, Linda Bauld of Cancer Research UK and Roseanna O’Connor of Public Health England. What’s behind this unusually strong condemnation?

We knew this was coming because an earlier version of this thoroughly flawed exercise is available on Professor Glantz’s blog: Meta-analysis of all available population studies continues to show smokers who use e-cigs less likely to quit smoking.

I could go on at length about this exercise… but fortunately, I don’t have to.  The U.S. anti-smoking agency, Truth Initiative has already provided an extensive and credible critique of this meta-analysis and underlying studies in a submission to the FDA (please note it was known as Legacy at the time of the submission). Here is its acerbic dismissal of Professor Glantz’s meta-analysis (which is ref 73):

While the majority of the studies we reviewed are marred by poor measurement of exposures and unmeasured confounders, many of them have been included in a meta-analysis that claims to show that smokers who use e-cigarettes are less likely to quit smoking compared to those who do not.73 This meta- analysis simply lumps together the errors of inference from these correlations. As described in detail above, quantitatively synthesizing heterogeneous studies is scientifically inappropriate and the findings of such meta-analyses are therefore invalid. (emphasis added)

Dismissing this and much else, the experts at Truth concluded:

Findings from the studies with the strongest methodologies suggest that e-cigarettes are effective in helping adult smokers to quit or to reduce their cigarette consumption and that rates of smoking cessation with e-cigarettes are similar to rates of cessation with nicotine replacement therapy.

This is consistent with the Cochrane Review of e-cigarette studies, which

…finds emerging evidence that smokers who use electronic cigarettes can stop or reduce their smoking.

The Cochrane Review gave a cautiously positive assessment of e-cigarettes but stressed the weakness of the evidence base. In short, there are few RCTs because there is little incentive for any manufacturer to conduct RCTs – and the value of RCTs is limited in such an uncontrolled real-world behavioural eco-system as vaping. The solution to that is not to take on lots of studies that are completely unsuitable for addressing this question and then aggregate them, it is to draw on different strands of evidence to build up a picture.

We’ll hear more from Truth later in this post. As well as the insights from this preview of Professor Glantz’s analysis, we also know this work must be very poor indeed because it was initially pitched to at least one US journal, which normally publishes anything hostile to e-cigarettes, and it was even rejected by them.

Why might a journal reject this ‘meta-analysis’? Let’s go in deeper…

There are multiple challenges with interpreting the e-cigarette studies routinely appearing in the scientific literature – and over-interpretation is all too easy or even deliberate. To make sense of what each study is telling us, isn’t telling us, and can’t tell us, it needs to be interrogated with questions like these (and many others):

  • Motivation. Were the e-cigarette users actually trying to quit? Several studies include people who happen to be using e-cigarettes but who aren’t trying to quit smoking.  They simply can’t be included in an assessment of whether e-cigarettes help with quitting smoking or be compared with motivated people trying other approaches to quitting. But they are included in this meta-analysis.
  • The intensity of use. Were the users making just occasional use of e-cigarettes and were the users classified sensibly? Did they find that people using e-cigarettes just occasionally didn’t quit smoking (no surprise) and then attribute that as a failure of e-cigarettes to help the user stop smoking? Many of the studies classified as e-cigarette users anyone who had ever used an e-cigarette. So if you tried one at a party years before you were surveyed, it was counted as e-cigarette use and expected to result in cessation years later. How does that make sense? There has to be real, significant use to see an effect on quitting – just as is true with NRT, or any other medication or treatment, for that matter.  But few of the studies in the meta-analysis consider this.
  • Technology. Were they using modern devices?  Many studies take 2 years so are way out of date a publication, and many users progress to more advanced devices with a higher chance of quitting as they get used to vaping.
  • Confounding. Did some other factor explain why people using e-cigarettes appeared to quit less? For example, were the users of e-cigarettes more nicotine dependent?  Maybe users that have tried everything else or really want to keep using nicotine are drawn to e-cigarettes rather than complete quitting?
  • Selection bias. Were users who had already quit smoking with e-cigarettes excluded? Studies that recruit smokers at baseline can be very misleading: they include smokers who tried e-cigarettes but carry on smoking and exclude those that tried e-cigarettes and quit – i.e. the sample is pre-sorted to include failures and exclude successes, and is, therefore, unrepresentative. The problem rarely arises for other forms of help like NRT because people tend not to use them after they fail, but vaping appears to be different with extended periods of dual use being common.
  • Appeal and acceptability. Is a higher proportion of users willing to try e-cigarettes? E-cigarettes may be more effective at the population level if a higher proportion of smokers is willing to try them, even if the effectiveness per user is the same. We are dealing with a behaviour, not a medicine.
  • Outcome measure. What counts as success? Did an arbitrary cut-off point distort the findings?  Were vapers who were still smoking progressing towards quitting at the study end point? Supposing smokers using NRT set a quit date, become abstinent and then typically relapse as time passes. Supposing vapers approach quitting differently:  gradually weaning themselves off smoking and on to vaping through a period of dual use.  An early cut off will make NRT (quit and relapse) look better and make vaping (gradual transition) look worse.

Deep methodological flaws

These questions are designed to tease out problems of selection biasconfounding and study design including the outcome and exposure measure. From these questions, it should be evident that great care must be taken with interpreting results and making comparisons within and between e-cigarette studies.  Even greater care is required if these studies are then going to be aggregated in a ‘meta-analysis’.  Meta-analysis is legitimate when several small studies of almost identical design can be pooled to create the equivalent of a larger study with better statistical power and greater confidence in the pooled result. For example, when several trials of a drug are made on patients with the same illness, pooling the results is legitimate and useful. But if they are totally different studies in design – and the ones in this meta-analysis are – then they cannot legitimately be pooled.  To be aggregated, it should obvious the studies must be almost identical in design – if they are not, then they are summing oranges and bananas.

Carl V Phillips provides a high-quality discussion of the methodological issues afflicting this ‘meta-analysis’

An example of just how poor this is

Let’s take one example to highlight the kind of studies being used in this analysis: Al-Delaimy WK, Myers MG, Leas EC, et al. E-cigarette use in the past and quitting behavior in the future: a population-based study. Am J Public Health 2015;105:1213–9. [link] cited at ref 36 and included in the ‘meta-analysis’.

In this study, the authors divided a sample of smokers at baseline into those who had ever used e-cigarettes (even just one) and those who said they never would use e-cigarettes. Amazingly, this was somehow regarded as a reliable proxy for trying to quit with and without e-cigarettes. They then measured smoking behaviour 12 months later and drew conclusions about the impact of e-cigarettes on quitting behaviour. They didn’t check whether e-cigarettes actually had been used during the 12 months or whether the smokers were actually trying to quit. So we (and the authors) have no idea who was actually using e-cigarettes, and how much, if at all, or whether they were trying to quit, and if they were, whether they were using e-cigarettes in the attempt. Apart from that, it is perfect!!

But this has found its way into an analysis of studies that purportedly tell us something about whether e-cigarettes help people to quit. Then it has been aggregated with studies with completely different designs, with different but equally misleading inferences drawn from them.

More on understanding what’s gone wrong here

In a hilarious and desperate effort to counter the criticism in this blog and mirrored elsewhere, the UCSF press release states: People who carefully read our paper, particularly the sensitivity analysis (Table 2), will see that we anticipated and dealt with his [i.e. my] predictable criticisms. Really? These criticisms are nothing to do with ‘sensitivity analysis’ – they are to do with deep flaws related to selection bias, confounding and study design – and, above all, the  preposterous idea of aggregating studies that are completely different in design, exposure, user characterisation and outcome measure.

Rather than try to explain all this myself, I thought it would be useful and educational – not least for the editor and peer reviewers at The Lancet RM  – to provide a longer extract of Truth Initiative’s critique which elaborates on these arguments in more formally.  This is set out in the box below. Access to the original and referencing is here from page 8.

State of the Science: Impacts on Current Users of Tobacco Products (Cessation/Dual Use/Exclusive Use)

Here we turn to patterns of use among current users of tobacco, especially the most harmful combustible products, and in particular, the current state of knowledge on the impact of e-cigarettes on cessation and dual use behavior. In our view, the current state of the scientific literature is inadequate to draw significant conclusions about this issue.  This is because most of the current literature is based on cross-sectional observations that have significant flaws making them of limited or no value in assessing this question.

Sources of concern include:

  • Approach is blind: vulnerable to heterogeneous studies that lack sufficient commonality of measures, methods, rigor
  • Poor exposure measures: inappropriate/insufficient duration (e.g. ever used an e-cigarette; no duration indicated; no reason for use, type of product, or co-use with other quit aids indicated)
  • Selection, indication bias: cannot rule out25
  • Replication myth: multiple uninformative studies create a false negative impression and are over

Evidence to Assess the Effect of E-cigarette Use on Smoking Cessation

Below we detail our concerns regarding the state of the evidence of e-cigarettes on smoking cessation and expand the presentation of Dr. Jennifer Pearson at FDA’s third e-cigarette workshop.

We conducted a review of the scientific evidence related to the effect of e-cigarette use on smoking cessation. We identified 42 studies that purport to report on the relationship between e-cigarette use and smoking cessation. We present the strengths and weaknesses of these studies, and highlight the best available evidence. Take-home points for each section are noted in bold.

Most existing observational studies related to the effect of e-cigarette use on smoking cessation are uninformative and are marred by poor measurement of exposures and unmeasured confounders.

We established a hierarchy to organize studies according to their methodological strengths. Studies that “fall out” of this hierarchy towards the top are weaker – and uninformative – compared to those that “fall out” towards the bottom. Figure 7 illustrates this hierarchy. We found that most of the existing studies related to the effect of e-cigarette use on smoking cessation are uninformative. An assessment of the included studies is outlined in detail below. Information about specific studies can be found in Table 2 (included at the end of this section).  We include three randomized, controlled trials published on this topic in the flowchart for completeness, while noting that their design preempts some of the categories listed in Figure 7.

Figure 7. Hierarchy of Methodological Consideration and Evidence. Studies included towards the bottom of the flowchart have the strongest methodologies and provide the best evidence.

Screen Shot 2016-01-13 at 15.08.30

It is important to note that the existing observational studies do not constitute reliable scientific evidence and therefore, meta-analyses of these studies are inappropriate for several reasons. First, as noted above and detailed below, the majority of studies that purport to address the relationship between e-cigarette use and smoking cessation do not provide evidence to answer that question.

Second, as noted in the Cochrane Handbook for Systematic Reviews of Interventions,26 the use of meta-analysis is not appropriate in all reviews and the first step to deciding whether to undertake a meta-analysis is to evaluate the similarity of studies with respect to population, intervention, comparison group, and outcome. The initial determination of heterogeneity should be evaluated qualitatively by the authors and evidence of statistical heterogeneity in the model should be presented ONLY after it is determined that the studies are comparable enough to warrant pooling of data. The observational studies identified in our review were determined to be sufficiently heterogeneous that they should not be pooled in such an analysis. The recent Cochrane review on the use of e-cigarettes for smoking cessation reached a similar conclusion. The Cochrane reviewers examined data from both randomized controlled trials and cohort studies, but conducted limited meta-analyses using data from only the two randomized controlled trials where the designs and populations were deemed sufficiently similar to compare.27 Third, non-randomized studies which use “different study designs (or which have different design features)…should not be combined in a meta-analysis” (p. 422).26 A key issue related to pooling non-randomized studies is adjustment for confounding which is not captured in simple numerators and denominators or means and standard errors.  The Cochrane Handbook warns: “Meta-analyses of studies that are at risk of bias may be seriously misleading. If bias is present in each (or some) of the individual studies, meta-analysis will simply compound the errors, and produce a ‘wrong’ result that may be interpreted as having more credibility” (p. 247)26 As detailed below, the existing observational studies suffer from measurement bias, selection bias, and confounding that render meta-analysis inappropriate at this time.

Hierarchy of Methodological Considerations and Evidence (top to bottom):

(1)   Studies must examine the outcome of interest (i.e., cigarette smoking abstinence or reduction).

To provide information regarding whether e-cigarettes can be used as an effective tool for smoking cessation, a study must appropriately operationalize the outcome of interest (smoking cessation). We considered smoking abstinence and cigarette reduction to be the most relevant/highest-quality outcomes. Of the 42 studies we reviewed, seven did not examine the outcome of interest.28-34 These studies assessed outcomes that are related to smoking cessation, such as intention to smoke,28 withdrawal-related symptoms,28 nicotine levels,31 reason for e-cigarette use,30 and other descriptive information about people who use e-cigarettes for cessation.33,34

(2)   Studies must assess e-cigarette use for smoking cessation as the exposure of interest.

For observational studies, it is crucial to confirm that participants are using e-cigarettes for the purpose of cessation. People use e-cigarettes for a multitude of reasons, e.g., because they are cheaper than cigarettes, they are less harmful, or they are only experimenting with the product. If smokers are not using e-cigarettes to help them quit it does not make sense that we should expect them to help smokers quit. Pearson et al. addressed this issue by asking, “What quit methods have you used in the past 3 months?”35 Participants who used an e-cigarette as a quit method were classified as “exposed” and those who did not were classified as “unexposed” regardless of other e- cigarette use.

For randomized controlled trials (RCTs), where participants are assigned by the researchers to either use e-cigarettes or abstain and followed for the cessation outcomes of interest, reasons for use are balanced across the exposed and unexposed study groups through randomization. As such, reason for use is unlikely to confound the relationship between e-cigarette exposure and cessation and, similar to studies of NRT on cessation, does not need to be directly assessed in order for study findings to be generalizable.

Of the remaining 35 studies in the hierarchy, three were RCTs.36-38 Of the other 32 studies in the hierarchy, 26 did not assess the reason for e-cigarette use as an exposure.39-64

(3)   Studies must employ strong study designs.

Apart from RCTs, the strongest epidemiologic studies examining whether e-cigarette use leads to smoking abstinence or cigarette reduction should be longitudinal and have appropriate comparison groups. Of the nine remaining studies in the hierarchy, seven did not have appropriate study designs.35,37,65-69 Four of these studies were cross-sectional65,66,68,69 and one was a case series.66 One study was longitudinal;35 however, in its assessment of the association between e-cigarette use and smoking cessation outcomes, it only employed cross-sectional data. One study randomly assigned participants to e-cigarette use or control during the initial lab phase of the study, but then provided the control group with e-cigarettes during the follow-up period.37 Since there was no longer an unexposed control group during the phase of the study in which the smoking cessation outcomes were obtained, this study was excluded at this point as well.

After considering whether studies assessed the outcome of interest, e-cigarette use for smoking cessation as an exposure, and study design, only two studies – both RCTs – remain in the hierarchy.36,38  These studies met all of the additional criteria described below.

(4)   Studies must precisely measure the exposure of interest (i.e., e-cigarette use).

In order to precisely measure the exposure of interest (e-cigarette use), studies should:

(a) Establish temporality by ensuring that the exposure preceded the outcome

Some studies report ever or past 30-day use of e-cigarettes at baseline.44,46,49,54,62 As measures, neither ever use nor past 30-day use accurately capture individuals who regularly use e-cigarettes. Amato et al. investigated this measurement issue in a 2014 survey of over 9,300 participants and found that, among current smokers who reported past 30 day e- cigarette use, 59% were infrequent e-cigarette users (use on 5 or fewer of the past 30 days), 28.7% were intermediate users (use between 6 and 29 (inclusive) of the past 30 days), and 12.3% were daily users (use 30 out of the past 30 days).30 Based on this finding, the authors concluded that defining adult prevalence as any e-cigarette use in the past 30 days may include individuals who are experimenting with e-cigarette and are unlikely to progress to regular use.

(b) Measure dose of e-cigarette use

One study, conducted by Biener and Hargraves, illustrates the importance of measuring e- cigarette dose for studies investigating smoking cessation outcomes.45 They measured e- cigarette dose and found that intensive e-cigarette use (daily use for at least one month) was associated with a 6.07 (95% CI: 1.11 – 33.18) increase in the odds of smoking abstinence, while there was no significant difference in cessation for intermittent users (e- cigarette use more than 1-2 times but not daily for a month) or for individuals who had never used e-cigarette or who had used e-cigarette only once or twice.45 Studies using weaker measures of ever or past 30-day e-cigarette use at baseline do not provide sufficient information on exposure to e-cigarettes to understand the relationship between e-cigarette use and cessation.

(c) Assess the e-cigarette product type

E-cigarettes are diverse .70 They vary by device type,71 performance,71 flavor52 and other characteristics.70 It is important to assess the e-cigarette product type because products may have different levels of effectiveness when used for smoking cessation. This phenomenon was seen by Hitchman et al., who examined differences between cigalike and tank e-cigarettes and found differential results based on product type.55

Many of the observational studies that “fall out” of the hierarchy before reaching the bottom report negative correlational findings between e-cigarettes and smoking cessation. 29,35,39,41,43,50,52,53,55,56,61,63-65,67-6944,46,54,62,72 These studies are uninformative and do not tell us how e-cigarette use affects cessation due to their inherent methodological limitations. To make inferences about the impact of e-cigarette use on cessation based on these studies would be akin to repeating well-documented errors regarding the negative effect of nicotine replacement therapy on smoking cessation that is the result of confounding with dependence. That is, more dependent individuals are more likely to try multiple cessation treatments, and they are also more likely to fail because of their higher dependence.25 Similarly, e- cigarette users may be more likely to try any number of cessation treatments and to fail because of the influence of a third variable, such as dependence. This does not imply that the treatments are ineffective overall, but rather that other, extra-treatment factors need to be taken into account when highly dependent smokers try to quit and that perhaps treatment dose was insufficient to address their needs. Like the observational studies of use of nicotine replacement therapies (NRT), selection and indication bias are important considerations and very difficult to eliminate from analysis of the negative correlation between NRT use and cessation outcomes.25 Research indicates smokers who used e- cigarettes consume more cigarettes per day, are more nicotine dependent, make more quit attempts, make longer quit attempts, and use more cessation aids than otherwise similar smokers who have not used e-cigarettes to quit. Most of these characteristics are associated with poor cessation outcomes in observational studies of NRT.25,26 So, it’s not surprising that smokers who use e-cigarettes to quit have worse outcomes due to selection and indication bias in observational studies.

While the majority of the studies we reviewed are marred by poor measurement of exposures and unmeasured confounders, many of them have been included in a meta-analysis that claims to show that smokers who use e-cigarettes are less likely to quit smoking compared to those who do not.73 This meta- analysis simply lumps together the errors of inference from these correlations. As described in detail above, quantitatively synthesizing heterogeneous studies is scientifically inappropriate and the findings of such meta-analyses are therefore invalid.

A Cochrane systematic review published in 2014, which used the highest methodological standards, examined the efficacy of e-cigarettes for smoking cessation and reduction.74  Of the 594 records screened, only two randomized trials were included in the meta-analysis.36,38 We also identified these two studies as the only ones that met all of the methodological specifications described in our hierarchy.

Findings from the studies with the strongest methodologies suggest that e-cigarettes are effective in helping adult smokers to quit or to reduce their cigarette consumption and that rates of smoking cessation with e-cigarettes are similar to rates of cessation with nicotine replacement therapy.

One of many weaknesses of the peer review system is that allows authors to keep presenting a paper to journals until someone says ‘yes’: just keep going until you find editors and reviewers with either too little understanding or too much ideological bias that they will nod it through. And there it is: a peer-reviewed paper ready to be cited by its own authors every other unscrupulous academic, official and activist trying to prove that something much safer is somehow more dangerous.

Who was duped?

Or is ‘duped’ just too generous?


37 comments to Who will be duped by error-strewn ‘meta-analysis’ of e-cigarette studies?

  • Nice analysis. You and Abrams/Pearson both — Legacy is very schizophrenic in its relationship with science.

    The critique nicely emphasizes the overwhelming fatal flaw, that it is utter nonsense to combine results from studies that did not even look at the same exposure and outcome. I mean, seriously? This is not just a matter of finding a gullible editor. It requires finding one with no semblance of scientific understanding in the field. The weakness with the journal review system is far more profound than you suggest (as you, of course, know).

    After this everything else is a minor detail. Of course, the same critique applies to the Cochrane meta-analysis of the RCTs, which even after narrowed to two were sufficiently different that combining them as if they were the same in nonsense. So there is a bit of a political game going here.

    The points about the (ultimately unimportant) details are a bit uneven. Obviously the study needs to have a measure of the outcome of interest to draw any conclusions. The temporality issue is a red herring — so few adults use e-cigarettes and then smoke, or quit smoking and then take up e-cigarettes, that it creates trivial noise compared to other problems. The type of data collection (cross-sectional, etc.) is another red herring — any design can collect the right data or not.

    Studies of whether vapers who are not intending to quit smoking actually do so are informative, and should not be dismissed. They should just not be mistaken for measuring the effect of concerted switch attempts. The two obviously should not be combined as if they were measuring the same thing, but we have already established that as the clear fundamental problem with the whole exercise. Similarly, the fact that different “doses” have different effects is information, not a reason for dismissal. (On these points, learning what you can from a study but not pretending to learn something you cannot, see the post I wrote simultaneous with this one: http://antithrlies.com/2016/01/13/whatever-happened-to-be-measured-is-not-the-same-as-exposureoutcome-of-interest/ )

    The fatal flaw with interpreting many of these studies, which is alluded to but not clearly identified, is selection bias. Several of them effectively compared quitting rates in vapers who had previously tried to switch and failed with people who had never tried to switch. This is noted in the critique. What is missed is the fact that this does not require ad hoc characterization of what was wrong with the studies. Selection bias is epidemiology 101 stuff. The critiques should be presented in those terms to make the problems more clear, rather than trying to rederive principles of basic epidemiology ad hoc.

    • John walker

      The hypothesis “that smokers who use e-cigarettes are less likely to quit smoking compared to those who do not.” Is so, wierd, so contrary to lived experience, that it makes me wonder whether the people who accepted the paper actually read it.

      Our Simon chapman in some articles published last year , almost simultaneously , claimed that ‘nobody is using ecigs’ , he had been in France and not seen anybody using them. And that ecigs were dangerous and my long after that also claimed that they were also a placebo…

      • Sue

        Just for the record, I live in France, equi-distant from 3 smallish towns, and I see vapers every time I’m out in any of them.

        • john walker

          at the time of Chapmans ‘observations’, we were holidaying in a flat right in the center of the CBD of Sydney. Over those ten days, just causally walking about, I saw about three to five other vapers a day.

          BTW the most bizarre study of all surely has to be the one that claims to have found that vaping a few mls of e liquid over a day ,that contains , in total, perhaps 1 ml of alcohol, could result in alcohol related impairment of driving ability.
          Maybe there is something in Homeopathy after all ?

        • Gummy Bear

          @John – Actually Chapman makes an interesting observation. According to the prevalence figures we should see one vaper for every ten smokers, in Australia perhaps one in twenty. But public visibility is far from that; he spotted that accurately. Chapman tried to use that observation for a nasty jibe against vapers, but in fact he scored a massive own goal.

        • Jude

          I think the reason Chapman didn’t see many vapers when he was holidaying with his wife in Europe, was that as elderly people, he and his wife probably didn’t visit places where younger people are seen, plus their eyesight may not be so good, considering their age, and Chapman is probably to vain to wear glasses.

          I have relatives in France and Germany, and they tell me that there are vapers everywhere, but that most vapers are working during the weekdays, so not so visible on the streets or the tourist areas.

      • Gummy Bear

        Has anyone noticed that Chapman’s vaper-spotting exercise (the closest he’ll ever get to a real scientific study) completely trashes the re-normalisation ‘hypothesis’? For anything to normalise anything else it would have to be visible, wouldn’t it?

        • Kevin McElligott

          I will give you 10 to 1 odds that Simon’s article on vaping observations while on holiday was a finance/tax driven exercise. Publishing an article allows him to writ off his travel expenses as a research or business expense. A cynical and stupid exercise published by some friendly dupe

        • John walker

          With the possible exception of ‘cloud chasers’, vaping is a lot less visible than smoking- Especially out of doors, it hardly smells and the vapor does not linger like smoke. And the user does not smell afterwards.
          I.e it is intrinsicly less noticble, for example a year ago I was waiting for a plane in a Asian airports bar, it took me sometime to notice the bloke on the next table was discreetly using a e cig…

        • Gummy Bear

          John – this is true though technology looks after that quickly. A decent off-the-shelf subohm kit already puts any Marlboro to shame, and many people now use those as starter kits. The other reason is that people vape where you don’t see them – in their offices, in their living rooms, in pubs…
          Simon ‘s holiday experiment suggests that vapers are an order of magnitude less visible than smokers, after correcting for prevalence. Seems about right.

        • Gummy Bear

          Kevin – maybe, maybe not; I don’t want to speculate. In any case he could have written any odd nonsense. But he writes a piece that smashes the re-normalisation myth. He has done us a great favour.
          Note that there are two myths the anti-vaping crowd cannot do without, no matter how much they are refuted by the evidence. One is the gateway, the other is that vaping somehow re-normalises smoking. Without these you can never make up a narrative that the net public health effect of ecigs is negative.
          Re-normalisation is almost unfalsifiable. It is not a scientific hypothesis but just an opinion. If you believe that bubble tea should be banned because it re-normalises whisky you are probably talking nonsense, but I can’t really prove you wrong.
          Except for one thing. No matter what you mean by normalisation, the necessary condition is that the thing that normalises something is widely visible. Bubble tea couldn’t re-normalise whisky if people drank it at home and nobody saw it.
          So whenever someone expresses the ‘concern’ that vaping ‘may re-normalise smoking’ a link to Simon’s piece should reassure them that no, it doesn’t.

  • Kent

    Reading the Legacy Submission, it appears they use the Nutt et al MCDA estimate in Figure 1…. shills

  • Chris Price

    A separate but relevant issue for smoking cessation study conclusions is revealed at #2 in T.I.’s pink breakout box above titled :
    (2) Studies must assess e-cigarette use for smoking cessation as the exposure of interest.

    The text clearly reveals the ongoing confusion about the meaning of ‘quit’, which means different things to different people almost according to the phase of the moon.

    There is a fundamental difference in user beliefs and outcomes centred around the term ‘quit’: people don’t know exactly what they are being asked when the term ‘quit is used – but they don’t know that they don’t know – and the scientists posing the question equally have no clue how this will be interpreted; and crucially make no attempt to differentiate or identify the separate outcomes.

    For the survey posers, ‘quit’ can mean ceasing cigarettes and vaping totally and that this meaning is the outcome always sought and always what the question addresses, in their minds (even though the subjects will probably not interpret it this way); or it can mean ceasing cigarettes but continuing to vape, or it can mean a combination of the two lumping together all outcomes, or it can mean ceasing cigarettes and an intention to cease vaping; or anything at all. They appear not to know the difference or to ask the subjects *exactly* what they want to ask, or to how to interpret an answer that could basically mean anything. As if such an answer has any use anyway.

    Because of this massive problem with the question logic, seen almost everywhere, I don’t think you can derive any useful information at all from many if not most of these studies; and a meta-analysis of them is just going to be a pile of junk even with the best intentions.

  • Hans Schmidt

    Nicotine dependency like in NRT dependency?

  • Jude

    What I find most annoying, and in some cases criminal, is not that there are mistakes, bias, toxic ideology from ANTZ, and pointless, or deliberately confusing, or outright false studies on vaping, or people who take up vaping, for whatever reason, its that these crappy studies, or blatant lies and misinformation, are used to punish people who simply want to either cut down their tobacco smoking, or switch completely to a safer alternative.

    For decades now, smokers have been the whipping boys of public health, they have been stigmatised, punished with huge taxes, discriminated against, been the butt of a campaign of hate from those in politics and the tobacco control industry, as well as the anti-smoker zealots in public health and the general community. When smokers have found a genuinely useful way of mitigating the risks involved in tobacco smoking, by switching to either Smokeless tobacco products like Snus, or to non-tobacco products like vapour products, suddenly they find that they are coming under the same fire as they did as smokers. Why? because they are a threat to the ideology of the ANTZ, they are a threat to the funding of the ANTZ, and they are a threat to the profits of governments, pharmaceutical corporations, and tobacco companies.

    Glantz will get his garbage published, and I have no doubt that his garbage “study” will be used by those who want to kill off vaping, and keep people smoking, and have the power to make policies that will achieve this outcome. We have seen this play out so often over the last few years. I can’t help but be cynical, you only need to look at what has happened in the last few weeks, the sheer amount of garbage coming from public health and the liars in tobacco control, to know what their game is.

    It is about Money and Power, and has never been about health.

    • John

      “Fanaticism consists of redoubling your efforts when you have forgotten your aim.” Think Stan has long since forgotten,whatever was his original aim.
      Think that it is simply about Stans need to be, important and unquestioned, as anything else.

      • Jude

        Yep, and if he gets paid to do so all the better for him. Reminds me a lot of Chapman, who leaps onto any controversy just to get his opinion heard and seen. From such issues as wind turbines, to fire works, (this opinion he tweeted complete with gory pictures of injured people), it a narcissistic trait that Glantz shares with Chapman, among others in the tobacco control industry,(which seems to attract these types).

  • Jukka Kelovuori

    I tried to post the link to this article in the comment section of Simon Chapmans article in ‘The Conversation’: https://theconversation.com/how-shaky-are-the-twin-pillars-of-the-case-for-e-cigarettes-53153#_=_ I was being polite and only told that my link is to an article about the systematic review. My comment was removed by the moderator in seconds. And I’m not the only one who had their comment removed bacause of the link to this excellent article. What is Chapman afraid of?

  • Looking at the actual paper, out of 38 studies reviewed, only THREE indicated using participants who were actually using e-cigarettes during the study. For all of the other studies used to make the calculations, the participants were defined as anyone who had used an ecigarette at least once in the past 30 days to 3 months and two were even “ever use.” So, if a smoker tried a puff off of a friend’s e-cigarette 2 months ago (or ever) and didn’t quit smoking, she would end up in the “used e-cigarettes but didn’t quit” group.

    Only ONE study in the 38 reviewed had participants clearly defined as daily vapers.

  • […] engineering. (For those who do not get the jokes, read on — I explain in the analysis. Clive Bates already explained much of this, but I am distilling it down the most essential problems and trying to explain them so the reasons […]

  • […] Clive Bates is also critical towards the actual and a former version of Kalkhoran and Glantz’s manuscript, asserting that the scientific journal (in fact its editors) “has been duped into publishing a ‘meta-analysis’ of e-cigarette studies. […]

  • So what will happen now, will it be retracted from peer review? Or rectified? I do not know how this works. But something like that should happen right?

  • john walker

    Clive something a bit more encouraging, Dr Attila Danko is a real MD, from Country Victoria AU, he a strong supporter of harm minimization( Ecigs helped him quit). He has formed a org to lobby for sensible regulation. This is a link to his testimony to a South Australian parliament inquiry into this area:


  • […] little bit of such bias? Similarly, supposedly bright-line rules require subjective interpretation: Typical lists of rules for meta-analysis say that we should not include any study that does not have a comparison […]

  • […] you read in the papers. For an unbiased look at e-cigarette coverage, we recommend Clive Bate’s the Counterfactual and Guy Bentley from the Daily […]

  • […] It is good that vapers don’t look for health information because most of what they would find is highly misleading and alarmist, designed to deter them from trying this much-safer alternative to smoking.  The official and academic sources of information about health and risks are heavily contaminated with poor quality science, misinterpretation and hyperbole – of the type manufactured in San Diego and elsewhere in California.  […]

  • […] in his meta-analysis. In fact, Stanton “not a real doctor” Glantz’ meta-analysis had been rejected by at least one other medical journal before he finally managed to get […]

  • […] It was heavily criticized by experts, and many great blog posts were written on it – by Clive Bates and Carl V. Phillips, for example – but the CliffsNotes version is that the analysis is highly […]

  • […] as lying. Plus the meta-analysis that you so clearly favour has been thoroughly ripped to shreds here and here. Hardly a shining example is […]

  • […] off smoking after unsuccessful trials with other methods, the meta-analysis in question is the controverted publication by Kalkhoran and Glantz; a manuscript whose conclusions are still promoted by its […]

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>