A recent blog post on Chronic Fatigue Syndrome linked in passing to a ‘treatment’ called Mitochondria Ignite™ with NT Factor®. This product caught my attention as an example of what Richard Feynman called ‘Cargo Cult Science’; a company dressing up like scientists, using chemical names and precise sounding figures, without actually having any science underlying it.
However, the product is not arguably not exactly pure Cargo Cult Science; there is a small amount of science content present. The product page contains a number of references, some of which point to peer review journals, and some of which are actually studies of the effect of some of the contents of the drug on humans. Of cours,e taking apart the studies shows that the product is still unproven, despite the thin glaze of real science; I can’t help but feel that this sort of thing has slightly grim implications for the future of accurate consumer information.
Here Comes The ‘Science’ Bit!
The first reference claims a decrease in fatigue when patients were treated with a different NT Factor%reg;-containing pill (Propax™), but refers to a small pilot study (page 23 of this issue) of self-reported symptoms of fatigue in group of patients without a control group.
The second reference claims a decrease in chemo side effects in a double-blind controlled study using the same drug as the previous paper. This one (p17 of this issue) consisted of blinded and unblinded phases; we’ll ignore the unblinded phased for obvious reasons. The blinded phase consisted of two groups, the first given a placebo for 6 weeks, followed by the drug for 6 weeks, and vice versa for the second. However, the authors at no point actually compare the difference between drug and placebo in either the first or the second 6 week period. The closest they come is to look only at the placebo-to-drug group, and compare the side effects during the first and last 6 week periods; this is, of course, entirely meaningless, without comparing to the drug-to-placebo group.
The third reference is a small study (20 subjects) that looked for differences in fatigue symptoms and mitochondrial function after treatment; the study had no control group. The forth reference points to an anti-aging newsletter that doesn’t seem to refer to the drug at all, and the fifth reference is a mouse study into a type of compound present in NT Factor.
So, 4 actual studies; 1 in mouse, 2 without control, 1 oddly flawed. Many of them pilot studies for future, proper clinical trials; as these papers are all nearly a decade old, I must assume that proper trials were either never carried out, or were carried out and found no significant effects.
Not Good News
As much as I hate the flashy ’science’ sections on shampoo adverts, in which ‘proteins’ ‘restore’ your hair’s ‘chemical balance’ and so on, at least these are easy to tell apart from actual scientific evidence; those who have any critical faculties realise that these are a con, and the problem faced by science educators is the (still relatively daunting task) of getting lay people to think critically about science.
The worrying thing about the example of the NT Factor® stuff is that it looks to the untrained eye just like science. When I saw it, I didn’t know whether it was kosher or not; it was only by tracking down the references and reading the papers that I actually found out that the treatment was unproven and probably ineffective. It is one thing to ask people to think critically about products; it is another to ask them to track down, read and appraise 5 references before you decide on the product’s efficacy. By jumbling together a bunch of small pilot studies, published in low-tier journals, you can assert the efficacy of your dollar-per-day pills, without requiring your treatments to be demonstrably effective.
This sort of Cargo-Cult-ish ‘evidence dumping’ is hardly new, and isn’t restricted to a particular company or industry (e.g. see the Simon Singh/British Chiropractic Association thing). If and when people start to become more informed and critical about science, I don’t doubt we will see a lot more of it.
I have been thinking about potential solutions to this problem, and I have a few ideas (require all products making claims of medical efficacy to demonstrate it scientifically before an MHRA/FDA-style official body, increase funding and awareness about meta-analysis groups such as the Cochrane Collaboration, more online collaboration to scrutinise products, etc). However, I really have very little idea what to do about it, and would be very grateful for any indication of other people who have given it some thought.
Tags: disease, slighty ranty
[...] This post was mentioned on Twitter by Luke Jostins, sabrina. sabrina said: Cargo Cult Science and NT Factor® « Genetic Inference: A recent blog post on Chronic Fatigue Syndrome linked in pa… http://bit.ly/6WdGzh [...]
Just wanted to say thanks for your post - I read the rave reviews and “scientific” studies about the efficacy of NT Factor, and was almost (almost) ready to fork over $50 to try the stuff, but thought I’d investigate a bit more. (I have relentless CFS/ME and have spent countless $$ (and ££ as I’m now in the UK) on vitamins and supplements that are at best useless and, more often, make me worse). I don’t know enough from a scientific standpoint to evaluate these “studies,” so I appreciate your dissection of them!
Look, fingers crossed that they do a big clinical trial and find out this stuff actually has some merit, but until then, I’ll save my $50.
thanks again.