Some pointers on critically assessing the validity of claims

Wide-scale access to the Internet has resulted in unprecedented access to information for the average citizen of any developed nation, and the more recent proliferation of mobile data devices and networks have exponentially increased our ability to reference the collective body of knowledge on a whim.

This access comes at a price however, as tech-savvy marketeers have outstripped science and education practitioners’ resources, funding and drive to make information easily accessible, effectively saturating the search-engine and news-reporting info-spheres with commerce-driven interpretations of research, opinion, tradition and in many cases, out-right pseudo-science or fraudulent claims.

Whilst most developed economies provide some levels of consumer protection, in Australia including bodies such as the Therapeutic Goods Administration, the Australian Competition and Consumer Commission (ACCC), Australian Securities and Investments Commission (ASIC) various industry ombudsmen and voluntary ‘societies’ with industry codes-of-practice, most consumers have very limited understanding of the significant differences in evidentiary support pharmaceuticals require, for example, as compared with “complimentary” or “alternative” medicines.

Whilst complimentary or alternative medicines (CAM), are subject to rules about causing harm, they are not subject to any rules of proven efficacy, and simply stating that a product “may help with…” any given condition, rather than outright claiming to treat or cure said condition is enough to side-step TGA rules of evidence and testing.

So to Australian Standards rules governing safety of any product do not carry any implicit ratification of said product’s claims of usefulness or fitness-for-purpose, instead simply confirming that where said product is used in the manner for which it was intended, it is reasonable to expect that it will not cause harm due to materials failure or the like.

Unfortunately for consumers, over-reliance upon standards organisations has resulted in complacency and a certain gullibility with regard to considering whether to purchase or endorse (as sports-people and celebrities are so often called upon to do), a product, as we tend to assume someone has surely undertaken the appropriate tests and due diligence, and therefore the manufacturer or seller’s claims must be correct.

What then, can consumers do, as a first line of defence, to protect themselves against shelling out hard-eared cash for products and services which either don’t do what they claim, or indeed may cause harm directly or indirectly through their use in place of evidence-based alternatives?

Thankfully much of the work has been done for you, and you only need take the time to look. Most ineffective CAM products or treatments have been researched, analysed and “outed” by scientists or skeptics, and you need only search for “[name of product or treatment] scientific criticism (or efficacy or skeptic)”. Such a search will invariably result in some skeptical or scientific review of the product or service, usually with plenty of links or references to evidence and further reading on which the analysis was based.

So here are some rules-of-thumb and suggestions to support your critical thinking and review of any given product or service:

• There are no cure-alls.

Our physiology is complex, made up of many systems. If there were some product capable of curing all manner of conditions, it would be adopted by mainstream medicine after being subjected to the rigorous testing, study and review science by design entails. If the product claims to treat or help with treatment of multiple conditions be skeptical and do some research!

• Testimonials, anecdotes and endorsements are a sales and marketing technique, not proof of scientific evidence.

Science requires precisely designed, controlled, repeatable and peer-reviewed experiments to build up a body of knowledge. If a product had resulted from such scientific research, the seller would provide that as evidence, and not rely on testimonials alone.

• Studies aren’t always what they seem.

Anyone or any organisation can commission a study. Real science is an evolutionary, incremental improvement of our body-of-knowledge. It is very rare for one study to be a “breakthrough”, and even if it were, there would be multiple, peer reviewed follow-up studies before any products with proven efficacy made it to market. Studies commissioned by the manufacturer or seller should always be viewed with a skeptical eye. When in doubt, use the search I listed above to find out what experts are saying.

• If it sounds to good to be true, it probably is.

It’s and old saying, and it’s spot on. Some things are a true breakthrough – it does happen sometimes. But it’s the exception rather than the rule, so once again, be skeptical about miracle claims and do the research!

• Look out for ‘payment plans.’

If it’s available on a payment plan, it’s quite possible the seller makes their money from the finance because if the product really were a miracle, they’d have licensed it and made millions. This one came from Brian Dunning from Skeptoid.com and it couldn’t be more true. Once again, the claims may be true, but be skeptical and do the research – it really is a search away!

• Scientists outside their field aren’t experts.

This one is one of my favourites. Just because someone has a science background, doesn’t mean they can’t be fooled when outside their field of expertise. Do the search to make sure actual experts in the field agree with the poster-scientist’s claims.

• News outlets are first and foremost commercial concerns, driven by advertising revenue.

Don’t assume just because it appeared in a newspaper or online news site, that the writer or publisher actually confirmed the claims. Staff writers often reprint organisation press releases verbatim, with little or no independent confirmation of the facts. News outlets will often follow formulae for stories, typically extrapolating widely from very early research, drawing conclusions which aren’t yet supported by evidence. Finally, look out for ‘advertorials’, which are paid advertising dressed up as journalistic pieces. These often follow the formula: “New study reveals astonishing link between [this bad thing] and [this product which can help]”, followed by several paragraphs discussing the alleged study, finishing with the company name who provide the product which can help, who invariably commissioned or conducted the so-called study in the first place.

Despite the double-edged nature of the Internet’s power, where purveyors of woo and pseudo-science take advantage of this cheap mass-media and the disproportionate value human-beings place upon anecdotes, being informed really is easier than it has ever been before if you’re prepared to look a little deeper. An understanding of what constitutes ‘scientific evidence’, a skeptical eye and a web browser is truly all you need.

Related Articles

• What it Means to be a Skeptic (randi.org)
• Open Letter to Dr. Josephine Briggs (sciencebasedmedicine.org)
• Oprah’s buddy Dr. Christiane Northrup and breast thermography: The opportunistic promotion of quackery (sciencebasedmedicine.org)
• Woo: The future of American medicine? (scienceblogs.com)
• Evidence-Based Medicine, Human Studies Ethics, and the ‘Gonzalez Regimen’: a Disappointing Editorial in the Journal of Clinical Oncology Part 1 (sciencebasedmedicine.org)