Some pointers on critically assessing the validity of claims

Power Balance
Image by PoweredByLarios™ via Flickr

Wide-scale access to the Internet has resulted in unprecedented access to information for the average citizen of any developed nation, and the more recent proliferation of mobile data devices and networks have exponentially increased our ability to reference the collective body of knowledge on a whim.

This access comes at a price however, as tech-savvy marketeers have outstripped science and education practitioners’ resources, funding and drive to make information easily accessible, effectively saturating the search-engine and news-reporting info-spheres with commerce-driven interpretations of research, opinion, tradition and in many cases, out-right pseudo-science or fraudulent claims.

Whilst most developed economies provide some levels of consumer protection, in Australia including bodies such as the Therapeutic Goods Administration, the Australian Competition and Consumer Commission (ACCC), Australian Securities and Investments Commission (ASIC) various industry ombudsmen and voluntary ‘societies’ with industry codes-of-practice, most consumers have very limited understanding of the significant differences in evidentiary support pharmaceuticals require, for example, as compared with “complimentary” or “alternative” medicines.

Whilst complimentary or alternative medicines (CAM), are subject to rules about causing harm, they are not subject to any rules of proven efficacy, and simply stating that a product “may help with…” any given condition, rather than outright claiming to treat or cure said condition is enough to side-step TGA rules of evidence and testing.

So to Australian Standards rules governing safety of any product do not carry any implicit ratification of said product’s claims of usefulness or fitness-for-purpose, instead simply confirming that where said product is used in the manner for which it was intended, it is reasonable to expect that it will not cause harm due to materials failure or the like.

Unfortunately for consumers, over-reliance upon standards organisations has resulted in complacency and a certain gullibility with regard to considering whether to purchase or endorse (as sports-people and celebrities are so often called upon to do), a product, as we tend to assume someone has surely undertaken the appropriate tests and due diligence, and therefore the manufacturer or seller’s claims must be correct.

What then, can consumers do, as a first line of defence, to protect themselves against shelling out hard-eared cash for products and services which either don’t do what they claim, or indeed may cause harm directly or indirectly through their use in place of evidence-based alternatives?

Thankfully much of the work has been done for you, and you only need take the time to look. Most ineffective CAM products or treatments have been researched, analysed and “outed” by scientists or skeptics, and you need only search for “[name of product or treatment] scientific criticism (or efficacy or skeptic)”. Such a search will invariably result in some skeptical or scientific review of the product or service, usually with plenty of links or references to evidence and further reading on which the analysis was based.

So here are some rules-of-thumb and suggestions to support your critical thinking and review of any given product or service:

  • There are no cure-alls.

Our physiology is complex, made up of many systems. If there were some product capable of curing all manner of conditions, it would be adopted by mainstream medicine after being subjected to the rigorous testing, study and review science by design entails. If the product claims to treat or help with treatment of multiple conditions be skeptical and do some research!

  • Testimonials, anecdotes and endorsements are a sales and marketing technique, not proof of scientific evidence.

Science requires precisely designed, controlled, repeatable and peer-reviewed experiments to build up a body of knowledge. If a product had resulted from such scientific research, the seller would provide that as evidence, and not rely on testimonials alone.

  • Studies aren’t always what they seem.

Anyone or any organisation can commission a study. Real science is an evolutionary, incremental improvement of our body-of-knowledge. It is very rare for one study to be a “breakthrough”, and even if it were, there would be multiple, peer reviewed follow-up studies before any products with proven efficacy made it to market. Studies commissioned by the manufacturer or seller should always be viewed with a skeptical eye. When in doubt, use the search I listed above to find out what experts are saying.

  • If it sounds to good to be true, it probably is.

It’s and old saying, and it’s spot on. Some things are a true breakthrough – it does happen sometimes. But it’s the exception rather than the rule, so once again, be skeptical about miracle claims and do the research!

  • Look out for ‘payment plans.’

If it’s available on a payment plan, it’s quite possible the seller makes their money from the finance because if the product really were a miracle, they’d have licensed it and made millions. This one came from Brian Dunning from and it couldn’t be more true. Once again, the claims may be true, but be skeptical and do the research – it really is a search away!

  • Scientists outside their field aren’t experts.

This one is one of my favourites. Just because someone has a science background, doesn’t mean they can’t be fooled when outside their field of expertise. Do the search to make sure actual experts in the field agree with the poster-scientist’s claims.

  • News outlets are first and foremost commercial concerns, driven by advertising revenue.

Don’t assume just because it appeared in a newspaper or online news site, that the writer or publisher actually confirmed the claims. Staff writers often reprint organisation press releases verbatim, with little or no independent confirmation of the facts. News outlets will often follow formulae for stories, typically extrapolating widely from very early research, drawing conclusions which aren’t yet supported by evidence. Finally, look out for ‘advertorials’, which are paid advertising dressed up as journalistic pieces. These often follow the formula: “New study reveals astonishing link between [this bad thing] and [this product which can help]”, followed by several paragraphs discussing the alleged study, finishing with the company name who provide the product which can help, who invariably commissioned or conducted the so-called study in the first place.

Despite the double-edged nature of the Internet’s power, where purveyors of woo and pseudo-science take advantage of this cheap mass-media and the disproportionate value human-beings place upon anecdotes, being informed really is easier than it has ever been before if you’re prepared to look a little deeper. An understanding of what constitutes ‘scientific evidence’, a skeptical eye and a web browser is truly all you need.
BlogBooster-The most productive way for mobile blogging. BlogBooster is a multi-service blog editor for iPhone, Android, WebOs and your desktop


3 thoughts on “Some pointers on critically assessing the validity of claims

  1. Hi Lucas,

    During a recent flight to Perth I have the privilege of sitting next to a psychiatrist for 5 hours. I a deliberate attenpt to keep the conversation of me and my almost endless list of mentally unique qualllities I chose to keep the subject on my new (trapped in a plane) friend.

    Use of the web as a medium for allowing patents to perform self assessment and research conditions was strongly supported by him and has embraced this technology by hosting his own site… This surprises me, I was expecting the use of online self assessment to be considered an affront to the years dedicated to learning the sacred art of charging hundreds of dollars an hour for many months just to finds it was all your mums fault in the end.

    The subject turned from online medical information to the topic of security and the need to satisfy my on paranoid needs for information security. As it turns out he has a good working knowledge of the next online health system to be released by the Australian Government. This is about 2 years away from release as system across multiple hospitals and related medical clinics are very disjointed and some level of uniformity is needed. This in itself was an interesting subject but for another time.

    What was interesting in our conversation regarding information security was that people generally don’t care or don’t understand what to do to protect themselves. The general approach is default all information to your current or registered GP’s and Specialists access to your online records unless your choice to turn this off. Strange, if the Microsoft / Novell market competition has taught IT anything it is better to be fully secure and turn things off, the Novell approach than leave everything open and expect stupid administrators to lock their system down. When I challenged this point my trapped friend provided an alarming statistic. Of the 5000 users using the system in trials now only 2 have changed their privacy settings. 1 was a mistake where they thought they were turning the privacy in and the other was a patient who changed doctors and was recommenced to do so.

    I hear a small voice whispering in my ear what is it saying, ‘Facebook’, yes good point have we learnt nothing. We CANNOT leave information security to morons, determining security levels and access control, we definitely can not trust the government to get it right.

    I am waiting for the day when my local metal health clinic sends my a flyer in the post with the follow offer: “You have been selected from a special online pool of mentally broken people to get a free session for your many problems”.

    Tell me this information will not get out, TELL me that there is the governance in place to ensure my “In confidence information” conversation with my GP is not used against me or provided in a public study without my knowledge.

    Looks like your priest is the only place where information is secret, I wonder if the church as considered online confessions?


Comments are closed.