The Scientific Method, Challenging Orthodoxy – Dr. Bedard

Some more of the scientific method, brought to you by JAMA:

The Journal of Assiginack Medical Association.

“Newt population density in a small pond behind Assiginack public school” – August 11, 2020.

Purpose

To determine the density of newts in a nearby pond.

Methods

Three study teams undertook this task using three methods, and three models of funding.

Team #1

Bob, Sue, and Sally – the research leaders in the Grade 3 class – had enough funding for one jar, one net, and limited their sampling to three random net-scoops in the north half of the pond.

Team #2

Johnny, Amy, and Lilliane – from the Grade 7 class – had funding for 2 nets, 2 jars, and limited their sampling to the southern half of the pond. Three scoops were done.

Team #3

A reassembled Kindergarten squad led by Mrs. Moggy (Redux) had big Manitowaning financial backing with 33 nets (because class sizes were not appropriately reduced in the pandemic), 33 jars and sampled the entire pond. Scooping was done until it was determined by Fred (who is a Kindergarten leader) that all the newts had been caught.

Results

Team #1

This team scooped up 3 newts from the north end of the pond and determined a population density of 1.0 newt per cubic liter.

Team #2

This team scooped up 16 newts from the south half of the pond and determined a population density of  5.2 newts per cubic liter.

Team #3

This team scooped up 16 newts and determined a population density of 2.3 newts per cubic liter.

Discussion

These studies deliver different, widely varying conclusions. Which one is correct? We can see that these studies were not equally powered. Team #3 has the most comprehensive sampling methods, but there is an inherent bias. We do not know Fred’s academic criteria and how he came to his conclusions. We also have an issue with all teams. Who did the sampling? Are the athletic abilities of the samplers similar? What about their eyesight? (Lots of possible confounding variables).

Further documentation was requested of Team #3 and it turns out Fred is on the ball, he had the pond pumped out and all visible newts were obtained as verified by Allison who has early OCD, and submitted meticulous raw data (What is Raw Data?) so the reader can ascertain accuracy. The best evidence, therefore, is a newt density of 2.3/L³. But it also turns out that the math is incorrect, as peer-reviewed by Dr. Smith who actually knows mathematics. Other readers had just assumed accuracy.

Analysis

It turns out that most of the newts actually prefer hanging out in the southern half of the pond. It is warmer, with more amenities. This skewed the results of Team #1 and #2 significantly and reinforces the point that the larger the number of study participants, the higher the “power”, and therefore the accuracy, of the study.

Big funding can be good. Spotting bias is the job of trained learners – “peer review”. One should EXPECT a variety of study conclusions. Some things happen due to causation.  Some things happen due to chance. The scientific method is designed to answer these questions. Researchers are supposed to argue. They are not supposed to suspend reason and replace it with emotion. That is analogous to a cop who only looks for evidence to support his pre-judged verdict.

What is currently happening is the politicization of science. Not good. Likewise, recent statements by the CPSO (College of Physicians and Surgeons of Ontario) presumably reinforcing toeing the line with “best public health practices” causes this writer concern. I do hope the angry rhetoric dies down, that the College rightly holds its members to appropriate conduct. But this turmoil, publicly witnessed, is like watching your neighbours argue, and it is dirty.

Please let the scientists do their jobs. The information is supposed to vary, and becomes more accurate as testing methods improve, data quality improves, and peer reviewers confirm the findings.

Side Rant:

Throughout my entire career I have followed the bouncing ball of “evidence”. Remember, that evidence is only as accurate as the study that produced it. To the layperson, the word evidence implies truth – not necessarily so. We in the scientific community have done a crappy job, IMO, explaining the nuance of the definition, which creates endless public confusion, and can often be manipulated for financial benefit. We didn’t sit in libraries, missing our 20’s, just to mislead the public. This is tough work and it takes significant mental training to keep your academic eye on the ball. The scientific method works (if you see hyphenated science, beware)… Eventually.

For those who feel it unwise to challenge the CPSO or others, consider the following cases who rose up against the academic orthodoxy of their time:

Dr. William Harvey

The 17th-century dude who correctly anatomized the circulatory system. Prior to that, they thought the blood all came from the liver.

Bill – 1, Orthodoxy – 0. Liver is gross.

Dr. Gregory Mendel

Monkeyed around with Gene theory in the 1800s. He stated that each parent contributes a blue gene and a red gene to offspring. Orthodoxy stated it was a parental pink blend.

Monk Mendel – Correct, Orthodoxy – Wrong.

Dr. Joseph Lister

Circa 1900s British Surgeon (Thank this guy for you pre-date Listerine). Laughed at for cleaning his instruments with phenol before re-use.

Joe – 1, dirty filthy Orthodoxy – 0.

Dr. Ignaz Semmelweis

Circa 1840 correctly identified doctors’ hands as a vehicle of infectious transmission. Imagine. Previously thought to be due to air transmission (also true).

Thank Iggy for the concept of handwashing.

Dr. Barry Marshall & Dr. Robin Warren

1983 – These guys have guts. They proposed (all science asks a question which then begs an answer yay or nay from best data available – see JAMA 2020 Assiginack newts) that stomach ulcers can be caused by a bacterium called H. pylori. They were mocked. So Dr. Marshall took the stomach juice of an afflicted patient, mixed it with a stiff drink (stiff drink untrue but the imagery helps the writer), and swallowed it, giving himself a stomach ulcer. He shoved a gastroscope down his own stomach, biopsied his own new ulcer, and proved Hpylori were present. He expanded on his data by treating the infection successfully, and demonstrated resolution of H. pylori and the ulcer – causation, not association.

Very gutsy, very correct, very gross. But thanks.

My own feelings about “best public health practices”  is that they initially did a very poor job (‘travel bans are racist’ – more political contamination), but that they have since caught up with intelligent messaging. I disagreed with them at first, I now agree with most of their statements, and what brought us together was a good scientific method producing improved data. Peer review is SUPPOSED to challenge the orthodoxy.

To all those who taint my beloved scientific method with the slime of politics and unreason, I again encourage a nice hike. But not to the newt pond. It is tranquil.

– M

PostScript

Kindergarten Arnold recognized the inherent sociopathy of kindergarten Fred, and had all the water and newts returned safely to the pond. Where they continue to hang out in the South end. The end doesn’t necessarily justify the means. Another debate for another time.