The advent of SARS-COV-2 has mobilized many tech people with ample resources and a wealth of ideas. A health crisis like this virus calls for all the help we can get. However, the culture of the tech sector exemplified with the phrase “Move fast and break things” is orthogonal to that of medicine exemplified with the Hippocratic oath of “first do no harm”. What happens when these two approaches meet each other? Unfortunately, the well-intentioned research and communication sometimes results in the trivialization of scientific methods resulting in borderline misinformation that may cause more harm than good.
Moving fast
Under much fanfare the following piece of research was presented by Sermo, a social network platform for medical professionals, on April 2nd: “Largest Statistically Significant Study by 6,200 Multi-Country Physicians on COVID-19 Uncovers Treatment Patterns and Puts Pandemic in Context” . This is a survey of what doctors are prescribing against COVID-19. So far so good. This would indeed be interesting to know. But already the next line sends chills down the spine of any medically informed person: “Sermo Reports on Hydroxychloroquine Efficacy”…Can you spot the the dubious word here? Efficacy? Let’s rewind and rekindle ourselves with what efficacy means: “the ability, especially of a medicine or a method of achieving something, to produce the intended result” according to the Cambridge dictionary.
It gets worse: The CEO claims:
“With censorship of the media and the medical community in some countries, along with biased and poorly designed studies, solutions to the pandemic are being delayed.”
What he means to say then is that the more than 400 clinical trials that are already under way are one and all “biased and poorly designed”? Criticism is always welcome because it sharpens our arguments and logic unfortunately the piece does not have one reference to even one study that would exemplify this bias and poor design.
This is the first clue that this is a tech person and not a medical or scientific, I would say not even academic person. This is a person that moves fast, throws out unchecked assumptions and accusations and then moves fast to his much better designed study that was presumably scribbled equally fast on the back of a napkin.
This is where clue number two becomes evident. What is this superior method that is above the entire medical world scrambling to produce scientific knowledge about the outbreak and efficient therapies? Naturally the inquisitive reader is drawn to the sentence: “For the full methodology click here”. I click and read with bated breath.
We are informed that the survey is based on responses from doctors from 30 countries with sample sizes of 250 respondents. Sounds fair although 30 times 250 is 7.500 not 6200 as mentioned in the title (what happened to the remaining 1.300)? We are told that the Sermo platform is exclusive to verified and licensed physicians. Let’s pause here. How is it exclusive? This is the methodology section, and this is where you tell me HOW you verified the doctors. Otherwise I have no idea whether the results actually mean anything. It could be a mixture of neo-nazis and cosplay enthusiasts for all I know.
Next we read :
“The study was conducted with a random unbiased sample of doctors from 30 countries”.
That’s it. For people unfamiliar with the basics of clinical scientific method this is the equivalent of a suspect in a trial getting up in front of the judge claiming “I totally didn’t do it. Just let me go”. Again, how do we know? Maybe you sent your list of invitations based on a secret list from Donald Trump with doctors that are fanboys of Cloroquine? Maybe the responding doctors are unemployed (for a reason), which would explain why they had time answering the questionnaire. What was the distribution of age and gender? Was it representative of the age and gender distribution of the country they come from? Traditional scientific studies based on samples like these can dedicate up to a third of the article just to demonstrate that indeed there was no bias. Here we are offered one line without any evidence.
The study was based on a survey that took 22 minutes. Basically, any Joe-Never-Was-A-Doctor could have done this with Survey Monkey and a list of emails scraped from the internet. That is also fine, but we don’t get any information about what the questions were. Next section is the “Data Analysis” (and remember we are still in the methodology section) informs us that all results are statistically significant at a 95% confidence interval. Why was 95% chosen and not 99%? What were the actual p values?
In a little less than a page we learn virtually nothing that would help us ascertain the validity of the reported results. And where is the discussion? Could it be that the preferred treatments were dependent more on local availability than choice on the part of the doctor? Was there a bias in terms of geography, gender or age in relation to what they prescribed? Did everyone respond? Was there a pattern in those who didn’t respond?
Although we are left with a lot of unanswered questions, the attentive reader can already from this very sparse information deduce a damning flaw in the study design that completely undermines any of the purported claims to efficacy: the study asks the doctors themselves about their treatments! Now why is that a problem? Doctors, like all humans are susceptible to confirmation bias. This means that they are prone to look for confirming evidence. If they prescribe something, they have reasoned is a good drug they will look more confirmation that this is indeed a good drug. This is exactly why any proper designed study that shows efficacy needs the administering doctors to not know what they are treating their patients with. This is why the double blind test is a hall mark of any demonstration of efficacy.
Where do we go from here?
I am from the tech sector myself and not trained in medical science (although I have taken PhD level courses in advanced statistics and study design), so don’t get me wrong, I believe strongly in the power of technology and I want tech people to engage and help as much as possible. However, as should be apparent by now, this is not helpful.
Had this been presented as what it is, a descriptive survey of what doctors are prescribing against COVID-19, it would have been fine and even valuable. Rather it is pitched as a revelatory study that undermines all current research, something it is not, which may undermine the serious efforts being undertaken currently to find adequate treatments. The clear favourite of the article is Chloroquine, but Chloroquine poisoning has already cost lives around the world due to the current hype. Recently an Arizona man died after ingesting Chloroquine on the recommendation of President Trump. How many more will die after reading this flawed and unsubstantiated “study”?
This is where the “move fast and break things” attitude has to be tempered with the “first do no harm” attitude. When tech people who know nothing about science or medicine use their tech skills they need to openly declare that they do not know anything and that this is not peer reviewed and only subjective opinion. Present the interesting findings as what it is and do not ever make any claims to efficacy or superiority to the medical system of producing knowledge that has increased our global average life expectancy from 30 years to more than 70 years in the past century.
Tech people should still engage but they should stay within their sphere of competence and not trivialize science. Scientists and medical professionals don’t correct them on software design or solution architectures either. So, please don’t get in their way.
Let me then give an example of how tech people should engage. The folding at home project simulates how proteins fold and thereby help the medical community in possible drug discovery. It has succeeded in constructing the world’s most powerful supercomputer delivering more than an exaflop, that is, more than one quintillion calculations per second . It works by letting people install some software on their computer and thereby contribute their compute power in a distributed network of more than a million computers worldwide. This is a good model for how tech people can support the medical community rather than undermine it.
We in the tech sector need to move over and support our friends in the medical world in this time of crisis and approach their world with the same respect and caution that we expect others to show our domain of competence. Even though we are extremely smart, we are just not going to turn into doctors in a few days. Rather than move fast and break things we should “move fast and do no harm”.
