Evidence Based Medicine: A Critical Look

In 1992, the first medical article discussing the concept of Evidence Based Medicine was published in JAMA.[2] By the time I was looking at residency programs in 1999, it had become the newest rage in training programs. At the University of Virginia program, residents were given pocket computers (a forerunner to the iPhone). These had the Cochran Database and residents could look up any article published, reviewed and approved in the then current literature and use that information to care for patients. It was incredibly compelling. Three years later at the Medical College of Virginia, my residency training program, the mantra had become ‘show me the evidence for your treatment.’ Evidence Based Medicine (i.e. EBM) had become the new standard of care. But today, in 2018, is this standard still valid?

What I’d like to do in this short article is describe EBM, how it was meant to be used, how we currently use it, it’s foibles and flaws, and how we move forward. This is important for both patients and students. Unless you know how the evidence in medicine is defined and utilized, it is easy to get confused about what the word evidence means and how it is used in patient care.

Three years later at the Medical College of Virginia, my residency training program, the mantra had become ‘show me the evidence for your treatment.’ Evidence Based Medicine had become the new standard of care. But today, in 2018, is this standard still valid?

Types of Evidence

When the phrase Evidence Based Medicine is used, it typically refers only to Randomized Double-Blind Placebo Controlled trials (i.e. RTCs). In these trials a theory is tested in two groups. One group receives a treatment and the other a placebo (non-treatment). No one knows which group gets what, this is the blinded part. At the end of the trial the outcomes of the groups are compared statistically. If the treatment group shows a statistical benefit, defined by a probability value <.05, then the treatment is considered different from placebo. This is the gold standard.[3] However, these studies are very expensive, can take years to get approved and published and they only answer the single question asked in the hypothesis. The reality is that most of what we do in clinical practice does not meet this stringent criterion.

In a Meta-Analysis, the results from multiple studies are compared critically and the findings from this critical examination are published. These studies are used to look back at a body of published research and re-examine the results to determine the validity of that collection of information and its clinical relevance, therapeutic efficacy and potentially plan future trials. These types of studies are much cheaper to perform, as the research has already been done and allows the published literature to get a ‘second look.’ However, it can’t answer new questions. It is used to verify old answers and generate new questions for future research.

A Systematic Review is a review of the medical literature using a collection of the available research around a clinical entity or question. The researchers use an organized method to locate, collate, critically evaluate and then summarize their findings. These studies are excellent to get an up to date overview of the medical literature around a disease or question that has already been researched and critically evaluated. UpToDate is best known resource used by physicians that contains this form of evidence. In Prospective Studies a group or population is observed over time evaluating for a change in health status. Many times, a known treatment, like a diabetes medication, is used and the outcome the researchers are looking for is cardiovascular disease prevention. This type of evidence is useful to evaluate existing treatments for either new side effects or other beneficial uses.

In Epidemiological Studies a different approach is taken. A population of individuals is observed and correlations between diseases and associated factors are elucidated. This is a kind of Meta Data approach to clinical research. The main goals are to discover what factors in a select population are associated with disease and which are associated with health. The information gathered in these studies is then used for further future research. The mantra one must remember in this type of research is ‘correlation does not equal causation.’ Just because something is related (like drinking coffee and lung cancer in the 1970’s) doesn’t mean it is causal (i.e. coffee consumption does not cause cancer, the smoking that occurred with coffee drinking in the 70’s was the cause). An example of this would be the population-based research of Weston A. Price that correlated processed food consumption in different ethnic groups around the world with dental diseases, dental arch malformations and tuberculosis.[4]

Case Based Medicine was the standard prior to our current era. In this, a physician would write an article about a specific patient. The writer would summarize the basic science behind a clinical condition then write how they approached the patient and then the outcome of that treatment. Typically, this was used for unusual cases, to help physicians care for unique diseases or conditions that they may not have seen previously in their clinical practices. These types of publications came out of the tradition of physician mentor-ship in medical schools. This is the way students start their training; an attending physician guides the student through individual cases the student has never seen and over time the student builds up a knowledge base to draw from in their clinical care.

There are many other forms of evidence acknowledged in the medical literature, but these are the most common ones referred to. When used in the current scientific environment, Evidence Based Medicine only refers to RTCs.

Original Intention, Misconceptions & Results

In its original intent, Evidence Based Medicine (i.e. EBM) was meant to integrate the best available evidence with a clinician’s experience and blend that with the patient’s values, preferences and unique clinical situation.[5] Though laudable, the original intent has been skewed. What has occurred over the intervening decades has been a medical system that only recognizes RTCs as valid and any other medical evidence and clinical experience as anecdotal. The result has been the slowing acceptance of new scientific data into clinical practice.

Many medical professionals refer to the new mantra as ‘cook book’ medicine. A patient enters the office with a complaint, it is the clinician’s job to find a RTC to match their complaint and then begin that specific treatment. However, because the data in RTCs is collected from specifically selected populations, it is not ultimately individualizable.[6]

To make matters worse, insurance companies have begun to use EBM as the standard for medical reimbursement. No longer does the clinicians experience and expertise determine what treatments are appropriate for patients. The specific diagnosis is matched to acceptable treatments from the EBM literature, interpreted by the insurance claims office, and if things don’t match up, the medicine or procedure is not reimbursed. Nowhere in the original articles on EBM was the intent for this to impede a clinician’s ability to practice medicine or determine what is paid for, but this is the current state of affairs.

Evidence Based Medicine also doesn’t take account for publication bias. In today’s competitive academic world there is a push to publish articles. These articles must meet certain criteria and ultimately result in grants to the university or an increase in its prestige. One result of this push is that research questioning the current scientific paradigm often isn’t published. Negative research results tend not to get published and mostly positive results are published. An example of this is the Autism Spectrum Disorder literature where 100% of 115 articles published on oxidative stress have positive findings and no articles on negative findings have been published. The result over time is an overall negative bias against publication in the literature of negative findings and a positive bias to publication for positive findings.[7]

As well, a large part of medical research is funded by pharmaceutical companies whose bias is to have positive results for their products. Negative studies tend not to make it to publication or are not considered relevant. The results can be disastrous. Medications, whose efficacy and safety are not proven, make it to market based on poor quality data that meet the minimal EBM criteria. When was the last time you hear about a drug recall? or a drug was taken from the market due to side effects? An example of the effects of this bias by pharmaceutical companies and research publication is seen with the drug lamotrigine. The negative studies on the drug were viewed as outliers and the positive ones seen as proof of efficacy. The result was a medication with significant side effects and questionable efficacy in Bipolar disorder was approved for use.[8]

According to Dr. Ioannidis, the published results of most scientific studies are later found to be false. In many fields, the research published only reflects the current bias of that field. Combine this with small sized studies with low power, the flexibility of altering research design, pretest selection of patients and study bias; the outcomes may be statistically significant but on future reevaluation are later found to be false.[9] In the scientific community where publication is king, later retractions are often overlooked in the quest for reoccurring publication.

Finally, the limitations to EBM are nicely summed up in the Parachute Article.[10] This article from the British Journal of Medicine shows that something as logical as the efficacy of parachutes has not had any RTCs performed to prove that they work. We only have anecdotal evidence that they work. Yet who would do such a study with a placebo arm (one group not given parachutes). The point is well taken, there are limitations to what RTCs can do. But where do we go from here?

How Do We Move Forward?

This quick review was not meant to be exhaustive, but rather to start the conversation on how best the practice of medicine should incorporate new research and data into clinical use. This is the Art of Medicine. Every patient is an experiment with a N=1 (single patient trial). Every patient is unique. Every individual has their own story, family, health and disease process. The original intent of EBM was to create a reputable body of scientific literature that an experience clinician can then interpret and synthesize in their daily interactions with unique individuals. In my 15+ years of clinical practice I have had over 70,000 office visits with over 8000 different individuals in 6 countries and participated in over 50 different clinical trials. My clinical experience is continually being reinterpreted based on new research that comes to light. Yet the incorporation of these findings is tempered with my experience. In the truest sense, a physician is a clinical scientist. EBM should not ignore the experience of every seasoned clinician but inform it.

This is the Art of Medicine. Every patient is an experiment with a N=1 (single patient trial). Every patient is unique. Every individual has their own story, family, health and disease process.

We should also recognize that scientific findings are interpreted based on the currently accepted paradigm. Thomas Kuhn, in his epic work The Structure of Scientific Revolution, shows us how scientific progress does not proceed in a linear fashion but in seismic jumps. He gives examples of things we now consider common knowledge that were once considered radical and impossible. It takes time and a labor-intensive compilation of information until an argument is so persuasive that the prevailing thought must change. Examples include the discovery of a round earth by Galileo, gravity by Newton, hand washing by Semmelweis. Or a current example would be smoking, which took over 50 years and 7000 research articles before being declared a carcinogen by the Surgeon General of the U.S.In summary, we just need to go back to the beginning, Ad fontes. We need to integrate the best available evidence with a clinician’s experience and blend that with the patient’s values, preferences and unique clinical situation. This was the initial intent of Evidence Based Medicine and we would be well served by returning to its original intent.[11]

Other Resources

Why Most Published Research Findings Are False

John P. A. Ioannidis

Summary

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

Read the Article

References

1. German theoretical physicist who originated quantum theory earning him the Nobel Prize in physics.
2. Evidence-Based Medicine Working Group. Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA. 1992 Nov 4;268(17):2420-5. PubMed PMID: 1404801.
3. Misra, Shobha. “Randomized Double Blind Placebo Control Studies, the ‘Gold Standard’ in Intervention Based Studies.” Indian Journal of Sexually Transmitted Diseases and AIDS 33.2 (2012): 131–134. PMC. Web. 12 Oct. 2018.
4. Price, Weston A. Nutrition and Physical Degeneration. Price-Pottenger Nutrition Foundation; 8th edition (2009).
5. Straus S, Haynes B, Glasziou P, Dickersin K, Guyatt G. Misunderstandings, misperceptions, and mistakes. ACP J Club. 2007;146(1):A8–A9.
6. Siwek, Jay. JAY SIWEK, MD, Evidence-Based Medicine: Common Misconceptions, Barriers, and Practical Solutions. Am Fam Physician. 2018 Sep 15;98(6):343-344.
7. Ridha Joober, MD, PhD, Norbert Schmitz, PhD, Lawrence Annable, Dipstat, and Patricia Boksa, PhD. Publication bias: What are the challenges and can they be overcome? J Psychiatry Neurosci. 2012 May; 37(3): 149–152.
8. S. Nassir Ghaemi, MD, Arshia A. Shirzadi, DO, and Megan Filkowski, BA. Publication Bias and the Pharmaceutical Industry: The Case of Lamotrigine in Bipolar Disorder. Medscape J Med. 2008; 10(9): 211.
9. John P. A. Ioannidis. Why Most Published Research Findings Are False. PLoS Med 2(8): e124. https://doi.org/10.1371/journal.pmed.0020124
10. Smith, Gordon C. S., and Jill P. Pell. “Parachute Use To Prevent Death And Major Trauma Related To Gravitational Challenge: Systematic Review Of Randomised Controlled Trials.” BMJ: British Medical Journal, vol. 327, no. 7429, 2003, pp. 1459–1461. JSTOR, JSTOR, www.jstor.org/stable/25458082.
11. Straus S, Haynes B, Glasziou P, Dickersin K, Guyatt G. Misunderstandings, misperceptions, and mistakes. ACP J Club. 2007;146(1):A8–A9.