Before reading studies and research, we first need to understand the strength of the evidence hierarchy. This will empower us with the ability to determine a study’s strengths as well as its limitations.
As you move down the list below, the ability to draw causative conclusions from each source gets stronger. The weakest forms of evidence are at the top and the strongest forms of evidence are at the bottom.
Let’s start this off with a bang.
TV documentaries, social media, podcasts, and random articles on the internet are not evidence-based! In fact, they don’t even belong anywhere on the evidence hierarchy. This is not where you should be getting your information from. Often, these sources strategically use “well respected” doctors and professors to make you think they know what they’re talking about. However, there are plenty of “experts” who do not care one bit about being truly evidence-based. They are simply there to push their agenda onto you.
Ok. Now that we’ve cleared that up, onto the real stuff!
In vitro (test tube) research is conducted outside of a living organism in a highly controlled environment. It allows us to study things like human cells. Results from these studies are extremely difficult to extrapolate to in vivo (inside a living organism) scenarios.
Animal research is conducted in a highly controlled environment and is generally used to determine the safety of substances prior to human trials. If you can find both human and animal research in a specific area of interest, you generally should defer to the human studies. We cannot predict human behavior from animal studies regardless of what those studies show.
Editorials, opinions, and ideas may be very useful if they come from the right source. An editorial composed by a world-renowned expert in their field who has completed much of the research in that area is likely a valid source of information and knowledge. However, be extremely wary of editorials written by uncredentialed individuals, as they tend to cherry pick data that supports their own personal ideas and conclusions.
A case report is an article, often written as a story, that describes and interprets an individual case. These cases may not be applicable to the population as a whole, especially due to the presence of many confounding and uncontrollable variables. Case reports are considered one of the lowest forms of evidence and can cause readers to focus on misleading elements. However, they aren’t completely without a place or purpose. They can highlight issues and help form the basis of new ideas and hypotheses.
Observational studies are complex and rich sources of information that allow us to observe trends within extremely large populations. However, they cannot control for confounding variables, and are very dependent on the researcher’s ability to record and interpret data. They show correlation and association, but not causation, and the methods used within these studies may not be reliable, valid, or accurate.
One type of observational study is a retrospective case-control study, which compares a group of patients with a specific disease to a group without the disease. Each groups’ frequency of exposure to a specific risk factor is retrospectively examined to assess the existence of a possible relationship between the risk factor and the disease. One issue with these studies is that they rely on the subjects’ memory recall, which can be extremely biased.
A more reliable observational study is a prospective cohort study. In a cohort study, one or more samples (cohorts) are followed and evaluated with respect to a disease or other outcome to assess possible risk factors. However, confounding variables are not measured or controlled.
A randomized control trial (RCT) assigns participants at random to either a control or an experimental group, and then measures the effectiveness of an intervention or treatment on the experimental group. Researchers can control for many confounding variables in these studies, and so can evaluate possible causal relationships since the only expected difference between the two groups should be the outcome variable of the intervention or treatment.
Practice guidelines, otherwise referred to as evidence-based or clinical guidelines, are statements produced by a panel of experts that outline current best practices to inform professionals and patients in making decisions around a specific area or topic. They are created after an extensive review of the literature and can be published by professional associations, government agencies, public, and private organizations. These panels should consist of a variety of experts with a diverse array of associations.
A systematic review is an evidence-based resource that answers a defined research question by comprehensively and exhaustively examining and combining all relevant studies on the specific topic of interest, and then summarizing the findings. Systematic reviews are more reliable and accurate than individual studies, and their findings can be generalized and extrapolated to the general population. Systematic reviews may include meta-analyses.
A meta-analysis uses statistical methods to form conclusions from the results of numerous, individual studies in a specific area of research. These conclusions are statistically stronger than the analysis of any single study as meta-analyses incorporate a larger group of subjects and contain greater subject diversity as well as accumulated effects and results. Meta-analyses that combine several selected RCTs are the highest level of evidence on the evidence hierarchy, and their results can be extrapolated to the general population.
Additional questions to ask:
Is this a peer-reviewed study? A peer-reviewed study is reviewed by several other experts in the field before the article is published in a journal to guarantee the article’s quality.
Was this study repeated, and if so, were the outcomes or findings different or the same?
Did the researchers use reliable and valid methods in their study? For example, using a DEXA scan would not be a valid or reliable method to measure blood glucose levels.
Did the study assess what it actually needed to assess? Say, a study concludes that beta alanine improves sports performance. However, the study’s results actually show that beta alanine increases muscular carnosine levels. There’s an important piece of information missing here. Does increased muscular carnosine levels translate over to improved sports performance? How is sports performance defined and measured?
Are the results of the study realistic? If a study reports that 7-8kg of muscle was gained in 12 weeks using their intervention, then there is likely something suspicious going on (cough, cough, steroids, cough, cough).
Was the sample size small? Did it only contain a few individuals? A small sample size could mean that the results are not yet applicable to the population as a whole. There are always outliers in sample groups, and the smaller the sample size is, the more likely it is that these outliers affect the results.
Speaking of the subject pool, are the groups in the studies similar or very different from the population you are interested in? For example, if a study uses trained male endurance athletes, then the results of that study may not be applicable to detrained males, or female endurance athletes.
Additional tips:
Read the entire article, not just the conclusion or the abstract. Why? Because if you only read the abstract, you will miss extremely important details included in the study. Researchers also have their own biases and opinions, and how they write their conclusions can affect how you interpret the results from their study.
Look at the study’s funding to see if there may be any conflicts of interest. For example, if the US Poultry and Egg Association funded a study that found that eating 12 eggs a day prolongs your life by 20 years, that is suspect.
Be conscious of the adherer effect, which is the concept that those who are more likely to adhere to a diet, treatment, or intervention are also more likely to adhere to other healthy practices. This may confound the results of a study.
Understand the difference between absolute risk versus relative risk. Absolute risk is your risk of developing a disease over a period of time. Relative risk is used to compare the risk between two groups of people.
Say your lifetime risk of getting colon cancer at age 50 is 0.018 (or 1.8%). If you eat 100g of red meat a day, your relative risk of getting colon cancer is 1.14. Your total relative risk of getting colon cancer at age 50 is therefore 1.14 x 0.018 = 0.021 x 100 = 2.1%. NOT 114%.
P-value = the probability of observed results arising by chance.
