The APA Official Definition of a meta-analysis is as follows: n. a quantitative technique for synthesizing the results of multiple studies of a phenomenon into a single result by combining the effect size estimates from each study into a single estimate of the combined effect size or into a distribution of effect sizes. For example, a researcher could conduct a meta-analysis of several studies on the association between self-efficacy and achievement, integrating the findings into an overall correlation. Although meta-analysis is ideally suited for summarizing a body of literature in terms of its impact, limitations, and implications, there are conditions that limit its applicability. For example, there is no minimum number of studies nor participants required, and information of potential interest may be missing from the original research reports upon which the procedure must rely.
Layperson Definition: Have you ever come across research articles that appear to say contradictory things? For example, during Covid you might have heard some people argue that research says that wearing masks can help contain the spread of respiratory viruses (e.g. Teleman et al., 2004; https://pmc.ncbi.nlm.nih.gov/articles/PMC2870165/ ) while others argue that research says that wearing masks do not help stop the spread of respiratory diseases (e.g. Barasheed et al., 2014, https://www.benthamdirect.com/content/journals/iddt/10.2174/1871526514666141021112855). How can you make sense of these seemingly contradictory results? Furthermore, if masks do work, to what extent do they work? These are questions that meta-analyses can help answer. As its name suggests, a meta-analysis is an analysis of analyses. In other words, a meta-analysis takes the results of many different studies that ask the same question and synthesizes them into one big study to help better understand what the research as a whole says. I’ll be referring to the mask example throughout this module, but if you’re curious, several meta-analyses about the subject indicate that wearing a mask does seem to help prevent the spread of respiratory viruses, if only a little bit (e.g. Liang et al., 2020).
It’s important to note that when meta-analyses combine individual studies, they aren’t just comparing the number of studies that find an effect to the number of studies that don’t find an effect. Rather, they are weighing each study in terms of its validity and power. In other words, just because five studies did not find an effect and one study does find an effect, it doesn’t necessarily mean that an effect doesn’t exist. It’s possible that the one study that found an effect had much great statistical power and validity than all five of the other studies combined. In this way, meta-analysis not only aggregates studies, but weights them according to a clear set of criteria on the process.
For an applied analogy of the process used in meta-analysis, think about political polling data. In the months leading up to political elections, polling data is released by a wide variety of sources. Some of these polls might predict that candidate A is in the lead, while other polls might predict that candidate B is the lead. Still others might agree that candidate A is the lead but differ in their estimation of how far a lead candidate A has. Individuals looking to this data to help predict who will win the election might feel very confused when faced with these contradictory findings. To remedy this, polling aggregators like Nate Silver’s FiveThirtyEight attempt to combine each individual poll in a manner that is like those who create meta-analyses. For example, instead of just averaging the information contained in each individual poll, aggregators weight the polls on relevant information, like the sample size or the recency of the data. In this way, the political aggregators are creating a type of meta-analysis in which they are combining and weighting individual data sets to find an overall and more accurate estimate.
Anywhere we are looking for an effect of an intervention. Here are some examples:
Medicine: What is the relationship between wine and beer consumption and vascular risk?
Cognitive Psychology: To what extent does AI significantly enhance students’ 4C skills (critical thinking, communication, collaboration, and creativity)?https://www.sciencedirect.com/science/article/abs/pii/S1747938X2500065X
Is there a relationship between video game violence and subsequent aggressive behavior? (https://www.pnas.org/doi/10.1073/pnas.1611617114)
Public Health: Is there a relationship between childhood vaccines and subsequent development of autism? https://pubmed.ncbi.nlm.nih.gov/24814559/
Education: What is the impact of ChatGPT applications on undergraduate students' learning outcomes? https://www.sciencedirect.com/science/article/pii/S2666920X25001766
When possible, scaffold onto Intro Stats knowledge base.
Power/sample size
As you may remember from your introduction to statistics class, statistical power can be very important in detecting an effect, if an effect exists. I like to have my students imagine standing at one end of a football field while two balloons float at the other end. In one scenario, one balloon is a hot air balloon and the other is a birthday latex balloon. I ask my students if they would be able to detect the difference between the balloons with their bare eyes. They all say yes, and we agree that with such a big difference (big effect), we don’t need high-powered tools to detect the difference. In the second scenario, I have them imagine that both balloons are birthday latex balloons of the same size and inflation, but one of the balloons has a small white dot on it. Students quickly recognize that they would probably not be able to detect that small difference (small effect) with their bare eyes and would probably need help by powerful tools like binoculars to help them detect this small difference. Statistics is a lot like this, and sometimes we need a lot of statistical power to help us find small effects. As you might remember, one of the easiest ways to increase statistical power is to increase your sample size. This is where meta-analysis comes into play. Because meta-analysis aggregates many smaller studies, the N size becomes quite large, giving meta-analyses to detect even extremely small effect sizes that any one individual study may not be able to detect.
Effect size
Since I’m talking about effect sizes, this might be a good time to remind you of what that means. An effect size tells us the magnitude of differences or relationships. So, for example, suppose you read a research study, and it tells you that increased anxiety is statistically significantly related to better test performance (e.g. Siaw et al., 2020). You may want to know the size of that effect before you start trying to stress yourself out in preparation for an exam. For example, maybe being anxious can explain 60% of the variability in exam scores; in that case, it might be a good idea to start stressing! On the other hand (and what the cited research study actually found), maybe being anxious can only explain 1% of the variability in exam scores. Instead of working yourself into a tizzy, it might be helpful to find alternative variables that have a larger relationship with exam success. Understanding effect sizes is important in understanding meta-analyses because meta-analyses collect and summarize effect sizes from many different individual studies on the same topic to create a more stable overall effect size.
Relative Risk, Odds Ratio, Mean Difference (not all intro, so maybe move to vocabulary in interpretation).
Meta-analyses often report effect sizes in terms of relative risk, odds ratio, or mean difference. Here is a quick review of what each of these terms mean.
Relative risk is a statistic that compares the risk of something happening to one group to the risk of it happening to a different group. In statistics, remember that risk is synonymous with probability, so relative risk is a statistic that compares the probability of an outcome for one group compared to the probability of the same outcome for another group. So, for example, if you see a that the relative risk of being diagnosed with skin cancer if you live in an area that has high UV compared to an area that has low UV is 1.8 (Qureshi et al., 2008) that means that people who live in high UV areas are 1.8 times more likely to be diagnosed with skin cancer than people who live in low UV areas.
Odds Ratio is very similar to relative risk, although it’s slightly more difficult to interpret. Where risk (probability) looks at the number of events out of the total possible outcomes, odds is a ratio of the number of events to the number of non-events. So an odds ratio compares the odds of something happening for one group to the odds of something happening to another group. Using the above example skin cancer example, if I was calculating the risk of being diagnosed with cancer, I would take the number of people who were diagnosed with cancer and divide by the total number of people. If I was calculating odds, on the other hand, I would take the number of people who were diagnosed with cancer and divide it by the number of people who were not diagnosed with cancer. It’s a subtle difference and the information provided is the same, although RR and OR provide the information on difference scale. It’s important not to assume these terms are interchangeable since they are calculated in different ways. Relative Risk is usually the preferred statistic, since interpretation is easier.
Mean Difference refers to the difference between group means (obvious, right?). This helps us quantify the size of an effect. For example, if you see a study that shows that people who eat a healthy diet lose significantly more weight than those who don’t eat a healthy diet, you might wonder how much more weight they lost, on average. By calculating the mean difference, you can see if they lose a lot more weight (large effect) or a little more weigh (small effect). Because different studies use different measures to study similar outcomes, it’s often helpful to standardize our mean differences so that we can interpret them across studies. A common standardized mean difference is Cohen’s d.
Confidence Intervals
Remember that the field of statistics exists because we do not have direct access to all populations and their parameters. As a result, we often collect samples from our population of interest and calculate statistics on those samples. We then use those statistics to estimate the population parameters. Because different samples from the same population often produce different statistics that are trying to estimate the same thing (this is known as sampling variability), we know that one statistic is usually not a perfect estimate of the population parameter. To enhance our interpretation, researchers often create confidence intervals around their sample statistics. These confidence intervals adding and subtracting a certain number of standard errors (a measure of variability and spread) to the sample statistic. The most common confidence interval is a 95% confidence interval which is when about two standard errors are added and subtracted to the sample statistic. A way to interpret a 95% confidence interval is to understand that 95% of the time, a sample that is randomly drawn from a population will produce a statistic that is within two standard deviations of the population parameter. In other words, it provides us a range of possible values that is likely to contain the true population parameter.
It’s important to remember that the larger the sample size, the smaller the range of possible values. This is because a larger sample size decreases error, so the two standard errors that are added and subtracted to the statistic in a 95% confidence interval will be smaller than in a confidence interval resulting from a smaller sample size.
Meta-analyses often use 95% confidence intervals to better understand the true effect sizes of individual studies and of all the studies when taken together.
Connection to sampling distributions
One last connection between meta-analysis interpretation and introductory statistics is the sampling distribution. If you remember that a sampling distribution is created by taking all possible samples of a given sample size from a population, calculating a statistic on each of them, and then arranging your statistic into a frequency distribution, then you might already have an idea of how meta-analyses work. In a sampling distribution of the mean, for example, the average of all possible sample means equals the population mean. Similarly, the goal of a meta-analysis is to take all possible effect sizes calculated on a topic, synthesize and average these results so that we can get closer to the true effect size.
Important vocabulary:
Heterogeneity vs Homogeneity of studies
In a meta-analysis, studies are homogenous when the studies used in the meta-analysis are fairly similar to each other, with similar samples, measures, and experimental protocols. Differences between outcomes would generally be due to only measurement error.
Studies are heterogenous when the differences between studies are likely due to more than just measurement error (e.g. different groups of participants, different experimental protocols, different measures). While some differences between studies is common and expected, too many differences require special attention and special statistics when interpreting the results of the meta-analysis.
If there is high heterogeny in the studies, follow up sub-analyses can be helpful to see if there are different effect sizes according to known sources of variability.
Random effects models vs. fixed effect models.
A fixed effect model is a model that assumes that there is one true effect size across studies and that differences between effect sizes in individual studies is the result of sampling variability. This model is often used in meta-analysis when the studies are thought to be fairly homogenous in nature.
A random effects model is a model that assumes that there might be more than one true effect size depending on differences between the studies (for example, maybe there is one effect size for older people, but a different effect size for younger people). This model is often used in meta-analysis when the studies are thought to be fairly heterogenous in nature (Dettori et al., 2022).
Literature search, source selection, and data extraction
Meta-analytic authors identify the process they used to select the data included in their meta-analyses. This information is often presented in the form of a flow chart according to what is called the PRISMA statement (Page et al., 2021). PRISMA stands for The Preferred Reporting Items for Systematic reviews and Meta-Analyses, and it was developed to facilitate the transparency, reproducibility, and complete reporting of meta-analyses. According to PRISMA, authors of meta-analyses must state clearly how they searched for their research articles, how they decided which sources would be included in their meta-analysis, and how they extracted data from each article included. A brief explanation of each is listed below.
Literature Search: authors must explain exactly how they searched for research articles to include in their meta-analysis by stating the databases and search terms used, publication date limitations if applicable, and where they looked for articles (e.g. database results, reference sections of articles produced by the search engine, etc.).
Source Selection: authors must be explicit in the process they used to decide whether or not an article would be included in their meta-analysis. The inclusion criteria often include things like the degree to which the article addresses the research question and publications including target populations, while the exclusion criteria often include things like duplicate publications, insufficient data, conference abstracts, or unavailable articles. To guide selection, it’s important that authors have a well-defined research question, often guided by defining the targeted population, the intervention being examined, a well-defined comparison group, and the outcome of interest (together, these four components of a research question are often referred to as PICO).
Data Extraction: authors must be clear about what data was extracted (e.g. means, odds, effect sizes, etc.), the variables used within each article (e.g. were they categorical? Measurement?), the demographics of the sample (e.g. sample size, target population, age, gender), and basic information about the treatment and design.
Below is an example of a meta-analytic flow chart from a meta-analysis looking at the efficacy of face masks in preventing respiratory virus transmission (Liang et al., 2020). This flow chart was completed in accordance with the PRISMA statement and is similar to what you might see in many meta-analyses. You can see that the authors guide the reader through their process of narrowing down their sources from those that their search produced to those ultimately included in the study.
Tests of Heterogeny: As mentioned above, it’s important to find out if the studies are fairly homogenous or heterogenous so that you researchers can chose between a fixed- or random-effects model. Both Cochran’s Q-test and I2 test for heterogeny.
Cochran’s Q-test: often uses a higher than usual alpha of .1. High p-value suggests low risk for heterogeneity.
I-squared test: percentage ranging from 0% (all variation is expected due to chance) to 100% (no variation is expected to be due to chance).
No consensus for I2 threshold for concern of heterogeny.
Usually greater than 50% is cause for concern that heterogeny might not be acceptable, with 50%-75% considered moderate heterogeny, less than 15%-25% considered low heterogeny, and greater than 75%-80% considered high heterogeny.
Synthesized effect size, with confidence interval
This represents the pooled effect size from the included studies. A z-test for overall effect is often performed to see if the pooled effect is statistically significant (p < .05).
How to interpret a data visualization of this analysis (when appropriate)
Forest Plots (blobbogram—my new personal favorite way to refer to this)
The most common visualization used to present the results of a meta-analysis is a forest plot. Forest plots list the effect size and corresponding 95% confidence interval from each article included in the meta-analysis (usually in order according to first author’s last name, publication date, or effect size).
Individual effect sizes are noted by a dot or square on the graph, often sized according to the weight of a study. If Study A had a large sample size and well-thought out methodology, while Study B had a small sample size and a few questionable methodological choices, for example, then the forest plot dot for Study A might be larger than that of Study B.
Each individual effect size has a line that intersects it. This line represents the 95% confidence interval. Generally speaking, lines that are more narrow indicate more precision (and often a larger sample size). If the confidence interval is very wide and cannot fit on the graph, authors denote this by putting an arrow at the end of the line.
Often times (but not always) forest plots include the relative weight of each study in determining the final summary effect size. Remember, each study is weighted in terms of how confident the researchers are that the study’s effect size is a good estimate of the true population effect size.
Effect sizes are generally reported as Relative Risks, Odds Ratios, or Standardized Mean Differences.
Along the center of a forest plot is a vertical line that is called the line of no effect. This line is usually labeled with a 0 or 1, depending on how the effect sizes are reported. Because both relative risk and odds ratio compare the risk or odds of one group to the risk or odds of another group, if there is no difference between groups, the Relative Risk/Odds Value would be 1. If we’re looking at standardized mean differences, then the value of the statistic when there is no difference between groups would be 0. If a confidence interval crosses this center line, then the accompanying effect size is not significant. Only when the confidence interval does not cross the line of no effect can we be 95% confident that we have captures a true effect.
It’s important to read the labels around the line effect, because some meta-analyses put the experimental group to the left of the line, while other studies put the control group to the left of the line.
As each study is added to the meta-analysis, effect sizes are pooled according to their relative weight, resulting in a final summary effect size. This summary effect size is represented by a diamond shape on the forest plot (usually at the bottom of the plot, but sometimes at the top). The height of the diamond represents the relative strength of the overall effect size, while its width represents the 95% confidence interval. As with the individual effect sizes, diamonds that do not cross the center line represent statistical significance.
Below is an example of a forest plot taken from a meta-analysis about the efficacy of masks in reducing the spread of respiratory infections (Liang et al., 2020. p. 6). As you can see, the left most column is a list of each individual study included in the meta-analysis. The next several columns show data about the two groups being compared (in this case comparing the number of respiratory viral infections contracted in both the mask and the control group). The sixth column indicates the relative weight of each study in terms of how much it contributed to the pooled summary effect size. You can see, for example that the Ailello (2012) study contributed 7% to the final effect size (and thus has a bigger dot on the graph) than the Jaeger et al. (2011) study, which only contributed 1.4% (and thus has a very small dot on the graph). The seventh column reports the individual effect sizes with their 95% confidence intervals, and the graph on the far right is a visualization of this data. At the very bottom of the graph, you can see the summary diamond, which does not cross the line of no effect and indicates that people who wear masks have lower odds of contracting a respiratory viral infection compared to those who don’t wear masks. The statistics listed at the bottom of the forest plot are the same statistics discussed above in the in-text interpretation section.
Funnel Plots
Another visualization that you are likely to find in a meta-analysis is a funnel plot. This plot looks a bit like a triangle with a line down the middle and bunch of dots scattered around. The purpose of this plot is to depict publication bias, with one side of the triangle representing studies that found an effect and the other side of the triangle representing studies that didn’t find an effect. Since the goal of a meta-analysis is to estimate a true effect size by summarizing many individual studies, it’s important to ensure that the authors are getting a full and well-balanced representation of current research. Sometimes, however, studies that do not find an effect are less likely to be published, representing publication bias. The funnel plot provides a quick visualization of whether there are published studies that both find and don’t find an effect (the dots will be scattered all around the funnel) or whether there are mostly only published studies that have found an effect (the dots will be mostly scattered around one half of the funnel). There are statistics that attempt to address this, too, of course, but the funnel plot is a quick and fairly accurate way to get an idea of whether or not publication bias exists.
Below is an example of a funnel plot from the mask article referenced above (Liang et al., 2020, p. 5). You can see that the studies included in the meta-analysis include both studies that have found a significant effect and studies that have not found a significant effect.
Big picture ideas
What can you take away from this analysis?
Meta-analysis provides researchers with a more thorough understanding of the efficacy of a treatment or intervention. By combining the results of many studies on the same topic, researchers are often able to resolve conflicts and get a more precise estimate of the true effect size.
What are the limitations?
The limitations of meta-analysis can be summarized as sources of bias and sources of heterogeneity, both of which are more fully described below. When interpreting meta-analysis, it’s important that both researchers and consumers of research keep these limitations in mind.
Sources of bias
Subjective nature of research: it’s important to remember that the decision to include or exclude an article in a meta-analysis is up to researchers, who may have a bias of their own. There are many guidelines and structures that researchers use to make this process more objective, but, ultimately, the final decision is made by researchers.
Publication bias: perhaps you’ve heard of the “file drawer problem” before in which studies that did not find statistically significant effects are often unpublished and left filed away in the researcher’s drawer. This means that it might be the case that the published findings skew towards significant results. Because meta-analyses combine previously published research, meta-analyses might also suffer from publication bias and favor significant findings over non-significant findings. The funnel plot attempts to check for this, but it is still something of which consumers of meta-analyses should be aware.
Have you heard the saying “garbage in, garbage out?” This is true of meta-analyses as well, and it’s important that when interpreting a meta-analysis, readers pay attention to the quality of articles that are included in the meta-analysis. Hopefully, the authors selected articles of high quality to include in the analysis, but a statistically literate reader will also want to check to make sure that the articles included are well-designed, representative, and statistically sound so that resulting meta-analysis is of high quality as well.
Sources of heterogeneity
Sometimes articles included in a meta-analysis might have drastically different samples, operational definitions, designs, and methodologies. When combined, this can lead to a situation in which researchers are comparing apples to oranges but treating them as if they are the same thing. As mentioned above, noting this sources of heterogeneity and engaging in methodology that acknowledges this heterogeneity is extremely important when interpreting a meta-analysis.
How do you think about the research question based on findings?
Ultimately, meta-analyses can be an excellent tool for researchers to gain precision and perspective about their research topic of interest.
Ailello, A. E., Perez, V., Coulborn, R. M., Davis, B. M., Uddin, M., & Monto, A.S. (2012). Facemasks, hand hygiene, and influenza among young adults: a randomized intervention trial. PLoS One, 7(1)1-8. http://dx.doi.org/10.1371/journal.pone.0029744
Barasheed, O., Almasri, N., Badahdah, A., Heron, L., Taylor, J., McPhee, K., Ridda, I., Haworth, E., Dwyer, D. E., Rashid, H., & Booy, R. (2014). Pilot randomized controlled trial to test effectiveness of facemasks in preventing influenza-like illness transmission among Australian Hajj pilgrims in 2011. Infections Disorders: Drug Targets, 14 (2), 110-116. http://dx.doi.org/10.2174/1871526514666141021112855
Dettori, J.R., Norvell, D. C., & Chapman J.R. (2022). Fixed-effect vs. random-effects models for meta-analysis: 3 points to consider. Global Spine Journal, 12 (7), 1624-1626. http://dx.doi.org/10.1177/21925682221110527
Jaeger, J.L., Patel, M., Dharan, N., Hancock, K., Meites, E., Mattson, C., Gladden, M.,
Sugerman, D., Doshi, S., Blau, D., Harriman, K., Whaley, M., Sun, H., Ginsburg,
M., Kao, A.S. Kriner, P., Lindstrom, S., Jain, S., Katz, J., …Kallen, A.J. (2011).
Transmission of 2009 pandemic influenza A (H1N1) virus among healthcare
personnel—southern California, 2009. Infection Control & Hospital
Epidemiology. https://doi.org/10.1086/662709
Liang, M., Gao, L., Cheng, C., Zhou, Q., Uy, J.P., Heigner, K., & Sun, C. (2020).
Efficacy of face mask in preventing respiratory virus transmission: a systematic
review and meta-analysis. Travel Medicine and Infectious Disease, 36.
https://doi.org/10.1016/j.tmaid.2020.101751
Page, M. J., McKenzie, J.E., Bossuyt, P.M., Boutron, I., Hoffmann, T.C., Mulrow, C.D.,
Shamseer, L., Tetzlaff, J.M., Akl, E.A., Brennan, S.E., Chou, R., Glanville, J.,
Grimshaw, J.M., Hrobjartsson, A., Lalu, M.M., Li, T., Loder, E.W., Mayo-
Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement:
an updated guideline for reporting systematic reviews. Research Methods &
Reporting, 372, 1-9. https://doi.org/10.1136/bmj.n71
Qureshi, A.A., Laden, F., Colditz, G.A., & Hunter, D.J. (2008). Geographic variation and
risk of skin cancer in US women: Differences between melanoma, squamous cell
carcinoma, and basal cell carcinoma. JAMA Internal Medicine, 168(5), 501-507.
https://doi.org/10.1001/archinte.168.5.501
Siaw, E. S., Tan Geok Shim, G. Azizan, F. L., & Mohd Shaipullah, N. (2020).
Understanding the relationship between students’ mathematics anxiety levels and
mathematics performances at the foundation level. Journal of Education and
Learning, 10(1), 47-54. https://doi.org/10.5539/jel.v10n1p47
Teleman, M.D., Boudville, I.C., Heng, B.H., Zhu, D., & Leo, Y.S. (2004). Factors
associated with transmission of severe acute respiratory syndrome among health-
care workers in Singapore. Epidemiology & Infection.
https://doi.org/10.1017/S0950268804002766