Meta-Analysis

When a review is performed following predefined steps (ie, systematically) and its results are quantitatively analyzed, it is called meta-analysis. Publication of meta-analyses has increased exponentially in pubmed.gov; using the key word “meta-analysis,” 1,473 titles were yielded in 2007 and 176,704 on January 2020. Well-designed and reported meta-analyses provide valuable information for clinicians, researchers, and policymakers. The aim of this study was to provide CHEST peer reviewers, as well as authors and researchers in training, with tools that can help to improve the quality and timeliness of journal reviews, as well as the quality of the meta-analyses submitted. This article also is intended to be a practical guide to inform authors about the key features of meta-analyses to be considered when producing their review. When a review is performed following predefined steps (ie, systematically) and its results are quantitatively analyzed, it is called meta-analysis. Publication of meta-analyses has increased exponentially in pubmed.gov; using the key word “meta-analysis,” 1,473 titles were yielded in 2007 and 176,704 on January 2020. Well-designed and reported meta-analyses provide valuable information for clinicians, researchers, and policymakers. The aim of this study was to provide CHEST peer reviewers, as well as authors and researchers in training, with tools that can help to improve the quality and timeliness of journal reviews, as well as the quality of the meta-analyses submitted. This article also is intended to be a practical guide to inform authors about the key features of meta-analyses to be considered when producing their review. Meta-analyses are powerful study designs that combine existing published and unpublished studies to pool the effects of interventions (eg, drugs, devices, surgeries, treatment strategies) on clinical and intermediate outcomes. There is no meta-analysis without a previous systematic review; therefore, solid methods for systematic reviews are required for all meta-analyses. A systematic review involves several steps that can be described in a protocol: defining a clear research question, describing a search strategy, defining clear inclusion and exclusion criteria for studies, using several search engines for searches (eg, PubMed-MEDLINE, EMBASE, Scopus, Cochrane library), independently selecting studies, extracting study and outcome data, and assessing risk of bias of studies. Lack of one or more of those systematic review steps diminishes the quality of meta-analyses. Conduction of systematic reviews and meta-analyses requires careful planning and design. The Cochrane collaboration (www.cochrane.org) spends considerable time with authors to create a sound protocol prior to embarking on the actual performance of a systematic review and meta-analysis. Protocols are published first on their Website. Other scientific journals request the registration of the systematic review protocol in a registry (eg, PROSPERO [https://www.crd.york.ac.uk/PROSPERO/]). Importantly, protocols can be registered in PROSPERO only if they are in the initial steps of the process. Systematic reviews and meta-analyses inform clinicians, researchers, and policymakers on the direction of the effect of interventions, the strength of available information of those effects, and the potential harms associated with those interventions. They are the foundation of more complex statements such as health technology assesesments and clinical practice guidelines. The most common classification of meta-analyses includes two types: traditional meta-analyses and nontraditional meta-analyses. Traditional meta-analyses assess effects of one intervention compared with another intervention (eg, an investigational intervention, usual practice, placebo) using aggregated data from previous studies; they pool randomized controlled trials (RCTs), observational studies (eg, cohort studies, case-control studies), diagnostic studies, and prognostic studies. Meta-analyses of RCTs represent the best possible option to summarize the beneficial and harmful effects of interventions. However, they are not free from bias, as RCTs can have high levels of bias related to weak randomization methods, lack of blinding, and incomplete outcome data. Meta-analyses of observational studies have more challenges because cohorts and case-control studies are different by design; however, meta-analyses often combine them. Several sources of bias are present in observational studies; a frequently occurring source is selection bias, which makes groups of interventions being evaluated not comparable at baseline. Therefore, the effects of interventions that are calculated may be due to other factors (eg, confounding by indication, where sicker patients may receive more aggressive or newer treatments than those in healthier patients). Meta-analyses of diagnostic studies or diagnostic test accuracy meta-analyses evaluate specific tests vs a gold standard in studies with patients at risk of having a disease (ie, there are patients with and without disease). Current methods adjust for the intrinsic correlation of sensitivity and specificity and calculate summary areas under the receiving-operating characteristic curves. Meta-analyses of prognostic studies calculate a summary effect of the association of a given factor with an outcome or summary effects of predictive accuracy (overall performance, calibration, and discrimination). Nontraditional meta-analyses include network meta-analysis (NMA), individual patient data (IPD) meta-analysis, and meta-analysis of rare events; all of these are mainly performed by using data from RCTs. An NMA compares multiple interventions at the same time and combines both direct (ie, actual comparisons) and indirect (ie, calculated from other comparisons) effects; some comparisons summarize both direct and indirect effects. IPD meta-analyses use data from every individual patient and allow specific subpopulations to be studied in greater detail than is possible with aggregated data; because access to individual-level data is more challenging, these meta-analyses are uncommon. Meta-analyses of rare events involve the assessment of outcomes that correspond to < 10% of the total number of individuals in a trial arm. The scarcity of events and the poor reporting of harmful events found in a meta-analysis of rare events dictate that these analyses use very specific methods. There was uncertainty about the effects of low-sodium salt substitutes (LSSS) on clinical outcomes. In a meta-analysis of RCTs, we evaluated the effect of LSSS vs regular salt on BP, detected hypertension, stroke, mortality, and other clinical outcomes.1Hernandez A.V. Emonds E.E. Chen B.A. et al.Effect of low-sodium salt substitutes on blood pressure, detected hypertension, stroke and mortality: a systematic review and meta-analysis of randomised controlled trials.Heart. 2019; 105: 953-960PubMed Google Scholar Because effects were expected to differ among populations with different baseline risk, analyses were primarily stratified according to type of population (hypertensive, normotensive, and mixed). Of 21 RCTs (n = 7,403), we found that LSSS decreased systolic BP, diastolic BP, and urinary sodium excretion; LSSS also increased urinary potassium and calcium excretion levels but did not affect mortality compared that of with control subjects. Other outcomes were scarcely reported. Risk of bias was high in five studies, and the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) quality of evidence (www.gradeworkinggroup.org) was low to very low for most outcomes. Comparative efficacy of recently approved bone anabolic therapies (BATs) for postmenopausal osteoporosis was unknown. We evaluated the effect of each of the BATs (teriparatide, abaloparatide, and romosozumab) vs bisphosphonates, placebo, or no treatment on fractures, bone mineral density, and bone metabolites.2Hernandez A.V. Perez-Lopez F.R. Piscoya A. et al.Comparative efficacy of bone anabolic therapies in women with postmenopausal osteoporosis: a systematic review and network meta-analysis of randomized controlled trials.Maturitas. 2019; 129: 12-22Abstract Full Text Full Text PDF PubMed Scopus (12) Google Scholar Because we were interested in knowing which BAT is better than other interventions, a frequentist NMA was performed per outcome, and p-scores were used to rank best treatments per outcome. Sixteen RCTs (N = 18,940) were evaluated; all BATs reduced the risk of vertebral fractures vs other interventions, and no intervention decreased the risk of nonvertebral fractures. Abaloparatide, romosozumab, and teriparatide were the best treatments to reduce vertebral/nonvertebral fractures, to increase bone mineral density, and to increase bone formation, respectively. Cohort data offer the opportunity to evaluate associations of patient characteristics with clinical outcomes. The association of obesity and postoperative atrial fibrillation (POAF) in patients undergoing cardiac surgeries was evaluated in a meta-analysis of observational studies.3Hernandez A.V. Kaw R. Pasupuleti V. et al.Obesity as a risk factor for postoperative atrial fibrillation after cardiac surgery: a systematic review and meta-analysis.Ann Thorac Surg. 2013; 96: 1104-1116Abstract Full Text Full Text PDF PubMed Scopus (38) Google Scholar In 18 cohort studies (N = 36,147), we found that obese patients had a modestly higher risk of POAF compared with nonobese patients, and there was no difference when evaluated according to type of surgery, design (prospective vs retrospective), or year of publication. The effect of confounders on outcomes has to be explored in a meta-analysis of this type. Study quality was good for all studies; importantly, all studies identified confounders or prognostic factors of the association between obesity/BMI and POAF, but these were not the same or defined in the same way across studies. No adjusted effects of this association were available, and therefore the comparison between adjusted and unadjusted effects was not possible. We also assessed the effects of platelet glycoprotein IIb/IIIa receptor blockers in non-ST segment elevation acute coronary syndromes in several age subgroups.4Hernandez A.V. Westerhout C.M. Steyerberg E.W. et al.Effects of platelet glycoprotein IIb/IIIa receptor blockers in non-ST-segment elevation acute coronary syndromes: benefit and harm in different age subgroups.Heart. 2007; 93: 450-455Crossref PubMed Scopus (19) Google Scholar Older individuals may have a higher risk of major bleeding but also larger reductions in death and coronary events compared with younger individuals. Although outcomes were available for age subgroups in individual trials, adjustment for confounders was not possible. IPD meta-analysis offers an ideal situation to assess subgroups and adjust for confounders. Using IPD from 6 RCTs (N = 31,402) and multivariable regression analyses adjusting for trial and other relevant confounders, we found that relative reductions in death or myocardial infarction and relative increases in major bleeding were similar across age subgroups. Patients aged > 80 years had larger absolute increases in major bleeding but also the largest reductions in death or myocardial infarction. A systematic review, if properly performed, finds all available information for a given research question and assesses the quality of the body of evidence. Results of the subsequent meta-analyses can affect clinical practice. It is important to develop good search strategies with the help of an information specialist or librarian. This often includes a search of at least three search engines as well as repositories of RCTs such as www.clinicaltrials.gov or www.clinicaltrialsregister.eu for extra outcome data of finished or ongoing trials. It is also important that at least two researchers independently select studies, extract study and outcome data, and assess the risk of bias, with discrepancies resolved by another researcher. Meta-analyses have the advantage of having more power than the individual studies to detect an effect of an intervention on an outcome. CIs of pooled effects can be more precise (ie, narrower) than those from most of the individual studies. Meta-analyses also allow researchers to assess how studies vary with respect to populations, interventions, controls, outcomes, time of intervention or follow-up, or design (ie, methodologic heterogeneity) and how different the effects of interventions are across studies (ie, statistical heterogeneity). Effects across subgroups of individuals within studies are also possible in meta-analyses; subgroups are defined by using the baseline characteristics of patients (eg, age, sex, severity of disease, diabetes). It is possible to adjust for confounders when IPD are available but only for exploratory purposes to develop new hypotheses for future studies. Systematic reviews also help judge the quality of the body of evidence for a given research question. Importantly, quality of evidence is evaluated per outcome, not per study. The GRADE working group developed the methodology to create Summary of Findings tables. These tables incorporate information on absolute and relative effects vs controls per outcome as well as judgment of the quality or certainty of evidence per outcome based on five items: risk of bias, publication bias, inconsistency, imprecision, and indirectness. An example is provided in Table 1 of Hernandez et al.1Hernandez A.V. Emonds E.E. Chen B.A. et al.Effect of low-sodium salt substitutes on blood pressure, detected hypertension, stroke and mortality: a systematic review and meta-analysis of randomised controlled trials.Heart. 2019; 105: 953-960PubMed Google Scholar Meta-analyses can be misleading if they are not preceded by a sound, appropriate systematic review. Combining several studies with methodologic differences and with heterogeneous effects on outcomes can be problematic if heterogeneity is high and its sources have not been fully explained and addressed (eg, adjustment for confounders, exclusion of heterogeneous studies). Authors should justify why they decided to run meta-analyses when methodologic and statistical heterogeneity is high; in most cases, we do not recommend performing meta-analyses in the presence of high heterogeneity. Summarizing large amounts of information using a single number can also be controversial. Combining several studies with a high risk of bias will not result in a meta-analysis with a low risk of bias. The strength of pooled estimates depends on the quality of individual studies being assessed and analyzed. There is no way to adjust for risk of bias of individual studies; one of the strategies to follow is to exclude those trials from primary analyses or from secondary sensitivity analyses. Combining studies in a meta-analysis requires an evaluation of the amount and sources of heterogeneity, an explanation of the ways to address heterogeneity, and appropriate theoretical knowledge and training. Performing a meta-analysis does not simply mean placing numbers in software to obtain pooled effects. We do not recommend combining different designs to obtain a pooled effect estimate; instead, we recommend stratifying according to study design. For example, when authors include both RCTs and observational studies, it is good practice to calculate their effects separately. Observational studies evaluating interventions have the problem that these studies are not comparable, and the effect of the intervention can be explained by other variables or predictors of the outcome of interest. Observational studies have different sources of bias than RCTs, which requires that different tools be used to assess risk of bias. Outcomes of individual studies are not completely reported by authors. Therefore, systematic review researchers should attempt to obtain them directly from authors or from online sources, such as public Websites (eg, www.yoda.yale.edu, www.clinicalstudydatarequest.com) or through partnering initiatives between universities and pharma companies (https://dcri.org/our-work/analytics-and-data-science/data-sharing/soar-data/). In most cases, only aggregated data are available for meta-analyses. This situation allows for the calculation of unadjusted intervention effects. The exceptions are meta-analyses with adjusted intervention effects, in which adjusters are the same or very similar to those in the included studies. The best-case scenario is having IPD, wherein several extra analyses are possible, including adjustment for trials and confounders, regression analyses, or subgroup analyses with adjustment. IPD meta-analyses should be preplanned, with permits from sponsors negotiated in advance. The first step in meta-analysis is to define the type of outcome(s), usually dichotomous or continuous. The most common dichotomous outcomes are ORs, relative risks, and hazard ratios, as well as the absolute risk difference and its inverse (ie, the number-needed-to-treat). The most common continuous outcomes are mean differences and standardized mean differences; the latter is used when different scales or units are used among studies. Second, authors should define the model as either fixed effects or random effects. The latter is the most commonly performed and assumes that there is a variability of effects across studies; the task is to calculate the average of those effects. The former is used in specific situations, such as when there is little heterogeneity of effects or when outcomes are rare. The third step is to define the method to calculate effect measures and their CIs. This depends on the type of outcome, effect measure, and model (eg, Table 2 of Rao et al5Rao G. Lopez-Jimenez F. Boyd J. et al.Methodological standards for meta-analyses and qualitative systematic reviews of cardiac prevention and treatment studies: a scientific statement from the American Heart Association.Circulation. 2017; 136: e172-e194Crossref PubMed Scopus (110) Google Scholar). In most cases, the inverse variance method is used. The Mantel-Haenszel method is used when outcomes are dichotomous, events are rare, and there is imbalance in the number of individuals between trial arms. The fourth step is to choose the type of estimator for tau2Hernandez A.V. Perez-Lopez F.R. Piscoya A. et al.Comparative efficacy of bone anabolic therapies in women with postmenopausal osteoporosis: a systematic review and network meta-analysis of randomized controlled trials.Maturitas. 2019; 129: 12-22Abstract Full Text Full Text PDF PubMed Scopus (12) Google Scholar, which measures between study variance; the Paule-Mandel or Sidik-Jonkman estimators are preferred over the DerSimonian and Laird estimator, particularly when the number of studies included is small. Authors would also want to adjust CIs of intervention effects using the Hartung-Knapp method when the number of studies is small. Specific methods and effect measures are used in some types of meta-analyses. For example, meta-analyses of diagnostic studies calculate summary receiving-operating characteristic curves with 95% CIs, generally using the bivariate or hierarchical methods that account for the correlation between sensitivities and specificities. Prognostic meta-analyses calculate pooled c-statistics as measures of discrimination and observed/expected ratios as measures of calibration. Meta-analyses of rare events represent a challenge for researchers, especially when the number of studies is small, they are imbalanced, or have zero events in one or more arms. The Peto OR is a popular effect measure for this situation; however, alternative measures such as the Mantel-Haenszel or inverse variance can provide less biased estimates. The use of methods such as the treatment arm continuity correction can be used to address the absence of events in one or more arms. Heterogeneity of effects among studies (ie, statistical heterogeneity) is quantified in two ways: the first is a χ2 test that determines the presence or absence of heterogeneity, and the second is the inconsistency (I2) statistic that measures the level of heterogeneity (low, I2 < 30%; moderate, I2 30%-60%; and high, I2 > 60%).6Higgins J.P. Thompson S.G. Quantifying heterogeneity in a meta-analysis.Stat Med. 2002; 21: 1539-1558Crossref PubMed Scopus (18940) Google Scholar I2 statistics need to be reported with their 95% CIs. Funnel plots depict the relation between effect measures per study in the x-axis and the SE of the effect measure or the sample size per study in the y-axis. Small studies are more prone to bias and therefore may be unpublished with negative effects or published with positive effects. Small study effects can be statistically evaluated with the Harbord test of asymmetry of the funnel plot.7Sterne J.A. Sutton A.J. Ioannidis J.P. et al.Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials.BMJ. 2011; 343: d4002Crossref PubMed Scopus (3064) Google Scholar Funnel plot asymmetry should not be equated with publication bias, as there are other causes such as true heterogeneity among study results, poor methodologic quality, reporting biases, and chance. At least 10 studies should be available to run small study effects tests. Secondary analyses include cumulative meta-analyses, subgroup analyses, meta-regression analyses, and sensitivity analyses. Cumulative meta-analyses calculate pooled effects by adding data of a second study to one previously published, then adding data of a third study to the previous pooled effect of the first and second studies, and so on; figures show how adding one study at a time changes the effect estimates and whether the pooled effect is maintained over time or is diluted at some calendar year point. Subgroup analyses split studies according to baseline characteristics and evaluate whether effects between those subgroups are different by using the test-for-interaction tests. Subgroup analyses in meta-analyses also should be hypothesis-generating analyses only. Meta-regression analyses evaluate whether intervention effects are associated with baseline study characteristics or summary patient characteristics. Usually linear regression analyses are used, with the log of the intervention effect as the dependent variable. Sensitivity analyses evaluate secondary models (eg, fixed effects instead of random effects), methods (eg, Mantel-Haenszel instead of inverse variance), or effect measures (eg, absolute risk difference instead of relative risk); they also evaluate subsets of studies by excluding some that may explain high heterogeneity (eg, excluding studies with high risk of bias). In NMAs of RCTs, additional assumptions must be explained and addressed. Transitivity among studies is evaluated by subjectively assessing patient characteristics, interventions, controls, outcomes, and times of intervention/follow-up; the more similar they are, the more transitivity is fulfilled. Consistency between direct and indirect effects is evaluated with a test of disagreement for each intervention comparison and with the Cochran’s Q statistic for the overall network. Finally, the geometry of networks per outcome is evaluated regarding what treatments were involved, the number of studies with specific direct comparisons, and the number of patients and events per comparison. Multiple intervention comparisons can be described in league tables and ranked as best interventions by using surface under the cumulative ranking curve in Bayesian meta-analyses or by p-scores in frequentist meta-analyses. Reporting guidelines for systematic reviews and meta-analyses were developed by the Preferred Reporting Items for Systematic reviews and Meta-analysis (PRISMA) collaboration and are available at www.prisma-statement.org. The PRISMA statement for systematic reviews and meta-analyses was published in 20098Moher D. Liberati A. Tetzlaff J. et al.Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.Ann Intern Med. 2009; 151: 264-269Crossref PubMed Scopus (15188) Google Scholar and replaced the 1999 Quality of Reporting of Meta-analyses guidelines. Also, the extensions of the PRISMA statement are available on the Website (NMA in 2015, IPD in 2015, harms in 2016, and diagnostic test accuracy in 2018). There are no reporting guidelines for systematic reviews and meta-analyses of prognostic factors or predictive models. Systematic reviews and meta-analyses should be reported according to PRISMA guidelines. Authors should describe their adherence to these guidelines in their manuscript, and a PRISMA checklist (www.prisma-statement.org/PRISMAStatement/Checklist) or a PRISMA extension checklist should be submitted as a supplemental file along with other manuscript files. The 2019 Cochrane Handbook is the most complete resource to guide authors performing systematic reviews and meta-analyses (https://training.cochrane.org/handbook/current). The RevMan 5.3 software of the Cochrane collaboration is also freely available: (https://community.cochrane.org/help/tools-and-software/revman-5/revman-5-download); this software is useful for analyses in non-Cochrane reviews, and for protocol development, analyses, and reporting/manuscript writing in Cochrane reviews. RevMan 5.3 can handle analyses of basic or intermediate difficulty. GRADE methods for quality of evidence assessment are available at www.gradeworkinggroup.org, and the free online GRADEpro GDT software to create Summary of Findings tables is available at https://community.cochrane.org/help/tools-and-software/gradepro-gdt. GRADEpro and RevMan 5.3 can share figures and tables. Statistical software such as R (www.r-project.org) and Stata (www.stata.com) are necessary for more advanced analyses, such as NMA, cumulative meta-analyses, diagnostic test accuracy meta-analyses or prognostic meta-analyses, use of specific tau2Hernandez A.V. Perez-Lopez F.R. Piscoya A. et al.Comparative efficacy of bone anabolic therapies in women with postmenopausal osteoporosis: a systematic review and network meta-analysis of randomized controlled trials.Maturitas. 2019; 129: 12-22Abstract Full Text Full Text PDF PubMed Scopus (12) Google Scholar estimators, or adjustment of CIs. 1.Do you have a clear research question?2.Was there a systematic review prior to performing meta-analyses?3.Were your searches performed using several search engines and were search strategies published in the supplement?4.Were inclusion and exclusion criteria for studies clearly described?5.Were selection of studies, extraction of data, and risk of bias assessment performed independently by at least two researchers and discrepancies resolved by discussion or with another researcher?6.Did you assess the quality of the evidence per outcome using GRADE methodology?7.Are you following one of the existing reporting guidelines for your work?8.Did you identify your study as meta-analysis in the title?9.Did you present a flowchart of study selection, describe selected studies, and describe risk of bias assessment of selected studies?10.Were methods for selecting effect measures, models and methods for meta-analyses, risk of bias assessment tools, heterogeneity statistics, small study effects tests, and subgroup and sensitivity analyses clearly described in the methods section?11.Did you present effects of interventions as forest plots, network figures, and/or tables or league tables?12.Did you summarize your main findings, highlight previous similar studies, highlight your new findings, describe the limitations of your work, and describe conclusions in the Discussion section of your study? When reviewing a meta-analysis, consider commenting on the following:1.Clinical variables and outcomes. Were the clinical variables and outcomes well described and appropriate for the research question? Was the potential for heterogeneity in the definitions and measurements of the clinical variables and outcomes assessed?2.The selection of studies included in the analysis. Was a comprehensive search strategy clearly outlined? Were multiple specific search engines used? Were appropriate inclusion and exclusion criteria applied? Was a flowchart of study selection presented? Was the risk of publication bias assessed?3.The analysis and interpretation of the findings. Was heterogeneity of the included studies evaluated and reported? Was the quality of the evidence assessed and reported (eg, with GRADE methodology)? Was a sensitivity analysis performed? Were forest plots provided? Were limitations described? Was the interpretation of the findings reasonable? 1.Clinical variables and outcomes. Were the clinical variables and outcomes well described and appropriate for the research question? Was the potential for heterogeneity in the definitions and measurements of the clinical variables and outcomes assessed?2.The selection of studies included in the analysis. Was a comprehensive search strategy clearly outlined? Were multiple specific search engines used? Were appropriate inclusion and exclusion criteria applied? Was a flowchart of study selection presented? Was the risk of publication bias assessed?3.The analysis and interpretation of the findings. Was heterogeneity of the included studies evaluated and reported? Was the quality of the evidence assessed and reported (eg, with GRADE methodology)? Was a sensitivity analysis performed? Were forest plots provided? Were limitations described? Was the interpretation of the findings reasonable? Financial/nonfinancial disclosures: None declared.