Multiple Effect Sizes

Analysis

Main and Moderator studies

For our reviews, we are faced with the issue that multiple effect sizes are reported from the same sample of participants (e.g., due to various outcome variables such as reaction time and error rates or due to multiple contrasts stated on the same outcome variable). This introduces statistical dependencies between effect sizes from the same study which violates the independence assumption of classical meta-analysis (e.g., Hedges & Vevea, 1998). Ignoring these dependencies can lead to unacceptably low coverage probabilities and increased type-1-error rates for primary and moderator analyses, respectively (Van den Noortgate, López-López, Marín-Martínez, Sánchez-Meca, 2013; López-López, Van den Noortgate, Tanner-Smith, Wilson, & Lipsey, 2017, respectively).

One frequently used approach to handle this issue is the strategy of averaging effect sizes for each independent sample. While this results in independent effect sizes, it leads to underestimations of between-study heterogeneity (Cheung & Chan, 2004) and is associated with reduced statistical power (López-López, Page, Lipsey, & Higgins, 2018). More importantly, our meta-analysis can be characterized as having a “divergent” framework (i.e., being focused on effect size variation and its explanation). For this review framework, it is recommended to include all relevant effect sizes instead of averaging on a sample level (López-López et al., 2018).

Therefore, we will use an analysis strategy that allows for the inclusion of multiple effect sizes from the same sample by modeling the arising statistical dependencies: Three-level meta-analysis (Van den Noortgate et al., 2013). Three-level meta-analyses can include an intermediate level to represent outcomes within studies. Specifically, classical meta-analysis has two levels (and sources of variation): level 1: participants (sampling error) and level 2: studies (difference in effect size between studies). Three-level meta-analysis with multiple effect sizes per sample has the following three levels (sources of variation): level 1: participants (sampling error), level 2: outcomes (difference in effect size between outcomes within studies), and level 3: studies (variation in effect size between studies). In contrast to traditional meta-analyses, three-level models result in two heterogeneity estimates, i.e., heterogeneity between outcomes within studies (level 2) and heterogeneity between studies (level 3).

Three-level-meta-analysis has been shown to perform well in the case of multiple effect sizes reported per sample (Van den Noortgate et al., 2013). Similarly, it has been shown to have favorable properties for moderator analyses (López-López et al., 2017). Specifically, three-level models have an acceptable type-1-error rate for level 3 (study-level) and level 2 (outcome-level) moderator variables if there is a moderate ( k > 20) and small ( k >10) number of studies, respectively. Regarding statistical power, the method consistently outperformed alternative (i.e., robust) meta-analysis methods (López-López et al., 2017).

All analyses will be conducted using the open source statistics software R (R Core Team, 2018). Three-level meta-analysis models will be implemented using the meta3 function from the metaSEM package (Cheung, 2015). The overall mean effect size and its 95%-confidence interval will be assessed as the intercept of a three-level model including all coded effect sizes with the sample as a clustering variable. The presence of heterogeneity will be determined using an overall Q-statistic for the homogeneity of effect sizes. It will be further interpreted utilizing the variance estimates (τ^2) and their statistical significance on level 2 (between outcomes within studies) and level 3 (between studies), respectively.

Moderator analyses will be performed by entering relevant moderators as predictors in the three-level model. Statistical significance of individual moderators will be assessed using a likelihood-ratio test by comparison to the same model without the respective moderator. This will be achieved using the anova function in R. The interpretation of moderator effects, if significant, will be based on their linear coefficients in the three-level model.

A more detailed analytic strategy, the concrete pre-planned models, and the alpha-level will also be provided in an addendum to this pre-registration before the coding of relevant studies.

Publication bias

Given the potential exaggeration of mean effect sizes by publication bias (Greenwald, 1975; Rothstein, Sutton, & Borenstein, 2006), meta-analyses should attempt to correct for it (see also Carter, Schönbrodt, Gervais, & Hilgard, 2017). One frequently method for this is trim and fill (Duval, Tweedie, 2000), but its performance has been shown to be inadequate (Moreno et al., 2009; Carter, Schönbrodt, Gervais, & Hilgard, 2017). More promising methods are the meta-regression PET-PEESE (Stanley, Doucouliagos, 2014) and especially the three-parameter selection model (Iyengar & Greenhouse, 1988; Vevea & Hedges, 1995). Carter et al. (2017) provide a simulation study on the effectiveness of these methods in the classical meta-analysis.

In the case of our meta-analysis, publication bias correction is complicated by the multi-level nature of our data. It will be assessed by entering publication status (published vs. unpublished) as a moderator variable in our three-level model. While some methods to simultaneously evaluate and correct for publication bias (e.g., PET-PEESE) could be integrated into a three-level model, others (e.g., the three-parameter selection model) would require independent effect sizes. The real analytic strategy to assess and correct for publication bias will be provided in an addendum to this pre-registration before data analysis.

Sensitivity analyses

The robustness of our results will be assessed by assessing the presence and if necessary after removing outliers (e.g., Viechtbauer & Cheung, 2010). The real analytic strategy to detect and remove outliers from the main-analyses and the moderator-analyses will be provided in an addendum to this pre-registration before data analysis.

Confidence in cumulative evidence

The strength of the increasing evidence for STIs will be decided based on the overall mean effect size before as well as after correction for publication bias and exclusion of outliers. Similarly, the strength of the evidence for specific moderator variables will be decided based on their robustness in the sensitivity analyses. The quality of studies will not be coded and analyzed since this is less straightforward in social psychology compared to, e.g., clinical trials (e.g., Berkman et al., 2014).

References: