We use cookies to give you the best experience possible. By continuing we鈥檒l assume you鈥檙e on board with our cookie policy

Multiple Effect Sizes

The whole doc is available only for registered users
  • Pages: 4
  • Word count: 957
  • Category:

A limited time offer! Get a custom sample essay written according to your requirements urgent 3h delivery guaranteed

Order Now


Main and Moderator studies

For our reviews, we are faced with the issue that multiple effect sizes are reported from the same sample of participants (e.g., due to various outcome variables such as reaction time and error rates or due to multiple contrasts stated on the same outcome variable). This introduces statistical dependencies between effect sizes from the same study which violates the independence assumption of classical meta-analysis (e.g., Hedges & Vevea, 1998). Ignoring these dependencies can lead to unacceptably low coverage probabilities and increased type-1-error rates for primary and moderator analyses, respectively (Van den Noortgate, L贸pez-L贸pez, Mar铆n-Mart铆nez, S谩nchez-Meca, 2013; L贸pez-L贸pez, Van den Noortgate, Tanner-Smith, Wilson, & Lipsey, 2017, respectively).

One frequently used approach to handle this issue is the strategy of averaging effect sizes for each independent sample. While this results in independent effect sizes, it leads to underestimations of between-study heterogeneity (Cheung & Chan, 2004) and is associated with reduced statistical power (L贸pez-L贸pez, Page, Lipsey, & Higgins, 2018). More importantly, our meta-analysis can be characterized as having a 鈥渄ivergent鈥 framework (i.e., being focused on effect size variation and its explanation). For this review framework, it is recommended to include all relevant effect sizes instead of averaging on a sample level (L贸pez-L贸pez et al., 2018).

Therefore, we will use an analysis strategy that allows for the inclusion of multiple effect sizes from the same sample by modeling the arising statistical dependencies: Three-level meta-analysis (Van den Noortgate et al., 2013). Three-level meta-analyses can include an intermediate level to represent outcomes within studies. Specifically, classical meta-analysis has two levels (and sources of variation): level 1: participants (sampling error) and level 2: studies (difference in effect size between studies). Three-level meta-analysis with multiple effect sizes per sample has the following three levels (sources of variation): level 1: participants (sampling error), level 2: outcomes (difference in effect size between outcomes within studies), and level 3: studies (variation in effect size between studies). In contrast to traditional meta-analyses, three-level models result in two heterogeneity estimates, i.e., heterogeneity between outcomes within studies (level 2) and heterogeneity between studies (level 3).

Three-level-meta-analysis has been shown to perform well in the case of multiple effect sizes reported per sample (Van den Noortgate et al., 2013). Similarly, it has been shown to have favorable properties for moderator analyses (L贸pez-L贸pez et al., 2017). Specifically, three-level models have an acceptable type-1-error rate for level 3 (study-level) and level 2 (outcome-level) moderator variables if there is a moderate ( k > 20) and small ( k >10) number of studies, respectively. Regarding statistical power, the method consistently outperformed alternative (i.e., robust) meta-analysis methods (L贸pez-L贸pez et al., 2017).

All analyses will be conducted using the open source statistics software R (R Core Team, 2018). Three-level meta-analysis models will be implemented using the meta3 function from the metaSEM package (Cheung, 2015). The overall mean effect size and its 95%-confidence interval will be assessed as the intercept of a three-level model including all coded effect sizes with the sample as a clustering variable. The presence of heterogeneity will be determined using an overall Q-statistic for the homogeneity of effect sizes. It will be further interpreted utilizing the variance estimates (蟿^2) and their statistical significance on level 2 (between outcomes within studies) and level 3 (between studies), respectively.

Moderator analyses will be performed by entering relevant moderators as predictors in the three-level model. Statistical significance of individual moderators will be assessed using a likelihood-ratio test by comparison to the same model without the respective moderator. This will be achieved using the anova function in R. The interpretation of moderator effects, if significant, will be based on their linear coefficients in the three-level model.

A more detailed analytic strategy, the concrete pre-planned models, and the alpha-level will also be provided in an addendum to this pre-registration before the coding of relevant studies.

Publication bias

Given the potential exaggeration of mean effect sizes by publication bias (Greenwald, 1975; Rothstein, Sutton, & Borenstein, 2006), meta-analyses should attempt to correct for it (see also Carter, Sch枚nbrodt, Gervais, & Hilgard, 2017). One frequently method for this is trim and fill (Duval, Tweedie, 2000), but its performance has been shown to be inadequate (Moreno et al., 2009; Carter, Sch枚nbrodt, Gervais, & Hilgard, 2017). More promising methods are the meta-regression PET-PEESE (Stanley, Doucouliagos, 2014) and especially the three-parameter selection model (Iyengar & Greenhouse, 1988; Vevea & Hedges, 1995). Carter et al. (2017) provide a simulation study on the effectiveness of these methods in the classical meta-analysis.

In the case of our meta-analysis, publication bias correction is complicated by the multi-level nature of our data. It will be assessed by entering publication status (published vs. unpublished) as a moderator variable in our three-level model. While some methods to simultaneously evaluate and correct for publication bias (e.g., PET-PEESE) could be integrated into a three-level model, others (e.g., the three-parameter selection model) would require independent effect sizes. The real analytic strategy to assess and correct for publication bias will be provided in an addendum to this pre-registration before data analysis.

Sensitivity analyses

The robustness of our results will be assessed by assessing the presence and if necessary after removing outliers (e.g., Viechtbauer & Cheung, 2010). The real analytic strategy to detect and remove outliers from the main-analyses and the moderator-analyses will be provided in an addendum to this pre-registration before data analysis.

Confidence in cumulative evidence

The strength of the increasing evidence for STIs will be decided based on the overall mean effect size before as well as after correction for publication bias and exclusion of outliers. Similarly, the strength of the evidence for specific moderator variables will be decided based on their robustness in the sensitivity analyses. The quality of studies will not be coded and analyzed since this is less straightforward in social psychology compared to, e.g., clinical trials (e.g., Berkman et al., 2014).


Related Topics

We can write a custom essay

According to Your Specific Requirements

Order an essay
Materials Daily
100,000+ Subjects
2000+ Topics
Free Plagiarism
All Materials
are Cataloged Well

Sorry, but copying text is forbidden on this website. If you need this or any other sample, we can send it to you via email.

By clicking "SEND", you agree to our terms of service and privacy policy. We'll occasionally send you account related and promo emails.
Sorry, but only registered users have full access

How about getting this access

Your Answer Is Very Helpful For Us
Thank You A Lot!


Emma Taylor


Hi there!
Would you like to get such a paper?
How about getting a customized one?

Can't find What you were Looking for?

Get access to our huge, continuously updated knowledge base

The next update will be in:
14 : 59 : 59