Introduction to research: Meta-analysis

What’s the idea?

Meta-analysisA quantitative study design used to systematically assess th... More is a way of combining results from a range of different studies in one particular area.

What does it mean?

A meta-analysis collects together a range of quantitative studies around a particular intervention. It converts the results of each study into a common metric – effect size, for example – and combines them so it can estimate the overall impact of the intervention.

There is a further level too. A meta-meta-analysis combines the results from various meta-analyses to rank the intervention/approach/strategy that has the most impact (Simpson, 2017).

What are the implications for teachers?

• Meta-analyses review the research and combine some of the available papers so you don’t have to read them all separately
• Meta-analyses use explicit, accountable, rigorous research methods. They potentially provide a more objective review of the research compared with other review methods
• Meta-analyses combine data from various independent studies, which should improve their statistical precision
• Meta-analysis can help identify inconsistencies in the research and explain any differences among research findings in a given area.

Wiliam (2016) identifies a number of limitations of meta-analyses:

• Intensity and duration of the intervention. Pupils in different studies will experience an intervention for different periods of time and at different levels of intensity. This makes it really difficult to accurately interpret the average effect size of an intervention
• Publication bias and file-drawer effect. Studies producing statistically significant results are more likely to be published. This means the importance of some interventions could be over-stated because you are more likely to only see research published with statistically significant results
• Age and effect size. Results of studies involving older pupils tend to be more varied compared with those involving younger students, so they tend to have smaller effect sizes. As such, research about interventions that involve younger pupils are more likely to be considered effective
• Outcome measures. Classroom test results are more affected by teacher instruction compared with national test results. This means that any analysis of interventions that have been measured using classrooms tests are likely to have a larger effect size
• Generalisability of the studies. Research is often carried out on undergraduate students. This is not representative of the pupil population and could distort how effective the intervention is likely to be for a school teacher with a group of pupils.

Top Tips

• Effect sizes, meta-analyses and meta-meta-analyses are all pieces of the evidence jigsaw. They are not uncontested, and you need to be wary of simplistic interpretations and recommendations.
• Use your professional judgement and consider the challenge you face: adopting an intervention with a large effect-size makes little sense, if it will not address the problem you need it to.

Want to know more?

• Coe R, Waring M, Hedges L et al. (2017) Research Methods and Methodologies in Education (2nd ed). London: SAGE.
• Simpson A (2017) The Misdirection of Public Policy: Comparing and Combining Standardised Effect Sizes. Journal of Education Policy 32(4): 450-466.
• Wiliam D (2016) Leadership for Teacher Learning. West Palm Beach: Learning Sciences International.