Leading without limits: The role of school culture in implementing evidence-based practices

According to its last annual report, half of all senior leaders in England consult the Education Endowment Foundation’s Teaching and Learning Toolkit to inform decision-making (EEF, 2018a). In just seven years, the Toolkit has become central to – and indeed driven (Coldwell et al., 2017) – evidence-based practice in schools here and overseas.

For an increasing number of school leaders, the initial answer to the question ‘what works?’ is ‘ask the Toolkit’. The thematic presentation of areas of research and practice, which can be ranked according to the strength of the evidence (EEF, 2018b), the additional months’ progress and indicative cost, offer ‘best bets’, based on what has and what has not worked. Importantly, it cannot provide guarantees that any given method will work.

The EEF’s inception coincided with the launch of the Pupil PremiumAdditional funding for publicly funded schools in England to in 2011. The Toolkit was promoted as a practical and independent way of providing schools with empirical evidence to inform how they invested their Pupil Premium. Schools must publicly declare how they spend their Premium via a statement on their website. This creates an audit trail, where bodies internal and external to the school can assess the extent to which Pupil Premium spending has improved outcomes for disadvantaged pupils.

The Toolkit’s assessment of particular approaches or specific (usually commercially available) intervention programmes is particularly helpful for the time-poor school leader, sat at her laptop trying to figure out how best to spend precious funds. If, for instance, she needs to improve reading in Key Stage 2, the Toolkit’s links to the EEF’s ‘Promising Projects’ page will indicate approaches and programmes found to be effective elsewhere, and under what circumstances. She will be conscious, too, of creating an audit trail to satisfy governors, OfstedThe Office for Standards in Education, Children’s Services and other scrutineers of the cost-effectiveness of her decision-making.

Evidence, then, is potentially very valuable for schools, but it needs to be used intelligently. Our accountability culture can have an unintended distorting effect on the way in which school leaders engage with research. The promise of evidence-informed practices to resolve persistent problems raises expectations, but it still requires a high level of critical engagement on behalf of schools if it is to be successful in terms of raising standards.

Evidence: limitations and context

A large part of the Toolkit’s success is attributable to its accessibility and concision, but presenting complex research findings in this way involves a trade-off. Some researchers may be a little nervy about the nuance being stripped from their work, and concerned that the Toolkit oversimplifies matters. However, with a few clicks on the EEF website, one can easily access the underlying data and methodology. Despite this transparency, school leaders under pressure to narrow the attainment gap are at risk of making cost-benefit judgments on the basis of a face-value encounter with the Toolkit. School leaders need not only to invest time in deepening their understanding of evidence, but also to recognise its limitations and, as importantly, consider evidence in the light of their own unique context.

Second to school leaders’ question of ‘what works’ is ‘how much bang can I get for my buck?’ Half of all senior leaders might use the Toolkit, but we do not know what proportion use it intelligently – for example, to select a proven approach or programme to help improve the chances of successfully addressing a specific learning need for a particular group of pupils. Sometimes, the imperative to show impact predominates, and school leaders ask these questions in the opposite order. Reacting to, for example, a poor set of SATs results or an unfavourable Ofsted grading distorts the type of engagement with, and application of, research evidence that the EEF encourages, and so increases the likelihood of poor decision-making.

Superficial engagement with the Toolkit is also a risk in cases where schools abandon particular approaches. For instance, school leaders regularly tell me of counterparts in nearby schools who ‘got rid of all their teaching assistants’ after viewing the Toolkit’s impact summary on TAs as ‘high cost, low impact’. Cutting TAs is at variance to guidance based on empirical research that says schools should instead be making better deployment decisions about TAs – not getting rid of them (Webster et al., 2016).

To be fair to the EEF, it is aware of these risks (EEF, 2017). Furthermore, it is not just the EEF Toolkit that might prompt this sort of behaviour among school leaders; the presentational style of John Hattie’s Visible Learning (2008) can provoke similar thinking. This is not a fault of the Toolkit or any other such resources. The risk lies in a combination of a lack of curiosity in the minds of school leaders, a failure to relate evidence to their own context, and the accountability and financial pressures that constrict the time and space for this thinking.

But there is a further possible unintended consequence of atomising educational approaches and innovations, and pitching them against one another in terms of impact. School leaders can lose sight of how two or more interventions interact with one another, and how one can amplify or moderate the effect of others. For example, take two of the Toolkit’s most popular strands: metacognition and self-regulation (low cost, high impact) and teaching assistants (high cost, low impact). We know that poor deployment of TAs can foster dependence and impede the development of pupils’ independence skills; yet, trained and deployed more thoughtfully, TAs have the potential to support efforts to improve pupils’ metacognitive traits and ability to manage their own learning (Webster et al., 2016).

The notion of the school or the classroom as an ecosystem has got somewhat lost in the ‘what works’ narrative. Much of this is prefigured in Cartwright’s (2011) argument that a similar situation has occurred in medicine. Educational interventions and approaches do not occur in isolation; they are interconnecting parts of an overall teaching and learning experience. Secondary schools that use setting, for example, reduce class sizes for lower-attaining groups, and deploy TAs in these classes; whereas classes for middle-attainers and higher-attainers are comparatively larger and do not contain TAs (Blatchford and Webster, 2018). Each of these structural factors (setting, class size and TAs) has its own discrete evidence base behind it, yet we know little of how they interact, and how they might be composed to optimise teaching and learning.

This is pertinent in the case of setting, class size and TAs (all Toolkit strands), because almost all schools adopt these approaches to some extent, but the evidence shows that their respective impacts on learning are disproportionately worse for disadvantaged pupils. When combined, these effects could be magnified. For example, experimental studies in the USA (Finn et al., 2000) and the UK (Blatchford et al., 2004) have found no differences in the outcomes for pupils in classes with TAs present; while research in the UK has found a negative effect of support from TAs on pupil learning, irrespective of class size, and that effect is greatest for lower-attaining pupils and those with special educational needs (Webster et al., 2010). The critically minded school leader might therefore reasonably conclude, perhaps counterintuitively, that low class size plus the presence of an additional adult (as a further class-size reduction measure) is more harmful for pupils in disadvantaged groups than just reducing the raw number of pupils in the room with only the teacher.

Putting research evidence into action

It is this approach to considering the interactions between different inputs that prompted Leading Without Limits (LWL). LWL is a professional learning programme for school leaders, which uses exposure to high-quality research as the basis for a forensic exploration of how evidence-based approaches can be implemented and actualised in schools and classrooms.

LWL is a partnership between Rosendale Research School and UCL Institute of Education, and is running over the 2018/19 school year. It explores key strategic areas of school organisation and pedagogy, including ‘ability’ labelling, setting, grouping and metacognition. It addresses head-on some of the most persistent and problematic structures that feature on the Toolkit, but for which there are few or no commercially available programmes. While each session majors on a particular theme, each one (for example, in-class grouping) is explored in the context of learning from previous sessions (in this case, the effects of ‘ability’ labelling and setting/streaming).

Another justification for LWL is that the operationalisation of evidence-based practice is often the under-discussed side of making ‘what works’ work. The principles and practices of putting research evidence into action (Sharples et al., 2018) inform and infuse LWL’s coverage. But engaging with evidence meaningfully also requires a supportive culture. The first LWL session explored an essential principle of effective implementation: creating an environment and school culture within which new ways of doing things can take root and flourish. Choosing this as a starting point was a deliberate strategy, informed by the experiences of the LWL leads (Marc Rowland from Rosendale Research School and myself) of working directly with hundreds of schools. We regard school culture and leadership as strong determinants of how effectively and how widely evidence-based approaches are adopted and embedded, and thus how impactful they are.

A good indicator of whether a healthy implementation culture exists is the extent to which a school lives its values. Values underpin culture, and establishing a set of guiding values is one important way in which school leaders set the weather. Visitors may be greeted at reception by a colourful display showing the school’s values; or they may be spelled out in huge letters around the site; or pinned-up in every classroom as a clever acrostic. But, as Mary Myatt (2017) says, values are ‘truly lived, not just laminated’.

When causation is not fully provable, researchers explore the full social and educational panorama to determine plausible reasons for correlations that their analyses may reveal. Leading Without Limits is an attempt to encourage and equip school leaders to apply this thinking to their evidence-based practice and decision-making. Curiosity about discrete areas of evidence is important, and the EEF and the Toolkit can take a lot of credit for providing school leaders with the means and impetus to initiate these kinds of discussions. But it is when connections are made between areas of research, when practitioners build an understanding of evidence in context, that they enhance the power and potential impact of their individual and overall decision-making.

The success of the evidence-into-practice movement, however, begins back in school, with leaders recognising the importance and influence of core values on school culture in informing and facilitating a positive and productive learning environment for staff and pupils.

References

Blatchford P, Russell A, Bassett P et al. (2004) The Role and Effects of Teaching Assistants in English Primary Schools (Years 4 to 6) 2000–2003: Results from the Class Size and Pupil–Adult Ratios Project. Final Report. London: DfES. Available at: https://www.education.gov.uk/publications/eOrderingDownload/RR605.pdf (accessed 8 January 2019).

Blatchford P and Webster R (2018) Classroom contexts for learning at primary and secondary school: Class size, groupings, interactions and special educational needs. British Educational Research Journal 44(4): 681–703.

Cartwright N (2011) A philosopher’s view of the long road from RCTs to effectiveness. The Lancet 377(9775): 1400–1401. Available at: https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60563-1/fulltext (accessed 8 January 2019).

Coldwell M, Greany T, Higgins S et al. (2017) Evidence-Informed Teaching: An Evaluation of Progress in England. Research Report. London: DfEDepartment for Education - a ministerial department responsi. Available at: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/625007/Evidence-informed_teaching_-_an_evaluation_of_progress_in_England.pdf (accessed 8 January 2019).

Education Endowment Foundation (2017) The Teaching and Learning Toolkit: What are the risks and how do we address them? Available at: https://educationendowmentfoundation.org.uk/news/the-teaching-and-learning-toolkit-what-are-the-risks-and-how-do-we-address/ (accessed 8 January 2019).

Education Endowment Foundation (2018a) Annual report 2018. Available at: https://educationendowmentfoundation.org.uk/public/files/Annual_Reports/EEF_-_2018_Annual_Report_print.pdf (accessed 8 January 2019).

Education Endowment Foundation (2018b) Evidence strength. The EEF ‘padlock’ rating. Available at: https://educationendowmentfoundation.org.uk/evidence-summaries/about-the-toolkits/evidence-strength/ (accessed 8 January 2019).

Finn JD, Gerber SB, Farber SL et al. (2000) Teacher aides: An alternative to small classes? In: Wang MC and Finn JD (eds) How Small Classes Help Teachers Do Their Best. Philadelphia, PA: Temple University Center for Research in Human Development, pp. 131–174.

Hattie J (2008) Visible Learning. Oxon: Routledge.

Myatt M (2017) Walking the talk. In: Mary Myatt. Available at: http://www.marymyatt.com/blog/2017-12-09/walking-the-talk (accessed 8 January 2019).

Sharples J, Albers J and Fraser S (2018) Putting evidence to work: A school’s guide to implementation guidance report. Available at: https://educationendowmentfoundation.org.uk/tools/guidance-reports/a-schools-guide-to-implementation/ (accessed 8 January 2019).

Webster R, Blatchford P, Bassett P et al. (2010) Double standards and first principles: Framing teaching assistantAn adult that assists a teacher in the classroom support for pupils with special educational needs. European Journal of Special Needs Education 25(4): 319–336.

Webster R, Russell A and Blatchford P (2016) Maximising the Impact of Teaching Assistants: Guidance for School Leaders and Teachers (2nd edition). Oxon: Routledge.