Â鶹ÒùÔº

England’s inspection system has always been fraught with challenges, with many of those challenges related to the reliability of the inspection process in relation to the stakes. In the new framework, a one-word overall judgment has been replaced by a ‘report card’ system and Ofsted has initiated other changes to the process. Based on our (April 2022 to March 2024), we argue that school inspections should give reliable insights regarding critical quality aspects of a school, and the process of coming to such insights needs to be transparent. To our knowledge, our project is still the only one that looked at Ofsted inspection at scale with quantitative and computational methods.

A significant concern regarding inspections is consistency. Respondents to the ‘’ consultation observed variable approaches among inspectors, deemed judgments subjective and noted that inspection outcomes varied depending on which inspector came to the school.

Consistency and reliability in school inspection

Findings from the Big Listen concerning inspection consistency are in line with our research. In  we used data from more than 30,000 school inspections conducted in England between 2011 and 2019 to see how consistent inspections were (Bokhove et al., 2023). We found that permanent Ofsted inspectors were harsher in their judgments than those who inspected schools on a freelance basis. We also found that male inspectors awarded slightly more lenient judgments to primary schools than their female counterparts. The reliability of the inspection process needs to be paramount.

In , we found that schools that have a member of staff working for Ofsted received better inspection outcomes than schools without an inspector on their payroll (Bokhove et al., 2024a). One thing we suggested in that paper is that other schools may benefit from having access to the training material and professional development opportunities that Ofsted provides, which is something Ofsted has now acted upon by making more materials available.

Finally, using more than 60,000 inspection reports for primary and secondary schools in England between 1997 and 2022, we . Themes in the inspection reports did not change much if the policy change was limited, suggesting that for inspection practices to really be reformed, structural policy changes were needed (Bokhove et al., 2024b). This has now happened with the new framework, but it remains to be seen if the change in framework supports increased reliability of inspections.

‘Under the new framework, a more fine-grained report card might provide more nuance for a school, but it also might complicate a quest for more reliability.’

The introduction of the new framework has already led to several changes to the inspection process. Ofsted has announced that it will deploy senior inspectors to ‘shadow’ inspection teams to scrutinise the consistency of judgments in the new report cards system. Under the new framework, a more fine-grained report card might provide more nuance for a school, but it also might complicate a quest for more reliability, as disagreements on each of the criteria could happen. Some evaluation areas will be measured with greater reliability than others, for example those that draw on larger samples of achievement and attendance data. Reliability is also related to the stakes of the inspection: if inspection outcomes continue to have large consequences for a school, we must be very certain about the judgment.

Recommendations

We think the reliability across inspection reports can be improved in three ways:

1. Ofsted should clearly explain what they consider to be an adequate level of reliability for each of the criteria in their new framework, especially those criteria that are considered to have higher stakes associated with them.
2. Ofsted should commit to an independent and transparent programme of research to test the reliability of inspection. This should include transparently making available data outside of the organisation, including process data for inspections. It is our understanding that new consistency work by Ofsted is to appear in due course.
3. Crucially, if it turns out that inspection grades are found not to meet standards for reliability, Ofsted should be prepared to make changes to the inspection processes.

To our knowledge, Ofsted has committed to the first two points, and should be prepared to enact on the third, if necessary, to ensure that inspections are reliable and transparent. Only in that way can Ofsted build public confidence.

References

Bokhove, C., Jerrim, J., & Sims, S. (2023). Are some school inspectors more lenient than others?. School Effectiveness and School Improvement, 34(4), 419–441.

Bokhove, C., Jerrim, J., & Sims, S. (2024a). Do schools that employ an inspector get better inspection grades?. British Educational Research Journal, 50(5), 2383–2406.

Bokhove, C., Jerrim, J., & Sims, S. (2024b). Inspecting the inspectorate: New insights into Ofsted inspections.