Institute for Transport Studies (ITS)

Research Impact Case Study

International benchmarking and econometric analysis used to set efficiency targets for Network Rail

Overview

Reductions in railway infrastructure and operating costs, through efficiency gains, deliver benefits to taxpayers (via lower subsidies) and/or passengers (via lower fares). Research undertaken by the Institute from 2005 onwards revealed a 37% efficiency gap in relation to rail infrastructure costs and operations, relative to international best practice. The key impact of this research was to inform the Office of Rail Regulation’s (ORR) setting, in 2008, of annual efficiency targets for Network Rail for the subsequent five-year period, resulting in a reduction in costs from £18.2bn to £15.8bn over the five year regulatory ‘control period’ starting 2009/10. A secondary impact of the ITS research was to provide key benchmarking and evidence in more recent ORR efficiency assessments (2010) and Sir Roy McNulty’s long-term policy-setting Rail Value for Money (VfM) study (2011). Extending the reach of these research impacts, the water and sewerage regulator OFWAT has, from 2013, adopted the ITS approach for its latest periodic review. 

Core Research Team

Professor Chris Nash, Dr Andrew Smith, Dr Phill Wheat. 

External Corroborators

Office of Rail Regulation (ORR), Department for Transport (DfT)

Underpinning research

A key responsibility of the Office of Rail Regulation (ORR) is to determine the efficient cost of operating, maintaining, renewing and developing the British rail infrastructure. The determination of these costs, referred to as the periodic review, occurs every five years and, given the structure of the rail industry in Britain, effectively amounts to a review of the costs incurred by Network Rail. In contrast to other regulated utilities, previous periodic reviews (prior to 2008) did not assess the efficiency of Network Rail against external benchmarks. Recognising this gap in knowledge, ITS was contracted in 2005 by ORR to develop two international benchmarking studies.

Econometric modelling – efficiency studies

Between 2005 and 2008, researchers from ITS carried out new research funded by the ORR to investigate the cost and efficiency of the rail system in Britain relative to its international peers. Employing advanced time-varying efficiency modelling techniques, Dr Andrew Smith identified drivers of rail infrastructure costs, and the relative efficiency of rail infrastructure managers (IMs) around the world [1] using a unique panel dataset of 13 European rail infrastructure managers collected by the International Union of Railways. This work exposed a substantial efficiency gap between Network Rail and its international peers of around 37%. As part of this work, Smith and Wheat also adapted stochastic frontier techniques to deal with ‘sub-company’ datasets (where firms are sub-divided into several business units). This new approach allowed inefficiency to be decomposed into a fixed element that applies across the entire company (external inefficiency) and an element that varies at the sub-company level (internal inefficiency). The ‘sub-company’ analysis revealed large variations in performance and practice between regional units within Network Rail [2], as well as a substantial efficiency gap between Network Rail and its international peers. In parallel to the above research, Wheat and Smith developed a new international collaboration with Professor William Greene at NYU Stern School of Business to develop statistical methods for assessing the uncertainties associated with the estimates of inefficiency generated by stochastic frontier methods [3].

Econometric modelling – evaluation of policy reforms

The research referred to above is part of a larger body of rail cost research on whole train systems (i.e. train operations as well as rail infrastructure) conducted by ITS over many years, supported by funding from EPSRC, DfT and ORR. Since 1993, ITS researchers have carried out econometric and wider policy evaluation work to assess the impact of competitive tendering / franchising on costs. Introducing competition into passenger rail, via competitive tendering, forms an important part of the European Commission’s proposals for further reform of the European rail sector. This strand of research was the first in the literature to demonstrate the sharp rise in train operating company costs in Britain (previously the policy focus had been on rising infrastructure costs); see for example [4]. Reference [5] found that a key reason behind rising train operating company costs in Britain was the franchising body’s decision to place operators in financial distress on “cost-plus” type management contracts for extended periods.

The impact

An undesirable consequence of rail reform and privatisation in Britain has been sharp increases in both rail infrastructure costs (from roughly £15bn during the first five years after privatisation to £30bn during the second five year period) and train operating company unit costs (up by around 15% since privatisation). The ITS research commissioned by ORR has enabled the regulator to benchmark the efficiency of Network Rail, by providing the key evidence for the regulator to set efficiency targets. The efficiency improvements delivered by Network Rail have directly benefited taxpayers via lower subsidies (although these benefits could instead be directed to passengers through lower fares).

Economic impacts – improving the efficiency of rail infrastructure provision

The first of these studies [1] was used by ORR as the central piece of evidence on Network Rail’s efficiency performance in the 2008 Periodic Review [A]. The efficiency gap, estimated at 37% by the ITS modelling, was used directly to derive a funding settlement for Network Rail covering the fourth ‘control period’ (CP4; 2009/10 to 2013/14). A quote from the ORR’s final conclusions document emphasises how the model was used: “...we [ORR] consider that it is reasonable, transparent and prudent to adopt the 37% value from the preferred model as our estimate for the efficiency gap for each of maintenance and renewals in 2006-07.” [A, p157]. Corroborating the role played by ITS, ORR [A, p128] noted: “…the econometric analysis work was undertaken by ITS working closely with both Network Rail and ourselves”. The value of the efficiency targets set by ORR, based on the ITS evidence, was to reduce the cost of maintaining and renewing Network Rail’s infrastructure from around £18.2bn to £15.8bn over the five year regulatory control period (2009/10 to 2013/14). Note that this reduction is smaller than 37% because ORR phased the target gradually over a ten year period. A letter of corroboration from ORR [B] notes: “Following the PR08 process, assisted by the ITS econometric efficiency model, ORR deemed that the efficient cost should be only £15.8bn (in 2006-7 prices). This work thus directly influenced the determination by the ORR of Network Rail’s funding for CP4, much of which is provided by taxpayers”. The second study commissioned by ORR [2] was also an international econometric modelling exercise, but this time expanding the sample size by utilising sub-company (regional) data within individual railways (drawn from a sample of participating European and North American railways). The model from this second exercise was quoted in ORR’s 2008 determination as a supporting, corroborating study to the first study outlined above [A].

Setting standards for rail regulatory reviews

In its review of ORR’s regulatory efficiency analysis, the National Audit Office in 2011 noted that: “...the Regulator initiated an innovative international benchmarking exercise with the Institute of Transport Studies (University of Leeds) during its 2008 Periodic Review" [C, p31]. Since the 2008 Periodic Review, the approach was further developed and used in subsequent assessments; ITS modelling supported the ORR’s Annual Efficiency Update process in 2010 [D]. Furthermore, Smith and Wheat were seconded part-time to ORR during 2012 and 2013 to advise the regulator on its benchmarking methods for the 2013 Periodic Review; their contribution is acknowledged in the 2013 Draft Determination [E].

Informing policy: the long-term future of Britain’s rail system

The modelling work [1, 2] also formed the basis of future efficiency targets for Network Rail (CP5; 2014/15 onwards) documented in the Sir Roy McNulty Rail Value for Money study (which commenced in 2010 and was published in 2011) [F]. This is arguably the most significant report on rail costs in Britain, which has assessed efficiency and value for money since privatisation and takes a long-term perspective on its future. A letter of corroboration from DfT [G] notes: “The McNulty study represented a major review of rail costs, efficiency and value for money, and set out a new policy direction based on its analysis and findings. The study identified savings for the whole industry of between £2.5bn and £3.5bn per year (or 20-30% of total system costs), of which the savings identified for Network Rail from the work conducted by ITS accounted for £1.8bn per year (£1.2bn per year earmarked to be achieved by the end of CP4 and a further £0.6bn per year by the end of CP5). Through implementing the recommendations of this study (and because of growing passenger revenues), the level of subsidy paid to the railway is forecast to fall from £4bn in 2010/11 to around £1bn by 2019’. In addition, ITS research (a project report for ORR based on [4] and an earlier conference paper version of [5]) is also quoted by McNulty [F] as a key piece of evidence, amongst others, in determining future targets for train operating costs.

Changing practice in other regulated sectors

Extending the reach of the research impacts to other sectors, ITS was commissioned (with consultants CEPA) by the water regulator OFWAT in summer 2012, to develop new econometric models and advise on how they could be used to set a ‘corridor’ for allowed cost levels, whilst complying with the OFWAT’s aim to achieve a “light touch” approach to regulation. The preliminary report [H] released in 2013 sets out the approach developed by ITS (with CEPA). This work has fed directly into OFWAT’s 2014 regulatory review (PR14), as OFWAT has now decided to adopt the ITS approach to econometric modelling for PR14. Furthermore, Smith has been appointed (from August 2013) as the regulator’s academic advisor on cost assessment.

References

[1]  Smith, A.S.J. (2012) The application of stochastic frontier panel models in economic regulation: experience from the European rail sector. Transportation Research Part E, 48, pp503-515. doi:10.1016/j.tre.2011.10.003. This output is based on an earlier project report produced for ORR and published on their website: Smith, A.S.J (2008) International Benchmarking of Network Rail’s Maintenance and Renewal Costs: An Econometric Study Based on the LICB Dataset (1996- 2006). Report for the Office of Rail Regulation, October 2008. 

[2] Smith, A.S.J. and Wheat, P.E. (2012) Estimation of cost inefficiency in panel data models with firm specific and sub-company specific effects. Journal of Productivity Analysis, 37, pp27-40. doi: 10.1007/s11123-011-0220-8. This output is based on an earlier project presentation produced for ORR (co-authored with ORR) and published on their website: Smith, Wheat, and Nixon (20080 International Benchmarking of Network Rail’s Maintenance and Renewals Costs. 

[3] Wheat, P.E., Greene, W, and Smith, A.S.J. (2013) Understanding prediction intervals for firm specific inefficiency scores from parametric stochastic frontier models. Journal of Productivity Analysis, 42 (1) pp55-65. doi: 10.1007/s11123-013-0346-y. 

[4] Smith, A.S.J., Nash, C.A. and Wheat, P.E. (2009) Passenger rail franchising in Britain: has it been a success? International Journal of Transport Economics, 36 (1), pp33-62. The McNulty report (see below) referenced (in footnote 106, p284) a project report produced for ORR based on this output, as well as an earlier conference paper version of output 4: Smith, A., Nash, C. and Wheat, P. (2009) The effect of passenger rail franchising on costs: evidence from the passenger rail sector in Britain (1996 to 2009). 

[5] Smith, A.S.J. and Wheat, P.E. (2012) Evaluating alternative policy responses to franchise failure: evidence from the passenger rail sector in Britain. Journal of Transport Economics and Policy, 46 (1), pp25-49.
The McNulty report (see below) referenced (in footnote 25, page 34 and footnote 
108, p285) an earlier, conference paper version of output 5: Smith, A.S.J. and Wheat, P.E. (2009) The effect of franchising on cost efficiency: evidence from the passenger rail sector in Britain. 

Corroborating sources

[A]   Office of Rail Regulation (2008) Determination of Network Rail’s Outputs and Funding for 2009-14 (October 2008). 

[B]  Letter of corroboration from the Deputy Director, Railway Markets and Economics, ORR. 

[C] National Audit Office (2011) Regulating Network Rail’s Efficiency. 

[D] Office of Rail Regulation (2010) International cost efficiency benchmarking of Network Rail and Technical Support Paper.

[E]  Office of Rail Regulation (2013) Draft determination of Network Rail’s outputs and funding for 2014-19 (see p300). 

[F]  Sir Roy McNulty report (2011) Realising the Potential of GB Rail: Final Independent Report of the Rail Value for Money Study: Detailed Report (see, for example, page 30). 

[G] Letter of corroboration from the Deputy Director, Rail Analysis, Department for Transport.

[H] OFWAT (2013) Cost Assessment