Abstract
Many applications of analytical methods in public policy relate to aspects of ex ante decision support, such as assisting policy makers assess projected costs and benefits of potential approaches to implementing policy objectives. This paper sets out some issues in a less widely discussed area of policy analytics, namely their application to holding government bodies to account, ex post, for the ‘value for money’ with which they have used public funds. It shows how the task of arriving at a judgment on value for money can be modelled as the construction of a partial order relation on an ‘outcome space’ of possible outcomes with ‘value’ and ‘money’ dimensions, and uses this framework to present case examples from the work of the UK National Audit Office. It shows how analytical methods are used to provide evidence to support such judgments. It concludes that there is scope for exploring the application of new techniques, such as big data analytics, to developing the ‘observed’ and ‘possible’ elements of value-for-money conclusions. There is also scope for applying decision-analytic techniques, such as multi-criteria decision analysis, to develop the ‘value-for-money criteria’ against which actual performance can be judged in fair, balanced, transparent, consistent, and technically rigorous manner.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Departments are also supposed to evaluate the success of policy interventions. Government guidance such as the ‘Green Book’ (HM Treasury 2011a) and the ‘Magenta Book’ (HM Treasury 2011b) distinguish between process evaluation (how the policy was implemented), impact evaluation (what difference the policy made) and economic evaluation (whether the benefits of the policy exceeded the costs). VFM audit is concerned not only with actual benefits and costs, but also with whether the benefits could have been achieved at lower cost, or the greater benefits achieved for the cost incurred.
Clearly the Supreme Audit Institution has to form a view as to which constraints are binding: and there may be differences of opinion on this point between different groups within civil society, the media, etc. In jurisdictions characterised by authoritarianism and/or corruption, forming such a view may present significant ethical challenges.
‘Defeasible’ reasoning is based on reasons that are acknowledged to be potentially fallible or corrigible, as opposed to the ‘indefeasible’ reasons of deductive logic.
Note that sometimes, as in this example, the ‘observed’ component includes both factual and counterfactual elements. Measurements of additionality or of (cost-)effectiveness, necessarily involve a comparison against a counterfactual: in this case, an assessment of what consumer harm would have occurred, absent the OFT’s interventions (the methodology used to establish this is discussed further in Sect. 8 below). Having ‘observed’ (i.e. measured, or described) the effectiveness of an intervention, however, it is then necessary to judge whether that degree of effectiveness was good or bad, against a standard of what would be a ‘reasonably possible’ level of effectiveness.
Some NAO reports do not have a conclusion on VFM. Such reports therefore aim simply to give an account of an issue: to establish ‘the facts’. In these cases the C&AG does not offer a judgment. Of course Parliament, through the Public Accounts Committee or other select committees, may then wish to come to its own judgment about the bodies or individuals concerned, on the basis of the NAO’s account and any other evidence it considers relevant.
It is important for the NAO to consider these questions in designing and running its VFM audit practice, because, by contrast with its financial audit practice, there are no external standard-setting bodies for VFM audit. There is extensive codification of standards in financial audit, undertaken by bodies such as the International Accounting Standards Board. There is codification of what ‘optimal performance’ looks like (articulated in detail in international financial reporting standards), and (via the international auditing standards) codification of what counts as appropriate and sufficient evidence for supporting findings on the extent to which actual performance measures up, to a level that is reasonable (defined in terms of ‘materiality’) within the constraints on the organisation being audited. For VFM audit, the NAO must itself do the thinking required to determine what should count as ‘VFM standards’, and how they should be applied in practice.
A binary relation, R, on a set \(X\) is a collection of ordered pairs of members of \(X\). If \((x,y)\) is one of the ordered pairs in \(R\), we say that \(x\) and \(y\) are related by \(R\), and write xRy. A partial order \(\lesssim \) on a set \(X\) is a binary relation with the following properties. It is reflexive (which means \(x \lesssim x\), for every \(x\) in \(X\)); it is anti-symmetric (which means that, for all \(x\) and \(y\) in \(X\), if \(x \lesssim y\) and \(y \lesssim x\), then \(x =y\)); and it is transitive (for all \(x, y, z\) in \(X\), if \(x \lesssim y\) and \(y \lesssim z\), then \(x \lesssim z\)).
Part of the VFM auditor’s core professional competence should be the ability to critique ‘good practice toolkits/models/checklists’ put forward by, for example, professional organisations or management consultants. The received wisdom about what counts as ‘good practice’ (in financial management, in risk management, in governance, in performance measurement,...) can change radically over time, when it becomes clear that practices that were thought to be associated with good outcomes turned out not to be. Thus in the early 2000s we witnessed the ‘Enron buzz’, where Enron’s business model was lauded by business schools, followed by the collapse of Enron and the dropping of Enron as a case study from MBA courses. In VFM terms, the warrant to allow one to draw a Type C conclusion with confidence (i.e., clarity about the strength of the evidence for the connexion between practices and outcomes) was lacking.
As the study team could not observe the payment values of the averted complaints directly, they assumed that an averted complaint from a particular company could be represented by a random pick from a lognormal distribution with the same mean and standard deviation to the sample of payment values recorded in the OFT’s database, for that company, over the period 2009–2010 to 2011–2012.
The cost-effectiveness ratio of 8.6 implies that the value of the detriment prevented by enforcement of regulations was \(8.6\times {\pounds }4.5\,\hbox {m} \approx {\pounds }39\,\hbox {m}\); and there was a further £450m of unremedied detriment. So we can estimate total detriment in the market as around £489m. For a spend of £5m, we estimate the detriment prevented as about \(8.6\times {\pounds }5\,\hbox {m} = {\pounds }43\,\hbox {m}\); so the remaining unremedied detriment is \({\pounds }489\,\hbox {m}-{\pounds }43\,\hbox {m} = {\pounds }446\,\hbox {m}\), i.e. £4m less than £450m.
After rounding. The actual modelled improvement in QALYs was 0.28.
The NAO has publicly stated (NAO 2013b) that ‘The bodies we audit must have confidence in our judgements and see that we practice what we preach’.
References
Bechberger, E., Lane, D., McBride, T., Morton, A., Quintas, D., & Wong, C. (2011). The National Audit Office uses OR to assess the value for money of public services. Interfaces, 41(4), 365–374.
HM Treasury. (2011a). The Green Book: Appraisal and evaluation in central government. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/220541/green_book_complete.pdf.
HM Treasury. (2011b). The Magenta Book: Guidance for evaluation. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/220542/magenta_book_combined.pdf.
National Audit Office. (2010a). Progress in improving stroke care HC 291 session 2009–10. London: The Stationery Office. http://www.nao.org.uk/report/department-of-health-progress-in-improving-stroke-care/.
National Audit Office. (2010b). Progress in improving stroke care: Report on the findings from our modelling of stroke care provision. http://www.nao.org.uk/wp-content/uploads/2010/02/0910291_modelling.pdf.
National Audit Office. (2012a). Regulating consumer credit HC 685 session 2012–13. London: The Stationery Office. http://www.nao.org.uk/report/office-of-fair-trading-regulating-consumer-credit/.
National Audit Office. (2012b). Regulating consumer credit: Technical paper. http://www.nao.org.uk/wp-content/uploads/2012/12/1213685_tech_paper.pdf.
National Audit Office. (2013a). Building capacity in the senior civil service to meet today’s challenges HC 129 session 2013–14. http://www.nao.org.uk/wp-content/uploads/2013/06/10167-001-Full-Report.pdf.
National Audit Office. (2013b). NAO strategy 2014–15 to 2016–17. http://www.nao.org.uk/about-us/wp-content/uploads/sites/12/2013/02/NAO_Strategy_2014-15_to_2016-17.pdf.
O’Neill, O. (2002). A question of trust: The BBC Reith Lectures 2002. Cambridge: Cambridge University Press.
O’Neill, O. (2013). Intelligent accountability in education. Oxford Review of Education, 39(1), 4–16.
Scharaschkin, A. (2011). Evidence and argument. In J. Lonsdale, P. Wilkins, & T. Ling (Eds.), Performance auditing: Contributing to accountability in democratic government. Cheltenham: Edward Elgar.
Shafer, G. (1976). A mathematical theory of evidence. Princeton: Princeton University Press.
Sharma, N. (2007). Interactions and interrogations: Negotiating and performing value for money reports. Financial Accountability & Management, 23(3), 289–311.
Srivastava, R., & Shafer, G. (1992). Belief-function formulas for audit risk. The Accounting Review, 67(2), 249–283.
Stewart, T. R. (2012). A Bayesian audit assurance model with application to the component materiality problem in group audits. PhD thesis, Vrije Universiteit Amsterdam. http://dare.ubvu.vu.nl/bitstream/handle/1871/39813/dissertation.pdf;jsessionid=1CE3B82EC50E80DBC71A899AB8BF7726?sequence=1.
Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34(4), 273–86.
Acknowledgments
The authors have benefited greatly from discussions on the ideas presented in this paper with colleagues at the National Audit Office, particularly Will Palmer, Henry Midgley, Joe Perkins, Max Tse and Michael Kell; and from discussions with Alec Morton at the University of Strathclyde. They are grateful to two anonymous reviewers for their comments on an earlier version of the paper.
Author information
Authors and Affiliations
Corresponding author
Additional information
Both authors previously worked at the UK National Audit Office (NAO). The views expressed in this paper, however, are the authors’ own, and do not necessarily represent the views of the NAO.
Rights and permissions
About this article
Cite this article
Scharaschkin, A., McBride, T. Policy analytics and accountability mechanisms: judging the ‘value for money’ of policy implementation. Ann Oper Res 236, 39–56 (2016). https://doi.org/10.1007/s10479-014-1723-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10479-014-1723-2