Efficiency data systems assurance
1 Overview
1.1. Departments (from Q1 06/07 onwards) are responsible for performing a self-assessment of the quarterly data they submit to HMT.
1.2. Reporting SR04 efficiency gains to Parliament and HMT requires underlying data systems and processes to be fit for purpose and operating properly.
Who is the guidance aimed at?
1.3. This document provides technical guidance, recommended criteria for assessing the robustness of gains, and some notable practice case studies.
What is contained within the guidance?
1.4. This guidance describes in detail the principles that effective data systems for collecting, processing and reporting efficiency gains should meet.
1.5. It includes guidance on:
- Data systems assurance classifications;
- How to gain assurance over efficiency data systems and assess what level of assurance is appropriate;
- Sources of assurance and use of independent sources of assurance;
- Audit trails
1.6. This document provides technical guidance, recommended criteria for assessing the robustness of gains, and some notable practice case studies.
1.7. Departments are being asked to classify the assurance they have over their efficiency data systems as Full, Substantial or Partial, according to the criteria set out within this guidance document (See table 1).
1.8. In general, Departments should aim for Full Assurance over all efficiency data systems. However, the act of seeking assurance should not impose undue additional burdens. Therefore where significant resources would be needed to change existing systems to achieve full assurance, departments should consider what level of assurance is appropriate (i.e. whether the degree of assurance sought is proportionate to the total value of the gain and the risk of misstatement - see section 5 for further details).
2. Application of assurance principles
This guidance is principle-based, rather than a comprehensive account of all possible scenarios, so that departments have some flexibility in the application of the principles to their programmes.
There are three different types of organisations whose activities contribute towards the achievement of departments' efficiency targets:
- Central Government Departments: Departments should apply the principles set out in this guidance for projects they are delivering directly themselves.
- Agencies, Non-Departmental Public Bodies and Wider Public Sector organisations with direct accountability to a single Department: Departments should avoid duplicating the agency/NDPB's own scrutiny processes, and instead satisfy themselves that the principles set out in this document are being applied by that organisation. To achieve this, Departments may need to assist in roll-out activities across their agencies/NDPBs to ensure that there is universal understanding of what is required. Departments can provide HMT with further information on particular issues faced by the agency/NDPB where appropriate.
- Agencies, Non-Departmental Public Bodies and Wider Public Sector organisations with accountability to more than one Department: Careful planning is required to ensure clarity and streamlining of reporting and scrutiny arrangements. For local authorities, the Cross-Departmental Review Group (formed of departments and local government representatives, and chaired by DCLG) will agree the way in which the principles set out in this document should be applied to councils and the treatment of data received through Annual Efficiency Statements.
3. Introduction
3.1. Gains are to be described initiative by initiative, in four ways: methodology including baselines; data maturity; service quality; and data systems assurance. Separate detailed guidance is available on each of the four aspects via 'Guidance Notes for Signing-Off SR04 Efficiency Gains'.
3.2. This guidance document provides guidance on the fourth element: data systems assurance, shaded grey in the diagram below
Figure 1: Components of an efficiency gain
3.3. Departments will describe the level of assurance they have achieved in the efficiency data systems as either Full Assurance, Substantial Assurance or Partial Assurance (see table 1). Gaining assurance takes time and may require technical expertise. Because resources are limited, departments will need to perform a risk assessment to judge (by initiative) the likelihood of the gain being misstated and the potential impact of that misstatement on the department's programme. This risk assessment will inform decisions about how best to direct the limited resources available for achieving assurance over efficiency gains. In most cases, internal sources such as the departmental efficiency team, departmental specialists and internal auditors will be able to provide sufficient assurance over gains. However, specialists such as external auditors could be commissioned to do additional work if appropriate.
3.4. This data assurance guidance applies to financial efficiency gains, headcounts reductions and Lyons relocations reported by central government. The primary audience is central government departments with SR04 efficiency targets. Separate arrangements have been agreed by the Cross Departmental Review Group for gaining assurance over local government gains reported in Annual Efficiency Statements (described in DCLG guidance).
3.5. HMT is responsible for overall reporting of delivery of the SR04 efficiency programme. It supports departments in the delivery of their efficiency gains, but also challenges departments to ensure that the status of individual gains is properly reflected when gains are classified and underlying data systems assessed. In relation to systems assurance, HMT will support the development of assurance arrangements and discuss with departments the steps they have taken to achieve assurance over their reported gains.
3.6. The SR04 period is already part complete and in most cases Departments have already devised systems to collect and verify data. However, we expect departments to assure data systems according the guidance below, bearing in mind that we expect the level of assurance should be proportional to the gain claimed. We would not encourage Departments to expend significant resource on changing systems unless the benefits claimed are high or the current level of assurance is 'low'. If Departments have any concerns on how this might impact measuring, assessing and reporting efficiencies then they should discuss with the relevant HMT account manager.
3.7. This document has been prepared in consultation with a range of departments, the National Audit Office and the Audit Commission.
4. Data systems assurance classifications
4.1. A data system is a series of records and tasks which documents, classifies, processes and summarises data. Efficiency gains may be generated by a chain of several data systems especially when gains originate from arm's length bodies or devolved sources (e.g. Local Authorities).
4.2. Departments will make their own assessment of their level of assurance in the system(s) used to record and report efficiency data for each initiative they are directly delivering, and in the chain of systems where initiatives are being delivered by NDPBs. This self-assessment will be signed off as part of quarterly reporting to HMT. Where gains are generated by the wider public sector (e.g. police forces or schools) but managed by a single department the classification should be applied by the lead department. For cross-departmental gains, assurance will be provided via arrangements set down in DCLG guidance, and the Cross Departmental Review Group (in which departments participate) will classify the gains.
4.3. There are five elements to data systems assurance: internal controls, risk and issues management, accountability, governance and audit trail. Assurance is described as Full, Substantial or Partial according to the definitions described in table 1 below.
4.4. In general, it is expected that the reported assurance level will reflect the lowest assurance achieved across the five dimensions. Where departments believe a higher level of assurance is appropriate despite known system weaknesses a clear explanation of the reasoning for this should be given when reporting the gain to HMT.
Table 1: Data systems assurance classifications
| Full assurance | Substantial assurance | Partial assurance | |
| Internal controls | Internal controls to ensure data quality during collection, processing and reporting are in place and regularly checked (at least annually) to confirm they are operating as intended. No significant weaknesses in design or operation have been identified. | Internal control systems are basically sound but some weaknesses in the design of controls or instances of non-compliance have been identified. |
Some internal controls are in place but either their design and operation has not been examined or significant levels of non-compliance have been identified during testing. |
| Data quality risk & issues management | A system exists to identify and manage significant risks to data quality. All significant data quality risks and issues are being adequately managed. | The system for managing risks and issues associated with efficiency gains data systems is basically sound. Some low or medium risks to data quality remain for which mitigating controls have not been established. | Some significant risks to data quality remain for which mitigating controls have not been established. |
| Accountability | Accountability for the quality of reported data is clearly allocated and well understood at all levels of the gain delivery chain. | Accountabilities are in the process of being communicated to all levels of the delivery chain. | |
| Overseeing data systems | A governance structure exists to oversee the design and operation of data systems and to provide challenge and steer in response to issues relating to the quality of reported data. | A governance structure is under development. | |
| Audit trail | A full audit trail is available. For parts of the chain involving external providers (e.g. arms length bodies or local authorities), the department does not need to hold the full audit trail itself. However, it should assure itself that a full audit trail is retained by the original data provider. | The full audit trail is being collated. | |
Internal controls
4.5. Internal controls in the efficiency context are procedures adopted to ensure the data quality of reported efficiency gains. Annex 2 identifies seven criteria which internal controls over data collection, processing and reporting systems should address in order to ensure data quality is maintained. Weaknesses in the design or operation of a system are significant if they could result in significant misstatement of the reported gain.
Risk and issues management
4.6. A system for identifying and managing risks and issues relating to data quality should be in place. Steps should be taken to mitigate significant risks that could threaten material misstatement of reported gains. In general, all risks to data quality should be mitigated unless to do so would require disproportionate resources. OGC's publication 'Management of Risk: guidance for practitioners' provides further guidance on general risk management, which you may find a useful reference.
Accountability
4.7. Accountability for the quality of reported gains at all levels of the delivery chain should be clearly documented and well-understood by all those involved in the generation of departmental gains. It is not practical for those signing off gains to review the detail of every initiative so they will need to satisfy themselves that systems at the lower levels of the delivery chain are fit for purpose and operating as intended. Clarity over accountability is especially important where the delivery chain is complex and involves other bodies.
Governance systems
4.8. The management of departmental efficiency programmes should include an overview of data quality and systems. This role could be performed by the efficiency board or a sub-committee of it, or an alternative structure. Further detail is provided in Annex 3.
Audit trail
4.9. Sir Peter Gershon's 2004 review of public sector efficiency established the requirement that planned efficiency gains be 'auditable and transparent'. Departments are responsible for ensuring that sufficient appropriate auditable evidence exists and is retained to support their reported efficiency gains.
4.10. The auditable evidence should form a trail that demonstrates the processes applied to source data at all stages of processing and reporting, and which enables the reconstruction and review of the reported gain from the source data. An audit trail which can be accessed quickly and without the need for ad-hoc, labour-intensive collation should be available for each initiative. An audit trail does not need to have been audited, but must be 'auditable'. Where there is a complex delivery chain involving other bodies, departments do not need to retain the entire audit trail themselves, but steps should be taken to ensure external providers retain a sufficient audit trail. For councils, this will be done through the Annual Efficiency Statement process and the guidance prepared for councils by the Cross-Departmental Review Group. Further information on audit trails is contained in Annex 4.
5. Gaining assurance
What level of assurance is appropriate?
5.1. The degree of assurance sought should be proportionate to the risk posed and the cost of achieving that assurance. Resources used to provide assurance should be proportionate to the total value of the gain achieved, and the act of seeking assurance should not impose undue additional burdens. Placing reliance on existing sources of assurance will help limit the costs associated with pursuing efficiencies. Nonetheless, where departments consider existing sources inadequate, they will need to consider commissioning further work.
5.2. In general, full assurance should be the aim. However, since some existing data systems are not ideal and the cost of gaining assurance can be high, it is not essential for all reported gains to be generated by systems for which full assurance has been achieved. Making full assurance an essential requirement for all gains would require disproportionate effort for many parts of the efficiency programme. However, where full assurance can be gained at relatively low cost, it should be sought.
Assessing the level of risk
5.3. When departments have to make a judgement about where to focus their assurance resources, this should be done by assessing the risk of misstatement in terms of the likelihood and potential impact on the departmental programme of a reported gain being misstated.
5.4. Risk factors to consider when assessing the impact of misstatement, and therefore the appropriate degree of assurance to be sought include:
- Size of targeted gain in terms of the overall departmental efficiency programme.
- Significance in terms of effect on future ability to deliver services. Departments may have key programmes which though small in financial terms can have a significant effect on the future ability to deliver services.
- Gains attracting significant public interest.
5.5. Risk factors to consider when assessing the likelihood of misstatement include:
5.6. Gains reported using a complex methodology or via a long series of linked systems.
5.7. Initiatives for which gains have required restatement in previous reporting periods.
5.8. Other common factors which can increase the risk of incorrect reporting are detailed in Annex 5. Departments may identify additional factors specific to their own efficiency gains.
Table 2: Relative size of gains and associated assurance levels
 |
Full assurance | Substantial assurance | Partial assurance |
| Departments should aim for Full Assurance for all large initiatives | Further work may be needed to improve systems assurance | Further work may be needed to improve systems assurance | |
| Departments should aim for Full Assurance if initiative is high profile, complex, or key to other deliveries (i.e. medium to high misstatement risk) | Substantial assurance is acceptable for small initiatives assessed as low risk | Further work may be needed to improve systems assurance | |
| Full assurance is ideal, although cost of gaining assurance should be proportionate to size of gain | Substantial assurance is acceptable for very small initiatives assessed as low to medium risk | Partial assurance is acceptable for very small initiatives assessed as low risk | |
 |
|||
| Ideal | Acceptable depending on risk of misstatement | May require further work | |
Case Study 1: Identifying priorities and developing assurance processes
|
Identifying Priorities DTI has a number of Arms Length Bodies (ALBs) in its delivery chain, which lead to particular issues when considering assurance over data systems: Low existing assurance: Arms Length Bodies (ALBs) are a step removed from the department and their systems are not linked to the department's own financial systems. DTI therefore had low existing assurance over the gains data reported to it by ALBs. Size of gain: The largest ALB gains come from Research Councils and Regional Development Agencies, so DTI decided one of each of these should be examined in the first phase of testing. Complexity: The measurement methodologies for the ALBs were the most difficult to develop, and it was felt that early testing of the practical operation of these methodologies would be helpful. This led DTI to prioritise work by its internal auditors towards the gains reported to the department by ALBs. For added independence internal audit was used to assure systems and the auditors were given the freedom to select the particular Research Council and Regional Development Agency examined in the first phase of testing. Example: Regional Development Authorities (RDAS) RDAs are non-departmental arm's length bodies with responsibility for regional economic development across the UK. DTI has taken steps to ensure that the programme is appropriately managed both at the RDA local level and departmental level. The agreed reporting process demonstrates the following elements of good practice for management of arm's length bodies:
|
6. Sources of assurance
6.1. There are a number of sources of assurance that departments can use to confirm that the system (or chain of systems) leading to a reported gain is fit for purpose and operating as intended. Table 3 identifies some sources of assurance and the type of assurance those sources can provide. The list is not exhaustive and departments may identify additional sources specific to their circumstances.
Table 3: Sources of assurance
| Potential source of assurance | Potential nature of assurance |
| Departmental efficiency team |
e.g. Confirming that baselines have been recorded Checks over input data e.g. initial review and cleansing (to ensure validity & completeness) Reconciliations to ensure accuracy of data Processing checks to confirm calculations are accurate Reasonableness checks of gains prior to reporting (e.g. cross-check with alternative data sources or testing congruence with past data trends and/or forecast trajectories). Reconciliation of data reported to other places e.g. ONS staffing data |
| Departmental specialists, e.g. analysts, statisticians, cost accountants |
e.g. Measurement methodology sound and practical Sampling methodology adequate |
| Internal audit |
e.g. Appropriate design of internal controls, governance systems and risk management systems. Internal controls, governance arrangements and risk management systems operate adequately |
| IT controls |
e.g. IT controls can ensure that: Data entered lie within a plausible range & are complete Data manipulations are recorded Error reports can be easily produced |
| Efficiency specialists |
e.g. Measurement methodology adequately formulated |
| External consultants incl. Academics or qualified auditors |
e.g. Estimations & assumptions fair Technical approaches (e.g. collection of survey data) are robust |
6.2. Where possible, and subject to consideration of appropriate costs, senior management should not rely solely on assurance reports provided by the teams that collect, process and report gains data since those teams may lack the detachment and expertise to provide the necessary assurance. Senior management can look to other departmental staff not involved in daily management of the programme, internal auditors or external consultants to provide the necessary assurance.
6.3. Where departments receive submissions from external providers they should seek the same level of assurance to support sign off as if the gain had been generated internally. For submissions from councils this process will be undertaken through the Cross-Departmental Review Group.
6.4. The National Audit Office's audit of departmental financial statements should not be relied upon to provide assurance over reported efficiency gains or the operation of the data systems used to record and report those gains. The NAO audit is designed to provide assurance that the financial statements are free from material misstatement and that expenditure and income have been applied to the purposes intended by Parliament. The audit is not designed to provide assurance over efficiency gains, which are likely to be immaterial in the context of the financial statements. The NAO's work is therefore no substitute for departments seeking their own assurance over the data systems used to record and report efficiency gains. However, departments should be alert to any issues identified during the Financial Statements audit which may be relevant to the recording and reporting of efficiency gains.
Case Study 2: Use of internal audit
|
In July 2005, DTI engaged its internal audit department to review the departmental efficiency programme. DTI management recognised that internal audit could provide an additional level of assurance to the programme team and key internal and external stakeholders that reported gains were robust. The DTI's use of internal audit incorporated the following elements of good practice:
|
7. Independent assurance
7.1. Although the Gershon Review required that planned efficiency gains are 'auditable' it did not say to what extent they should be audited, or by whom. Internal auditors can be an excellent source of independent assurance over efficiency gains because of their experience and knowledge of the business. The use of internal audit to provide assurance over efficiency gains is not mandatory, but HMT Public Expenditure System guidance (Public Expenditure System, (2005) 21 Guidance for the Spring 2006 Departmental Reports, HMT 20/12/05) requires that, where gains are subject to audit, departments define how those gains were audited and outline the auditors' conclusions.
External scrutiny
7.2. In most cases, internal sources of assurance are likely to be sufficient to provide departments with the necessary comfort over their gains. However, circumstances may arise when a department feels it would benefit from an external view on its efficiency programme arrangements. This could be, for example, because internal audit do not have the necessary specialist expertise, or simply because the department feels it could benefit from 'fresh eyes' on the programme. Alternatively, it may be felt that scrutiny by an independent, external provider has greater public credibility. In such cases, departments can commission external experts e.g. external auditors, or statistical or measurement specialists to provide independent comment on particular areas of concern.
7.3. As the responsible owner of the efficiency programme overall, HMT will act jointly with departments to commission work in areas where concerns exist over reported gains. If issues arise in relation to cross-departmental gains, they will be addressed by the DCLG via existing review arrangements.
7.4. As external auditors to government departments and HMT, the National Audit Office (NAO) reports to Parliament on the progress of the efficiency programme. In the first report of a rolling programme of Value for Money studies on the efficiency programme published in February 2006, the NAO commented in detail on 20 initiatives from across the programme. It is likely that subsequent studies will take a similar approach.
7.5. From 2005-06, Audit Commission appointed auditors will review arrangements for local authorities' Annual Efficiency Statements within the wider context of departmental Comprehensive Performance Assessments and Use of Resources. Reporting will be on an exception basis where auditors become aware of significant weaknesses in arrangements.
Annex 1: Additional sources of information
Guidance
1. A number of other efficiency guidance documents are available:
- Guidelines for Measuring Efficiency
- Guidelines for Assessing Data Maturity
- PDF file of Procurement Efficiency and Value for Money Measurement: Efficiency Programme Measurement Guidance
- PDF file of Productive Time: Efficiency Programme Measurement Guidance
- Successful delivery pocketbook (OGC website)
- Management of risk: guidance for practitioners - please contact the OGC Service Desk on 0845 000 4999 for a copy.
Some of the documents above are available in Adobe Acrobat Portable Document Format (PDF). If you do not have Adobe Acrobat installed on your computer you can download the software free of charge from the Adobe website. For alternative ways to read PDF documents and further information on website accessibility visit the HM Treasury accessibility page.
Gershon Review
2. Releasing resources to the front line: Independent Review of Public Sector Efficiency
HM Treasury
3. Choosing the Right Fabric: A framework for Performance Information is a joint publication prepared by HM Treasury, Cabinet Office, National Audit Office, Audit Commission and the Office for National Statistics.
Public Audit Forum
4. The Public Audit Forum brings together national audit agencies in an advisory capacity. Its report Improving Performance Information can be downloaded from the Public Audit Forum website.
DCLG
5. The DCLG's guidance on local authority gains can be obtained from the Regional Centres of Excellence website
National Audit Office
6. The first report in the National Audit Office's rolling programme of reports on the efficiency programme Progress in Improving Government Efficiency, HC802, February 2006, can be downloaded from the National Audit website. Appendix 5 of the report provides guidance on data systems for efficiency projects. The NAO website also has a section devoted to public sector efficiency http://www.nao.org.uk/efficiency/.
Audit Commission
7. The Audit Commission has developed guidance on data quality in the context of performance improvement.
Annex 2: Data collection, processing & reporting
1. Table 1 requires internal controls to be in place to ensure data quality during data collection, processing and reporting. If reported gains are to be considered reliable, the internal controls need to ensure the following criteria are met, see table 4.
Table 4: Data systems criteria
| Criterion | What this means for data | What this means for data recording & processing systems | What this means for reported gains |
| Accurate |
Values should be recorded to a level of accuracy proportionate to the purpose, i.e. sensitive enough to measure the targeted change in activity during the period of the programme, but not so accurate as to be a drain on resources. The level of error which could go undetected should be significantly less than the amount of change targeted. The financial value that is apportioned to the activity should also be accurate in terms of being based on relevant recent data sources (e.g. using payroll data for calculating productive time gains). |
Data values should not be corrupted during processing. Mathematical functions applied to data should be arithmetically correct. Aggregations of data by type should be accurate. |
Reported gains should agree with gains shown in the department's data systems. |
| Complete | Data should include all relevant instances. For example, if the intended gain arises from headcount reduction, data for all posts in all relevant organisations should be collected. | Data should retain their completeness during processing, i.e. no relevant data should be lost (although data cleansing may remove incorrect data items). |
Reporting should be informative, and in a manner the lay reader can comprehend. It should not exclude or obscure unfavourable results. Limitations in the systems which generate efficiency gains should be made clear to the reader. Specific reporting requirements are set out in HMT's Public Expenditure System guidance. |
| Valid | Data should exclude irrelevant instances. For example if the intended gain involves reducing the annual running cost of the finance department, the data should not include costs relating to other corporate services such as IT or HR. Irrelevant instances can also include data which should be reported in another time period or by another accountable body. | Data should retain their validity during processing, i.e. irrelevant data should not creep in during processing. |
Reporting should be informative, and in a manner the lay reader can comprehend. It should not exclude or obscure unfavourable results. Limitations in the systems which generate efficiency gains should be made clear to the reader. Specific reporting requirements are set out in HMT's Public Expenditure System guidance. |
| Appropriate | Data collected need to be fit for purpose and cover the aspects of performance expressed in the measures. This is especially important where pre-existing systems are used to provide data: consideration should be given to whether data which may originally have been collected for a different purpose can meet the requirements of the measures. Where significant limitations exist these should be reported. |
Data should be processed so as to generate information that enables:
Aggregations and mathematical functions applied to data should accurately reflect the intention of the measurement methodology. |
Reporting should be informative, and in a manner the lay reader can comprehend. It should not exclude or obscure unfavourable results. Limitations in the systems which generate efficiency gains should be made clear to the reader. Specific reporting requirements are set out in HMT's Public Expenditure System guidance. |
| Consistent | Data should be comparable across different time periods and different sources. Where data is fed in from a number of external providers detailed guidelines should be issued to ensure consistency in collection. | Where data systems change, checks should be made to ensure comparability of data over time has been maintained. | Reporting should be consistent with the measures and methodologies set out in the published revised Efficiency Technical Note (including any supplementary notes on methodology published) and with information provided in departmental reports and for Budget or Pre-Budget Reporting purposes. Reporting should also be consistent with HMT measurement guidance. |
| Timely | Data should be collected on a timely basis to ensure reporting of gains at the earliest opportunity. | The time between data collection and reporting should be minimised. This assists monitoring of progress and allows early corrective action to be taken. | The time between data collection and reporting should be minimised. This assists monitoring of progress and allows early corrective action to be taken. |
| Owned | Responsibility for data quality should be clearly identified, particularly when data are provided from a number of sources. | Responsibility for data processing and the maintenance of data quality during processing should be clearly allocated. | Responsibility for reporting robust gains should be clearly allocated. |
Case Study 3: Use of internal controls by MOD when collecting and processing efficiency data
|
MOD has incorporated a number of manual and automated internal controls (for definitions of the criteria see table 4 above) in their efficiency reporting process. The table below gives some examples of how MOD has used these controls in practice.
Further points to note:
|
Data collection
2. Data sources vary widely. For example data can be:
- comprehensive or sample-based;
- generated by the department's own systems or received from others (e.g. arm's length bodies, consultants etc.);
- actual data or extrapolated;
- snapshot or period data.
3. The cost of collecting and reporting data should be proportionate to the size of the gain being targeted. In some instances, comprehensive data collection across an entire population is not feasible and samples are relied on instead. Sample data present a higher risk of error and additional steps should be taken to ensure the quality of such data, see table 5.
Table 5: Good practice - using sample data
|
Data from external providers
4. Expectations of data quality should not be lower simply because data originate from outside the department. Where a department uses data from an external provider that has direct accountability to it alone, it should assure itself that the provider's experience, skills, capacity and quality assurance processes are adequate to ensure the data provided to the department are fit for purpose for an example see Case Study 1 above). Arrangements should be made to ensure that the provider alerts the department to any data quality issues as they arise.
5. For external providers that have accountability to more than one department then the assurance process needs to be agreed collectively. For councils, this will be done through the Cross-Departmental Review Group. From 2005-06, the Audit Commission will alert DCLG to any local authority which does not have appropriate arrangements in place for preparation and certification of Annual Efficiency Statements.
6. Responsibilities for data collection and quality should be clear to all parties. Where data is managed by a contractor, the contract should specify data quality requirements and quality assurance arrangements.
7. When departments disaggregate local authority gains reported via Annual Efficiency Statements and allocate them across a number of initiatives, assurance should be sought over the appropriateness and accuracy of the disaggregation process.
Data reporting
8. Treasury guidance (Public Expenditure System (2005) 16 Publication of Autumn Performance Reports, HMT 15/9/05) requires departments to provide regular public updates on progress towards their SR04 efficiency targets. Reporting should state progress towards the headline efficiency target, workforce reductions and the Lyons relocations and should follow the principles established for PSA reporting, namely clear, informative and objective reporting with supporting information (including any data/statistics), and set in the context of wider policy where appropriate.
9. Although it is not a Treasury reporting requirement, reporting improvements that have resulted from efficiencies is a good way to increase credibility of claimed gains. Although not mandatory, departments could consider reporting how they have reinvested cashable gains in order to provide additional goods or services. Similarly, improvements in output or outcomes which arise from non-cashable gains could be reported to beneficial effect, see table 6.
Table 6: Public reporting and reporting to the centre
| HMT reporting purposes | Public reporting/credibility purposes | |
| Cashable | Financial value, e.g. 15 million gain | What released money has been spent on e.g. 500 new nursing posts |
| Non-cashable | Financial value representing the improvement in quality and/or quantity of outputs, e.g. 28 million gain | Improvement in quality and/or quantity of outputs, e.g. average case duration reduced from 5 days to 3 days |
Annex 3: Overseeing data systems
1. Departments should have a governance structure into which assurance mechanisms can report. Many departments have established structures similar to those set out in table 7. The responsibilities of these different elements and the reporting lines between them should be clearly documented and understood by relevant staff.
Table 7: Governance structures
| Governance element | Role |
| Main departmental board | The board regularly reviews and challenges the department's efficiency programme. This increases the profile of the programme which contributes to the sustainability of the work. |
| Senior Responsible Owner (SRO) |
An SRO has been appointed for the programme; in some departments SRO roles have also been created for each workstream. SROs are responsible for overall data quality. In particular they ensure the following are in place:
|
| Efficiency board |
The main board often delegates the detail of overseeing the department's efficiency programme to an efficiency board. This board:
Efficiency boards might consider whether they would benefit from experienced independent members who could provide a valuable critical perspective. |
| Efficiency team | Manages day-to-day delivery and reporting of gains. |
Case study 4: Programme oversight
| The MOD's Efficiency Delivery Board (EDB) meets every six weeks and is chaired by the Second Permanent Under Secretary. It scrutinises, challenges and supports the work of the department's efficiency programmes and workstreams, and progress is reported upwards to the department's main Management Board. As part of its work, the EDB considers the risks arising from data quality and is putting in place a rolling programme of audit to provide assurance. The EDB recently charged the programme team with implementing the latest internal audit recommendations on formalising procedures for data validation and the identification of double counting. |
Annex 4: Audit trail
1. Auditors, both internal and external, may wish to examine the audit trail underpinning reported gains. The level of evidence required depends on a number of factors including: the complexity of measurement methodology; materiality; and the nature of the controls in place.
Table 8: Good practice - supporting documentation
|
For ease of review, audit trail documentation (both paper and electronic) should:
As an overall test, documents should stand alone without the need for explanation from staff (who may not be around in the future to explain them). Electronic data systems should in addition maintain a record of data entry and data manipulations. |
Case study 5: DFT - Vehicle Excise Duty - Use of audit trail
|
DfT is working to reduce the number of unlicensed and uninsured vehicles on our roads. DfT has been set the target of collecting 70 million additional vehicle excise duty income by 2007-08 from a reduced level of tax evasion. DfT has introduced continuous vehicle registration, enforcement from the vehicle record not just from roadside observation, late licensing penalties, a more targeted enforcement effort and employing civil debt recovery agencies. DfT has a number of management information sources to track levels of evasion including monthly returns from Automatic Number Plate Recognition cameras. But these measures are only a supplement to an annual National Roadside Survey (produced by Transport Statistics: DFT). The audit trail for this initiative displays the following elements of good practice:
|
Annex 5: Factors increasing the risk of misstatement
Table 9: Factors increasing the risk of misstatement
| Factor contributing to increased risk | Risk arising | Short & long term mitigating action(s) |
| Manual calculation and data entry are susceptible to human error. | Data set may be incomplete or contain invalid instances; individual items may have inaccurate values. |
Independent reworking of calculations to check accuracy. Establish a data quality assurance process. Computerise as far as possible. Minimise data entry points in process. |
| Risk increases as complexity of data collection and processing increases. | Data set may be incomplete or contain invalid instances; individual items may have inaccurate values. |
Set clear data standards and procedures; hold managers responsible for data quality |
| Risk may be greater if the methodology or system is new or has been recently modified or if there have been significant changes in key staff. | Data collection or processing may be inconsistent with previous periods. | Document procedures for the operation of the system. Where systems are modified the implications for comparability of data over time should be considered. |
| A lack of expertise of those operating the system can lead to poor-quality data collection and data entry to IT systems. | Data set may be incomplete or contain invalid instances. | Ensure appropriate training and support is available to those operating the system |
| Subjective judgement: Certain aspects of targeted performance may be difficult to measure directly, e.g. customer satisfaction. | Reported gain may be inaccurate if based on poor assumptions or unrepresentative samples. | In such circumstances, approximations such as sample surveys of customer satisfaction may be used. Employ technical specialists with relevant expertise, e.g. survey experts to ensure surveys are well-designed and the sample population is representative. |
| Use of data to manage and reward performance encourages commitment to efficiency but can also provide incentive for the introduction of bias in reported gains. | Reported gains may be inaccurate. |
Mechanisms to encourage objectivity and independence (e.g. separation of operation and monitoring functions) Active monitoring by people not involved in daily management of the programme of:
|