[ARCHIVED CONTENT] Efficiency Programme: measurement: sign off

Guidance for signing-off SR04 efficiency gains

1. Overview

This draft guidance explains how departments should verify their efficiency gains and sign them off.
The principles underlying the guidance are:

Departments should sign off efficiency data along four dimensions (measurement method; data maturity; service quality; and data systems assurance). This follows the approach in the NAO report.

The process should be based on self assessment by departments.

Because the nature of efficiency initiatives varies widely, the aim is to be flexible while ensuring sufficient consistency across the Efficiency Programme.

With effect from Q4 (fy 05/06), departments will be responsible for performing a self-assessment of the data they submit on a quarterly basis to HMT, beginning with data for the fourth quarter of 2005/06. Departmental sign off will confirm those efficiency gains are accurately reported and correctly classified.

This document provides an overview of how to classify efficiency gains and assess underlying data systems. It is intended to guide those who are running departmental efficiency programmes and/or who are required to sign-off quarterly efficiency returns.

The guidance sets out, at a high level:

The components of an SR04 Efficiency Gain
The reporting classifications which lead to Efficiency Gains being classified as Preliminary, Interim and Final; and
An overview of the criteria for efficiency data systems assurance

Annex 1 to this document provides a sign-off sheet for departments to use to report their gains. Annex 2 provides a checklist for anyone who is required to sign-off efficiency gains for public reporting.

More detailed guidance on Efficiency Data Maturity and Efficiency Assurance which delivery teams may wish to consult is signposted within this document. The SR04 Efficiency Team can also provide further assistance in these areas. Please contact your HMT Efficiency Account Manager in the first instance.

2. Background

The NAO report, 'Progress in Improving Government Efficiency', published in February 2006, concluded that all SR04 efficiency gains reported so far should be regarded as provisional.

In response to the NAO report,HMT has refined quarterly reporting and sign-off procedures and is providing detailed guidance to make it clear what departments need to do to move their reported Efficiency Gains towards 'final' status. The guidance comprises this document, Guidance notes for efficiency data systems assurance, Guidance for assessing data maturity, and Measurement of SR04 Efficiencies - Guidance on efficiency methodologies.

Departments are responsible for reporting their SR04 efficiency gains to the public and Parliament via departmental Annual Reports, Autumn Performance Reports and submissions to HMT, and for ensuring those gains are publicly defensible.

The SR04 Efficiency Team is responsible for challenging and supporting departments in the delivery of their Efficiency Programmes. This includes being able to assure Ministers that the necessary rate of delivery is being achieved and that reported numbers are reliable.

Ministers or Permanent Secretaries sign-off department efficiency returns to confirm that the numbers reported across the Efficiency Programme are a faithful representation of efficiency delivery within their department and are fit for public reporting.

3. Application of assurance principles

The principles of assurance set out in this document apply to all public sector organisations. However the actions to be taken by departments when assuring their efficiency data will differ according to the delivery agent for the particular activity or programme being examined.
There are three different types of delivery agent whose activities contribute towards the achievement of departments' efficiency targets:

Central Government Departments: Departments should apply the principles set out in this guidance for projects they are delivering directly themselves
Agencies, Non-Departmental Public Bodies and Wider Public Sector organisations with direct accountability to a single Department: Departments should not try to duplicate the agency/NDPB's own scrutiny processes, but instead should satisfy themselves that the principles set out in this document are being applied by that organisation. Departments may consider it appropriate to provide further advice that is specific to the particular issues faced by the agency/NDPB on the application of the principles described here
Agencies, Non-Departmental Public Bodies and Wider Public Sector organisations with accountability to more than one Department: Careful planning is required to ensure clarity and streamlining of reporting and scrutiny arrangements. For local authorities, the Cross-Departmental Review Group (formed of departments and local government representatives, and chaired by DCLG) will agree the way in which the principles set out in this document should be applied to councils and the treatment of data received through Annual Efficiency Statements

The points mentioned above apply throughout the application of the principles described in this guidance.

4. Signing-off efficiency data returns

Key individuals within departments have responsibility for signing off the quarterly Efficiency Returns reported to HMT. The sign-off requirements were revised in April 2007. Ministerial or Permanent Secretary sign-off is required to confirm each quarter's data is reported accurately and classified appropriately along four dimensions (measurement method; data maturity; service quality; and data systems assurance). Ministers and Permanent Secretaries will need to assure themselves that the numbers are a faithful representation of efficiency delivery within their department and are fit for public reporting in line with our measurement guidance.

Sign-off covers Efficiency Gains, Workforce Reductions and Lyons Relocations.

Any changes to or clarifications of data already supplied to HMT but prior to that data being publicly reported also need to be signed off.

Efficiency signatories do not need to examine every project in detail before signing off, but they do need to assure themselves that responsibilities are well understood and have been met at all levels of the delivery chain before signing-off.

The act of signing-off efficiency numbers confirms that the department's self-assessment is a fair reflection of the current status of the department's Efficiency Programme in terms of the four elements of efficiency gains described below.

5. Efficiency gains

Efficiency gains are classified in four dimensions:

Measurement Methodology - whether or not there is an agreed measure(s) to quantify the efficiency gain with recorded and agreed efficiency baseline(s) in place (agreed between the department and HMT e.g. in the Efficiency Technical Note).
Data Maturity - the extent to which data are likely to change
Service Quality - whether there is an agreed method to assess quality of service delivery, with recorded baseline(s), and whether or not this demonstrates that the level of service quality has been maintained for the period across which the efficiency gain is claimed
Assurance - the level of assurance the department has in the data systems used to record and report the gain

Figure 1: Components of an efficiency gain

A department classifies its reported Efficiency Gains as Preliminary, Interim or Final. These classifications combine a number of factors: the measurement of efficiency gains, assurance about quality of service delivery and data maturity, as shown in table 1.

Table 1: Efficiency gains route map

A. Measurement methodology & baselines defined and agreed	Yes				No
B. Data maturity	Will not change		May change	Likely to change
C. Quality measure agreed & quality evidenced	Yes	No
Classification	Final	Interim		Preliminary

For an Efficiency Gain to be classified as 'interim' or 'final' the measurement methodology which allows the financial value of the gain to be quantified must have been agreed with the SR04 Efficiency Team. Further details about efficiency measurement can be found in the Guidelines for Measuring Efficiency.

In addition to an agreed measurement methodology an Efficiency Gain can only be classified as 'final' if:

The data can be described as 'will not change' in terms of data maturity; and
The method of assessing service quality (including a service quality baseline) has been agreed with the SR04 Efficiency Team; and
Evidence that quality of service has been maintained is available and has been generated by an appropriate and reliable process

Data relating to a single initiative can be reported across more than one classification. For example, this may be appropriate where older data has been subject to post-year end adjustments but more recent data has not.

In contrast to gains achieved by central government, local government gains tend to be secured through a large number of low value projects. Appropriate processes should be in place to enable these gains to reach final status.

6. Data systems assurance

A data system is a series of records and tasks which documents, classifies, processes and summarises data. When quarterly Efficiency Returns are signed off the department indicates the level of assurance it has in the efficiency data systems for each initiative, as Full Assurance, Substantial Assurance or Partial Assurance, see table 2 below.

Table 2: Overview of efficiency data systems assurance classifications

Full assurance	Substantial assurance	Partial assurance
Internal controls are in place and their operation is regularly checked. A data risk management system is in operation and all significant risks are managed.	Internal controls are basically sound but some design weaknesses or non-compliance have been identified. A data risk management system is basically sound but some risks remain unmitigated.	Internal controls are in place but have not been examined OR significant levels of non-compliance identified. Significant risks to data quality remain Accountabilities are being communicated. A governance structure is under development. A full audit trail is being collated.
Full audit trail available. Accountability for quality of data is clear at all levels. An oversight body reviews the design and operation of data systems.

Further detail on data systems assurance is available in Guidance notes for efficiency data systems assurance.

7. Quarterly self-assessment efficiency sign-off

A sign-off sheet for departments to use to report their self-assessment of their Efficiency Gains is provided in Annex 1.

8. Workforce reductions

The assurance principles set out for Efficiency Gains also apply to departments' workforce reductions. Departments reconcile these numbers with the civil service headcount numbers that the ONS reports publicly on a quarterly basis (reported one quarter in arrears). Departments should agree and maintain a standard reconciliation with Treasury's Workforce, Pay and Pensions Team (WPP), who own policy in the area. Departments should state the level of assurance they have in the data systems used to record and report workforce reductions (see Annex 2).

9. Lyons relocations

The assurance principles set out for Efficiency Gains also apply to departments' reported relocation numbers. HMT require that departments ensure these numbers are consistent with those held by the OGC GET team as part of quarterly efficiency reporting. Departments should state the level of assurance they have in the data systems used to record and report Lyons relocations (see Annex 2).

10. HMT's responsibility for efficiency reporting

Reporting on the financial value of delivery to date, the current number of workforce reductions and the current level of relocations are key efficiency activities. The SR04 Efficiency Team collects data on progress from departments on a quarterly basis, which is consolidated and reported to Ministers and ultimately used to provide updates to Parliament and the public about progress of the Government's Efficiency Programme.

HMT's primary means of ensuring that publicly reported numbers are robust and defensible is to work with departments to ensure that the Measurement, Data Maturity and Assurance Guidance is appropriately applied.

If areas of concern arise, HMT will work with departments to commission joint detailed reviews of measurement, data collection or reporting.

11. External work on efficiency

The NAO and Audit Commission have been consulted in the development of this and associated guidance documents on Efficiency Measurement, Data Maturity and Data Systems Assurance.

The NAO's report, 'Progress in Improving Government Efficiency,' published in February 2006, states that to improve confidence in publicly reported Efficiency Gains the following key principles should be met:

Agreement of clear baselines for inputs, outputs and service quality;
Comprehensive calculation methodologies that help sustain efficiency;
Clear and reliable audit trails including regular assurance of data systems.

A further NAO report was published in February 2007 entitled "The Efficiency Programme: A Second Review of Progress". The NAO is expected to maintain a close interest in the Government's Efficiency Programme.

HMT will support departments in applying the principles set out in the Efficiency Measurement, Data Maturity and Data Systems Assurance. Guidance to help improve the robustness of reported data for the remainder of the SR04 Efficiency Programme.

12. Further information

Contact details

For further information please contact the SR04 Efficiency Team.

Annex 1: Sample efficiency gain sign-off sheet

View the spreadsheet

Annex 2: Points to consider before signing-off efficiency gains

Measurement methodology

Required

Has the efficiency measurement methodology been agreed with the SR04Efficiency Team?
Has the efficiency baseline been agreed and recorded for the start of measurement period?

Data maturity

Required

How mature are reported Efficiency Gains according to the following categories?:

Classification	Definition
Likely to change	Figure is an estimate which is highly likely to change.
May change	Figure provides a good indication of total gain but is still considered subject to change, e.g. because year-end adjustments are anticipated.
Will not change	Figure is not expected to change and will be post any year-end adjustments. (N.B. Figure may be based on estimates if this is set down in the methodology).

Note that data lags are considered separately. See Guidelines for Assessing Data Maturity for further detail.

Demonstrate no reduction in quality of service delivered

Required

Has the quality measurement/assessment methodology been agreed with HMT?
Has the quality baseline been agreed and recorded for the start of the measurement period?
Do recorded measures/assessments of service delivery demonstrate that there is no reduction in service quality?

Data assurance

Assessing risk of misstatement

Has the risk of misstatement of gains been considered on an initiative basis (in terms of both impact and likelihood)?
Have appropriate sources of assurance been identified for each data system?

Internal controls

Are controls in place and operating as intending to ensure the completeness, accuracy and validity of data during collection, processing and reporting?
Where sample data is used, has assurance been gained that the data are representative?
Where data have come from external sources have clear guidelines been issued? Where data have come from an external provider, what steps have been taken to confirm the provider's data is fit for purpose?
Are processes and controls clearly documented?
Are staff adequately trained in the operation of systems?
Are responsibilities for operating controls clearly allocated?

Managing data quality risks and issues

Are risks to data quality during collection, processing and reporting identified and mitigated?
Are the controls in place to manage risks to data quality proportionate?
Are there any significant risks that cannot be managed cost-effectively which will require disclosure in Efficiency Technical Notes or departmental reports?

Accountabilities

Are responsibilities for the quality of data and reporting lines clear to everyone (at all levels of the delivery chain)?

Overseeing data systems

Are senior management satisfied that controls over data quality are operating as intended?
If there are control failures, are they reported promptly to an appropriate level of management and responsibility for addressing failures allocated?
Are data quality issues addressed promptly as they arise?
Are overall reported gains consistent with your knowledge of the department's business?
Are any systems limitations identified and disclosed (as required by HMT guidance - HMT guidance on reporting efficiency gains: PES (2005) 16, 15/9/05; PES (2005) 21, 20/1/2005)?

Audit trail

Do all staff understand the need for a full audit trail?
Do all staff understand the requirements of an audit trail?
Has a full audit trail been maintained for each quarter?
Are satisfactory arrangements in place to ensure the safe retention of the audit trail?

See Guidance notes for Efficiency Data Systems Assurance for further detail.