|
home | strategy development | strategy
skills | site
index | access
keys
Multi-criteria analysis
>
in practice
Once the preferred strategic direction has been
determined and policy options to achieve that direction have been
designed, analysis is needed in order to select the preferred policy
option. Multi-criteria analysis can be used for this purpose.
The term multi-criteria analysis (MCA) is applied to a
nested family of techniques, all of which enable policy options to be
assessed against a range of appraisal criteria. The different MCA
techniques include some or all of the following stages:
- Identify policy options for analysis
- Identify criteria against which options will be assessed
- Assess options against criteria using quantitative or qualitative
data
- Score options against criteria on a consistent basis
- Weight criteria and compare options
- Carry out sensitivity analysis & revisit conclusions.
Process
The Process is extremely important to enable
successful multi-criteria analysis. However many of the above stages are
employed, a key characteristic of MCA is the exercise of explicit
judgements - for example in choosing options and criteria, determining
scores, and weighting criteria against each other. This requires an
answer to the question, "Whose judgements are being used?".
In some cases it may be reasonable for officials or
for Ministers to exercise these judgements. But in a climate of public
suspicion of government, and lack of trust in institutions more
generally, this may not deliver sufficient credibility. Hence a more
appropriate approach may be to use the general public (for example in a
"Citizens' Jury") or stakeholders (for example in
"Stakeholder Workshops") to make the judgements that are
necessary. A decision on the most appropriate process to employ at each
stage should be made at the planning stage for the MCA.
1. Identify policy options for appraisal
MCA will typically be used to assess a number of
options for achieving a policy objective, one of which should be a
"do nothing" or "base case" scenario. Ideally, the
starting list of options should be as comprehensive as possible.
However, an iterative process may be necessary, in which new options are
generated in response to the assessment of the initial options (e.g. if
none of the initial options perform well).
2. Identify criteria against which options will be
assessed
There are a number of different ways in which the
range of possible criteria can be categorised, and each individual issue
is likely to employ its own set of criteria. However, the criteria
employed should certainly cover the:
- suitability
- feasibility
- acceptability
- risks of each of the options
Risk can be defined as uncertainty of outcomes (whether
positive or negative). There are two types of uncertainty: uncertainty
that is a result of a lack of information, and uncertainty in terms of
unpredictable events. There are a number of different techniques for
identifying risks,
these include check lists, prompt lists, workshops, questionnaires and
brainstorming.
Work from the organisational
analysis should also feed into the
development of the criteria, particularly those looking at suitability,
feasibility and acceptability. Understanding the organisational structure
and culture of the department can help in understanding the ease or
difficulty with which new strategies can be adopted. Some consideration
should also be given to whether the Department has the resources and
competencies available to deliver a new strategy. The key areas to assess
include:
- Availability of and sources of finance
- Skills: organisational, leadership, technical expertise
- Availability of physical resources e.g. buildings, offices etc
- IT capacity
- HR capacity.
The criteria should also encompass a range of different
perspectives on the policy problem, including the following (where
applicable):
- economic
- social
- environmental
- ethical
- legal
- scientific.
Sponsor, stakeholder and public attitudes should be
reflected, together with relevant local, national (including the devolved
administrations) and international perspectives. It should also include
any specific values or principles that could underpin the success of
future policy.
3. Assess options against criteria using quantitative
or qualitative data
Once the options have been agreed and the criteria
determined, assessing each option against each criterion brings them
together. All available evidence should be employed, both quantitative and
qualitative. The results are then typically presented in a matrix format.
If a wide range of criteria is employed, then the quality and type of
information available to make each assessment will vary considerably. This
will mean that the results in turn will vary in nature, including:
- monetary values
- other quantified data
- rankings
- naïve descriptions (e.g. positive/neutral/negative).
In principle, the MCA could stop at this stage (though
preferably including stage 6) and the matrix presented in its raw form to
decision-makers. The exercise has added value by presenting and appraising
options in a systematic and comprehensive way. But unless one option
clearly dominates all the others, implicit judgement is still required.
Subsequent stages help to make the judgements involved in decision-making
much more explicit.
4. Score options against criteria on a consistent basis
Scoring takes place primarily as a pre-cursor to
weighting, and is designed to present in a common format all of the
results generated in stage 3. A typical approach is to decide a range of
scores for each criterion, for example 0 (the lowest score) to 100 (the
highest score). The end points are then fixed in relation to the raw
results.
Example A: if the cost to the Treasury of an option
varies from £1m to £10m, then a score of 100 may be assigned to £1m and
a score of 0 assigned to £10m.
Example B: if options are ranked as "positive,
neutral or negative", then a score of 100 may be assigned to
"positive" and a score of 0 assigned to "negative",
with "neutral" scored as 50.
Once end points are fixed for each criterion,
intermediate scores are assigned, usually on a linear scale. The matrix
can then be re-written using this common scoring basis.
5. Weight criteria and compare options
The next step is to determine relative weightings for
each criterion. This is in many ways the most difficult stage, and may be
where public and stakeholder input is most crucial - different groups
will have very different weightings. There are a number of approaches. One
often-used approach is to divide a number of points (typically 100)
between the criteria, in line with their perceived weighting relative to
each other. Alternatively, a more qualitative approach may be used (e.g.
"essential" versus "desirable" versus
"irrelevant").
Once weights have been assigned, they are used to
adjust (quantitatively or qualitatively) the scores from stage 4, so that
options can be compared. This should enable an explicit ranking of options
to be carried out.
6. Carry out sensitivity analysis and revisit
conclusion
The ranking emerging from stage 5 may well be sensitive
to some relatively small variations in data, scoring or weighting. The
presence of uncertainty makes it almost inevitable that the assumptions
and judgements employed in the analysis are less than firm. Hence before
any decisions are taken, it is essential that sensitivity analysis is
carried out on each of these aspects of the analysis. The question to be
addressed is: "Do the rankings stay the same when data, scoring and
weighting is adjusted within reasonable bounds?".
Strengths
MCA can typically incorporate a wider range of criteria
(e.g. social, environmental, ethical) than that employed in a typical
financial analysis, and unlike a cost-benefit
analysis, does not require
monetisation of all costs and benefits. At the same time, it brings a
systematic approach to appraising and comparing options with a wide range
of quantifiable and non-quantifiable impacts, and is a more robust process
than the implicit judgements that may otherwise be made.
Weaknesses
The main disadvantage is that - if carried out to its
fullest extent - it can be a very time-consuming process.
References
HM Treasury's
Green Book
Multi-Criteria Analysis - A Manual, appraisal
guidance prepared for DETR by NERA, The Stationery Office (2000) ISBN 1
85112 454 3
The Office
of Government Commerce Policy to
Successful Delivery site provides some useful guidance on evaluating
options
Multi-criteria analysis
In Practice 1: SU Global Health Project
The Global Health Project Matrix Framework
used by Global Health project to evaluate proposals assessed a number of
proposed policy instruments against eight top-level criteria, beneath
which were a number of sub-criteria. A matrix of scores was constructed,
and the scores were then scaled to enable comparison, before the different
criteria were weighted and the policy instruments compared.
|
Multi-criteria analysis
In Practice 2: SU Waste Project
Background and Approach
A central aim of the SU Waste Project was to identify a
preferred option for meeting the EU Landfill Directive which sets tough
targets for the diversion of biodegradable municipal waste from landfill
sites (historically, the main method of waste disposal in this country).
As the UK lags most other developed leading nations in
waste management, the first step in identifying a preferred option was to
benchmark and study the various waste management options used by other
nations. The options in use included actions: to reduce waste streams; to
re-use waste; to recycle and; to develop alternatives to landfill such as
incineration and mechanical and biological treatment (MBT).
The benchmarking work showed that countries varied in
the use made of these different actions. However, most countries generally
intervene at all stages of the waste hierarchy whereas debate in the UK
has tended to focus on the choice between recycling and incineration with
little attention paid to actions that would result in reductions in waste
streams.
Building on this benchmarking exercise, the SU Waste
team consulted with UK waste experts and drew on modelling work to examine
a range of options for tackling UK waste. The first option was the status
quo. The other options covered a range of waste management options with
varying degrees of emphasis on incineration or recycling as well as more
balanced packages of action. Each option had to be able to meet the
requirements of the Landfill Directive.
The options are outlined below:
- Option 1 - do nothing
- Option 2 - High incineration (50% + incineration and 25%
recycling)
- Option 3 - High incineration (50% incineration and 35% recycling)
- Option 4 - Maximum recycling (60% recycling and incineration at
current levels 10%)
- Option 5 - Reduce/recycle - a more balanced package of waste
reduction, recycling, greater variety of residual waste technologies
etc
A lowest cost option was also considered initially (but
abandoned early on as it was judged to meet none of the environmental
criteria and to be a step back from current government waste policy).
Choosing between the options
The options were compared using two analytical tools.
The first of these examined the number of different waste facilities
required for each option and the costs of those facilities over time. This
allowed the present value cost of each option to be estimated over a 20
year period.
The second was adapted from an Environment Agency tool
called "STOAT" - a Strategic Option Appraisal
Tool. Essentially, this model is a multi-criteria analysis tool
that allows one to analyse the benefits and risks of different waste
management options including environmental impacts (e.g. C02 emissions,
leeching), potential land use planning difficulties, consistency with
public preferences and whether an option has been operated successfully in
other countries. These criteria could be weighted in different ways to
establish how this affected the preferred option.
An expert panel was used to assess the feasibility of
each option as alternative ways of meeting the Landfill Directive and to
assess the flexibility of each option (i.e. the extent to which it risked
locking-in to one option). An overall judgement could then be made about
the cost-benefit ranking of the different waste management options, and
their respective strengths and weaknesses.
The options and the results of the analysis are summarised in table 1
below, which is taken from the SU waste report. It shows that a balanced
package of measures with a focus on waste minimisation was judged the
preferred option in cost-benefit terms (under a range of weightings for
the benefits and risks).
Benefits and costs of alternative strategic approaches, 2002-2020:
|
^ top
|
