|
home | strategy development | strategy
skills | site
index | access
keys
Looking forward - Counterfactual
analysis
>
in practice
Counterfactual analysis has theee broad aims:
- To establish evidence of a causal relationship between a new policy,
or change in policy, and outcomes the policy seeks to influence
- To account for confounding factors, additional to the influence of
policy, that might lead to measured change in outcomes
- To provide estimates of the impact of policy
What is it?
The counterfactual is an estimate of the circumstances
that would have prevailed had a new policy or policy change not been
introduced. By comparing counterfactual outcomes (often referred to as
either control or comparison group outcomes) with outcomes measured for
those units subject to the new policy or policy change, causality or
attribution can be established.
A counterfactual analysis tool used by government to
identify causality or attribution is the use of pilots. These enable the
government to test new policies, or changes in existing policy, in a
limited number of geographical areas prior to introducing them more
widely. The objective is to determine whether the new policy gives rise to
changes in the outcomes that policy seeks to alter. For example,
counterfactual analysis might answer the question - is there a direct
relationship between a new initiative to cut car crime and subsequent
change in the number of reported car thefts, independent of other factor
influencing car theft? Counterfactual analysis explicitly acknowledges the
fact that the outcomes government attempts to influence are subject to a
range of factors beyond the immediate scope of the policy being studied.
For example, it can't necessarily be assumed that measures to cut
worklessness are entirely responsible for an observed fall in aggregate
unemployment.
Units exposed to the new policy or policy change are
alternatively referred to as the programme, treatment or action group. In
theory, causality can be attributed to the new policy because there are no
systematic differences between the programme group and a 'true'
counterfactual group, except for the fact that the programme group has
been exposed to the new policy. Differences in average outcomes between
the programme group and the 'true' counterfactual group therefore
represents an unbiased measure of the programme's impact.
In reality, measuring the counterfactual is a difficult
task. Evaluators use a variety of methods, depending on circumstances and
opportunities open to them, to estimate the 'true' counterfactual. The
following approaches can be used:
- Single group pre and post-test designs
- Two group pre and post-test designs
- Model-based econometric methods (simple regression adjustment,
instrumental variables (IV), the Heckman selection estimator)
- Statistical matching designs (e.g., propensity score or cell
matching)
- Interrupted time series analysis
- Regression discontinuity designs
- Randomised control trial (RCT) designs (alternatively referred to as
random assignment, random allocation, experimental or randomised field
trial designs).
It is the latter of these that is considered to be the
most powerful method of establishing a net effect over and above the
counterfactual. This is because programme evaluators explicitly construct
control and programme groups at random. In other words, the two groups are
statistically equivalent, the only systematic difference between them
being that the programme group has been exposed to the policy being
investigated. Evaluators can randomly assign individuals, or other units
such as institutions (for example hospitals or schools), or geographical
areas (for example Wards, or Local Authority Districts).
At present, this approach, while commonplace in
clinical trials, is less often used to evaluate social programmes in the
UK, although there are examples. It is, however, widely used in North
America to investigate the impact of various interventions from changes in
taxation, welfare reform programmes, initiatives in education and criminal
rehabilitation.
Strengths of random assignment
- If implemented correctly, it guarantees that the experimental and
control groups will be identical. Thus it eliminates the influence of
extraneous factors by ensuring that the only differences between the two
groups arise by chance.
- Easy to interpret.
Weaknesses of random assignment
- Two groups are unlikely to be identical apart from some policy
intervention .
- Only provides a measure of average impact.
- Can be complicated to implement correctly- two administrative systems
are required.
- Can create political problems by denying services to controls.
- Risk of contamination if those in the control group are not prevented
from participating in the pilot programme.
Many of these practical problems can be avoided if
whole areas are divided into intervention and control groups, but for
practical reasons this is usually difficult to do.
References
Cost Benefit Analysis, Boardman, Greenberg, Vining and
Weimer (2001)
Research Methods for Policy Evaluation, Department for
Work and Pensions, Research Working paper No 2. (Chapter 4 gives an
excellent description of counterfactual analysis and the different methods
available.)
Counterfactual analysis
In Practice 1: SU Waste Project
In choosing between options the impact of a "do
nothing" option (i.e. what happens if current policies continue, or
the counterfactual) must be considered. The waste team undertook such a
counter factual analysis as part of their work.
To do this assumptions were made about future waste
growth and waste composition (provided by a waste analysis expert working
with the SU team). The team considered current waste funding and looked at
the rate of progress over the last 5 years in recycling and incineration
based on this funding. This showed that the recycling rate had been
increasing at 1% per year and only one new incinerator had been built in
the last 7 years. At this rate of progress, and without kerbside recycling
or more bring sites, recycling was likely to remain below 25% of the waste
stream even by 2015, notwithstanding the fact that this target was
originally set for 2005. It was assumed that current levels of opposition
to incinerators would continue and only those currently approved would get
built.
This analysis established the amount of waste that
would end up in landfill sites on unchanged policies and could be compared
with EU Landfill Directive targets to which the UK was bound. It showed
that, on unchanged policies, many more landfill sites would be needed,
resulting in the UK falling further and further from meeting the Landfill
Directive.
The chart below shows the results of the counter
factual analysis, and the increasing gap between the Landfill Directive
targets and the volume of municipal waste likely to be sent to landfill
sites in England in future.
Estimated biodegradable waste for landfill in England
versus
the EU Landfill Directive targets (million tonnes)
|
Counterfactual analysis
In Practice 2: Jobseekers Allowance (JSA) interview
Random assignment was used to evaluate the introduction
of a Restart interview for Jobseekers in 1989/90. Those claiming benefit
for six months were invited to an interview to encourage return to work.
8,000 people were randomly assigned to receive an interview (intervention
group), while 500 people were randomly assigned to the control group that
was not interviewed. The trial measured the average time it took both
groups to get a job. Those receiving a Restart interview spent 5% less
time claiming benefit.
|
^ top
|
