Are we there yet? – a game of consequential metrics

One of the questions we get asked regularly, not least by those who supply the money we use, is “how are you measuring your success?”  The truth is that the impact of what we do is quite complex and there is no simple measure we could use that is not extremely historical, and therefore meaningless for making decisions. The other factor is that we act at several points along a chain of consequent activities, and act in concert with many others.  Splitting out our exact impact would therefore be difficult and it would be problematic to conduct an experiment where we withdrew our support for some companies to see if they did less well!

That said, there are things we have done to make measurement of our progress easier.  When we started as a agency, one of the first things we did was to start at the “challenge” end of the process.  If you know what you are trying to solve as a problem, it is easier to judge if you have solved it – or at least made progress towards solving it.  If you are developing an answer and looking for a problem to solve, it’s a lot more difficult to test your progress.

We start with properly analysing the challenge we are addressing – the nature of the underlying system, the influencing factors and the value of a solution.  That involves a lot of interaction with the various communities that are affected by, or might be able to impact on, the challenge.  We ask them how they see the challenge and how they would tackle it.  We ask lots of people this question and assemble a composite view of what the challenge is.  It may sound odd, but the hardest part of the task is getting a proper definition of the question!

Once we have a defined challenge, we can get down to designing a path to address it.  Again we speak to lots of people – those in business, universities, government, both in the UK and overseas.  We look at how people are currently addressing this or similar problems.  We use the technique of “technology road-mapping” to collect all the ideas of different ways to address the problem, and put them on the same graphic – so that anyone can see the whole picture and judge, using their capabilities, the relative merits of the different routes!  Even with an agreed definition of the challenge, we find different companies address it in different ways.

Once we have a roadmap agreed with the community, we start designing competitions and other interventions to support those who want to address the challenge.  We try to define the intermediate successes that are necessary to advance towards overall success.  These can be either size or time bounded.  For example, it may be necessary to solve one problem before you can address another.  It makes sense to address that problem first.  Or the problem may stand alone, but be a very big one that will have to be solved towards the end of the process.  We might have several attempts to find answer to that sort of challenge.  We end up with each competition having its own “scope”.  This defines what we are looking for in a proposal – something that the roadmap suggests would move the UK closer towards success in addressing this challenge.

Within each competition, the proposals are assessed for the technical and commercial potential using a standard set of criteria.  Those that display the most compelling case are funded.  Since being able to articulate the problem and being able to answer it are different skills, we monitor the projects we fund.  Obviously, we track their progress against agreed milestones and budgets, but we also check that they display the characteristics of successful projects – that they adhere to their plans for management of the generated intellectual property, that they have their own internal control processes, and that they deal with the inevitable setbacks and changes in the external environment.  This increases the likelihood that the projects deliver on their aspirations.

We also need to measure whether those plans do actually address the challenge in a commercially successful manner.  Not many of our projects lead immediately to commercial success, so we work with the companies to monitor the impact of the projects for some years after they have finished.  Although it is early days (we started as the Innovation Unit within the Department of Trade and Industry in 2004 and were spun out as an agency in 2007) and most of our projects have only finished in the last couple of years, we are starting to see that the challenge-led approach leads to higher pay-back than simple support for the development of technology, and that the Innovation Platform, where we work much more in concert with other players such as Government departments and trade bodies, are even more effective at turning our investment into economic activity than would not otherwise have happened.

We know that innovation contributes to growth.  We believe that the sum of all the impacts of the individual projects leads to economic growth for the whole UK economy, but we are not sure how we can unambiguously measure our impact on that!

I started this post with an assertion about how complex measuring our progress was.  Over the last few years, however, we have developed this suite of integrated ways of measuring parts of the overall progress and are beginning to see how they mesh together and give us the feedback we require to do the job more effectively over time.  My only worry is the timescale of the direct impact of our activities is such that I might well have retired before our activities are demonstrated to be effective.  That said, we are becoming more and more aware of the secondary benefits of what we do – the boost in confidence, the connectivity we foster that has benefit we cannot count and so on.  But that’s another story……


Last updated on Friday 24 February 2012 at 10:05

  • Recent Comments

    Post your comment
  • Simon Aliwell|01/04/11 at 2:19 PM

    A nice illustration of just how difficult measuring impact in the innovation support game is. And you are absolutely right about the timescales. Some of the best case studies of success trace their history through multiple support programmes over 10 years or so. At least TSB looks like it might have some staying power!

Leave a Comment
*Required fields
Please enter the contents of the verification image. This is to help us prevent automated ‘spam’ comments.
Post comment
Copyright © 2013