Do we know what we are measuring?

A couple of weeks ago I attended the launch of the pilot Innovation Index at NESTA, which highlighted again an issue which is never far from my mind: how to measure the impact of innovation initiatives and the importance of metrics to the innovation landscape.

Since joining the Technology Strategy Board two years ago I have found the question of how we measure the impact of our investments and activities amongst the hardest to answer. In fact the difficulty of the impact measurement question is widely recognised - but that doesn't mean we shouldn't try to answer. Indeed we have an accountability for investing taxpayers money; they have a right to ask the question, and we need to know the relative impact of different sector investments and tools, to aid our own decision making. And so we are focusing intensely on developing our own answers. There is a lot we can and do measure already about the impact or specific programmes, but we need to get closer to the heart of the matter.

I have argued that we do need to keep it simple. For example, I have always thought that a straightforward measure could be the number of large multinationals who have made investments here in the UK and who would cite our interventions as making a difference; or the number of high tech companies in the Tech100 index who would cite the Technology Strategy Board as helping them to the next stage of their business development. I have strong anecdotal evidence, for example, that the Low Carbon Vehicle Demonstrator Programme investment influenced at least three major corporates to make investment decisions here in the UK that were potentially aimed elsewhere.

There is little doubt that what gets measured gets done - but there are also numerous stories illustrating how measuring a particular parameter has had exactly the opposite effect to that intended.

One typical scenario is the call centre.  The first obvious metric to apply is the number of calls answered.  But if you incentivise people along these lines, they may tend to simply cut off a percentage of the calls, working on the principle that the customers will call back - adding to the number of calls answered.  There are many case studies along these lines, usually involving individual rewards that end up driving the wrong behaviour. 

I have come across three simple rules for good metrics.  The first is that they should measure the path towards the strategy.  Whatever your strategic goal, the metrics should somehow measure how you are doing in achieving them.  The second is that the action necessary to reverse a bad trend should be clear.  One of 3M's measures is the proportion of sales attributable to new products. If the measure goes down, the response is easy - launch more new products. The third rule is that metrics should be few and simple.  A good measure is owned and recognised by the whole organisation and enables all within it to work together to achieve the strategic goal. So finding the right metrics is important.

The pilot Innovation index just launched by NESTA is a very useful contribution, aiming to develop and deploy significantly improved measures of innovation in the UK. It measures innovation 'in the round', much more broadly than the current measure of R&D, and includes factors such as product design, training in new skills, organisational innovation, developing new customer offerings and brands, and copyright. NESTA has referred to this as 'hidden innovation' which is as important to productivity as R&D.

This first Index is a strong step towards measuring how much the UK invests in innovation per year, what this investment contributes to economic growth, and how favourable the wider social, economic and political conditions in the UK are for innovation.

The findings are valuable. For example the Index demonstrates a direct link between innovation investment, productivity output and business growth.

The Innovation Index will evolve over the coming year in terms of methodology and scope, and I am sure that it will have an influence on our own metrics, but I do not think it will provide all the answers we need as an organisation.  For example one key parameter for our investments is what is being termed 'stickiness' - that is, stickiness to the UK - but what exactly does this mean and does it lead us to the right metric?

A number of forecasting organisations already produce high quality reports on the economic impact of certain sectors in the UK. I have seen reports on the importance of aerospace, RTOs, and the chemistry sector, to name a few. I don't doubt their conclusions, but it is unclear how these analyses can be built into an overall UK economic picture. Again they provide some clues - but can these help us determine the impact of our own interventions?

A good example in our own metrics debate is the relaunched SBRI (Small Business Research Initiative) programme, which we were discussing with our Governing Board a couple of weeks ago. David Connell has been one of the principal commentators arguing the benefits of SBRI, the main arguments being based on benefits seen in the US where a similar scheme has operated for some 40 years. We presented progress on SBRI to our Board - and it was a very positive story.  22 new competitions have been launched this year, across eight government departments - resulting in 296 contracts, together worth some £20m - mainly to small and micro businesses, many of which have never interacted with government contracts before. But the metrics we shared on SBRI are still mainly activity based. Our intuition says that the government procurement of R&D against a defined 'need' is a highly effective way of promoting innovation in small businesses, leading to subsequent commercial success - but as yet our metrics don't measure this impact.

The principal impact measure must be economic - whether it be contribution to GVA, or to jobs. Last week I presented to the KTP Managers' conference in London. The Knowledge Transfer Partnerships scheme is one of our longest-standing initiatives and we have a reasonable set of metrics around it. For every £1 million of Government investment in Knowledge Transfer Partnerships, the benefits achieved by UK businesses amount to over £3.6 million increase in annual profits before tax, the creation of 32 genuine new jobs and 189 company staff trained.

This is quite compelling but cannot be the whole picture; economics may be the primary metric but it isn't the only one. We are considering using a variant of the PESTEL analysis across our programmes to set targets, to estimate the impact at time of submission and then measure impact through the fullness of time. We have to be careful though. Firstly we will get what we measure - the metrics do need to be meaningful. Secondly we don't want to create an industry of measuring things. Indeed, the last thing business wants is a bureaucratic government body seeking lots and lots of measurement information; they would argue that as long as they have their own investment in the game they are heavily incentivised to succeed in their own business and that providing metrics information to government provides little additional benefit.

So, it is not an easy question. We need to keep our metrics simple, meaningful, and focused on economics as the primary impact - but at the same time develop them further to find the right spread of measures at the right depth to get to the heart of how we drive innovation. I would welcome your views to help us in refining our thinking over the coming months.


Last updated on Friday 24 February 2012 at 10:16

  • Recent Comments

    Post your comment
  • Tony Smee|30/04/12 at 10:54 PM

    Hi Iain,
    I come from blogging on a dozen other sites, linkedin, innovation & engineering discussions, national newspapers and specialist sites such as stationary engine. These sites have lively debates on all kinds of engineering and technology issues, hundreds of posts and comments each day.
    Now there’s one big issue in the UK today and that is how to stimulate growth, how to connect small businesses with the “cutting edge technology” from our universities. How to sell more British products and services overseas. I go Nesta, I go to Technology Strategy Board & _connect and I’m the only one there, apart from the staff posting press releases, and praising each other. I make provocative comments, no reply, I give links to information or facts from my own long life as an electrical engineer, but hardly any response. I do get a lot of views on my blog, and quite a few personal e-mails. Does anyone at TSB think this is something to be addressed? Perhaps a measurement of interest index could be produced, it might highlight the lack of interest in this site by the top management.
    The real measure of course is that £2billion has been put into the TSB and there has been no growth, no jobs, and not one major innovative idea that has shaken the markets. Always vague statements such as multinationals would say that the TSB money was useful, and some businesses would say the TSB helped them to the next stage of their development - pretty meaningless hype.
    You might notice my comments on TSB and my blog where I get more views than most of the articles. My lobbying is mainly for the lone engineering inventor who is blocked from government support or any funding for proof of concept or prototypes. My blog “Why _connect is not working” It tries to show that in fact engineers and inventors are just the kind of guys who should be supported.
    My own way around these barriers is to find universities who think my ideas are worth pursuing and so far I have found three with another three still considering. Aberdeen and Heriot Watt are the most advanced with proposals ready to apply to ITF the oil & gas industries innovation firm, who have already expressed interest some 6 months ago. Both these ideas are solutions to oil & gas problems from my own experience and I have demonstrated the basic principle with models & simulation. I would estimate and hope we have a 90% chance of success and the value to the oil industry if fully successful is in billions of pounds.
    I have an unanswered e-mail to David Bott at the moment giving specific technical detail of the proposals going through at the moment and asking if the TSB has no mechanism to support such innovative ideas then what is the way forward. I can forward a copy if it is of interest.

    Tony Smee|16/04/12 at 1:06 AM

    Nesta has in their title "making innovation flourish", we don't need metrics to see that in reality in some 14 years of making innovation flourish, it is not flourishing.
    I feel in fact that innovation has flourished without Nesta or any other body. What is missing in my experience is the funding for trials, prototypes, proof of concept experiments and exploitation of the innovation.
    I have to share a story I have posted with a couple of national newspapers & David Bott's blog, which tells of my own experience:

    This is the story of a nation that went from workshop of the world to a nation in debt.

    Clement Atlee recognised the opportunities created by the war and formed the National Research Development Corporation (NRDC) in 1948. Then in 1975 the National Enterprise Board was founded “to stimulate innovation and create jobs and growth”.
    Now in 2004 we have the Technology Strategy Board with the target “to stimulate innovation and create jobs and growth”.
    So why after 60 years of stimulating innovation to create jobs and growth has very little happened, in fact most of British industry is foreign owned, unemployment rising and growth hovering around zero.

    I have lived through it all, Clement Atlee, Harold Wilson, Margaret Thatcher, and the words do seem remarkably the same. We were always going to improve education in maths & science, more apprenticeships, investment in new technology, more exploitation of our cutting edge university research and cutting down red tape and barriers to innovation.

    Here is one story out of hundreds of similar ones. There is an idea being developed at Aberdeen University to seal an oil well blowout in a much shorter time than is normal in the oil industry. Eagerly grasped and understood at the engineering and academic level this has been blocked by the slow funding processes at the top. The idea was proposed to BP at the time of the Macondo Gulf of Mexico disaster and considered but not implemented. We think the technique has a 90% chance of success, so if it succeeds it means that the Gulf of Mexico oil leak could have been sealed in 3 weeks instead of 3 months, perhaps an embarrassment for BP. A typical innovative idea, but who will check it out, who will fund a proof of concept?
    Will the funding be forthcoming for this experiment? The people at ITF the oil industry’s own innovation body feel very positive, so do the academics at Aberdeen University, we chat on the phone & e-mail regularly . But from past experience the answer to the funding is "probably not".
    A 90% chance is too much of a risk for the people with the cash.
    In fact what we need are people who will take a risk with a 50% chance of success or less, because every big world beating idea by its nature looks risky or even crazy to most people. If a jet engine principle was obvious then Frank Whittle would not have had a hard time convincing the RAF top brass, if everyone could see the possibilities for a cyclone vacuum cleaner then James Dyson wouldn’t be where he is.
    If the above story is typical then we are not short of innovation or ideas, just the risk capital to see if they work or not.

    Dr Martyn Polkinghorne|15/12/10 at 4:09 PM

    Very interesting.

    Thinking about KTPs, they are supposed to transfer knowledge, but how can we ever be sure that KT has occured?

    We can look at a project to see if it has the potential for KT (I've been working on a tool to help KBs to do this in terms of organisational and personal knowledge, and peer2peer and expert2novice knowledge transfer processes), and we can see the results of growth that may occur in response to successful KT which implies that it did happen, but will trying to see (and/or measure the actual) KT itself always remain an impossible dream?

Leave a Comment
*Required fields
Please enter the contents of the verification image. This is to help us prevent automated ‘spam’ comments.
Post comment
Copyright © 2013