Is it better to ask permission than to seek forgiveness?

There are many questions I have run into during my career, but the most vexing has been that of how to value research. So I was interested to read a well-balanced piece in Times Higher Education last week.

As a research manager/director in several organisations, I was often asked to justify investment in research (or at least, the parts that I was responsible for!). The question was usually combined with a discussion on research metrics in general, and led to a significant amount of work seeking a simple number which would indicate whether we were doing the right things – and doing them properly.  There are a number of points I have learnt through these various quests.  

The first is that before you measure something, you have to know what you are trying to achieve.  If you are in a business that develops new products, you need to go back to your overall strategy and ask what success would look like.  For example, are you going for market share – in which case, you need to work on developing attractive products that can be made cheaply so that the business can undercut the competition to gain success?  Or does your strategy require you to produce the most innovative products and charge more for them – in which case the cost of manufacture is less important than your speed to market? There are many strategic options and each one requires a different approach.

Once you have aligned your metric with your goals, the challenge doesn’t get any easier. You need to make sure the measurement you choose gives an indication to action if it goes in any given direction. If you start to underperform, do you know what to do to get back on track? If you exceed your goals, do you know what action caused this overperformance, so you can do it more?

Then comes the trickiest aspect of all.  Is your metric simple enough to make sense to the whole organisation so that they know what they need to do to deliver? This is where many measures of performance slip up. The ability of nearly everyone to “game” metrics is legendary. The Times Higher Education article quotes the explosive growth of spin-outs from universities once that became a metric of success. The fact that many might be “hobby companies” to get their founders a Porsche is not considered in the raw number and therefore it is easy to miss their underlying goal. In industry I saw how wasteful rewarding employees for producing patents can be, when our patent portfolio - and its associated costs - ballooned but no new products or processes were produced!

At one point I was “asked” by my employers to talk to their annual conference of accountants. Asking a research person to speak at this event was on a par with throwing slaves to the lions, but it proved to be a useful, if somewhat painful, experience. What I learned on that day near Richmond over 10 years ago was that accountants are not particularly bothered if past investment in research has led to commercial impact. After all, the money has been spent and cannot be recovered.  What they care about is whether the research you are currently doing is focused on outcomes of value. This is a different question (and one asked in the article).  What it led to in our case was a more rigorous front end to select and prioritise projects and a process, shared by the whole company, to regularly check that the projects were still aligned with strategy and still on track to deliver the promised gains in product attributes, process flexibility or savings and so on. Even as we implemented it, we knew we were not setting the world on fire. The building blocks we used had been in those management books you can buy in airport bookstalls on “how to be better at…” catalogiung the approaches used by successful companies from the preceding decades. It was agreeing a simple and appropriate system that everyone bought into that made it work.

That this complexity existed at company and group levels suggests that the task of determining whether national programmes are “of value” might be a long and arduous quest. To analyse whether a particular research programme has delivered value – be it economic or societal – to the nation will run into all the problems that industry has struggled with for many decades – but with a scaling factor that is difficult (for me at least) to comprehend.  I will, of course, admit to an interest here.  The article focuses on investment by the research councils into the underlying knowledge base.  However, even as a new organisation we have been in front of the Innovation, Universities, Science and Skills Select Committee to discuss similar issues - an interesting experience which reflected the sort of things our Government sponsors might expect of us.  One section (quoted from the minutes) is of relevance to the issue of metrics. Iain Gray, our Chief Executive, was being asked about impact.

Q19 Dr Harris: Do you feel that you need to demonstrate that you have had an impact already?

Mr Gray: Yes, because I believe that as an organisation we can do so much more. To do so much more we need to be able to get messages across quite quickly. The problem and challenge, not for us but for every country involved in innovation, is it is a long-term game.

Q20 Dr Harris: I was attracted by what you wrote in your strategy for business innovation where on UK innovation trends you said it is very hard to tell what impact you have had on that because measuring your impact is so long-term and the very nature of innovation makes it difficult to do that, it will take time for you to be able to measure that. I think you were forced to say that anyway because it is true, and I thought that was quite honest, but very few organisations can just say, "No, we can't really tell and I am not going to sit here in front of you and cite some business survey that purports to do that because we have to develop the metrics, establish the baseline and then check on a long-term basis". Can you see why I am a little disappointed that you leapt straight in somewhat defensively with your view "we are having an impact" instead of sticking to your guns and what you wrote here?

Mr Gray: That is a good bit of counselling.

Dr Harris: You cannot win obviously!

Dr Gibson: Whatever you say you are dead!

Even aside from the good-natured way it was carried out, that last exchange suggests that everyone recognises the complexity of our situation. It also suggests that recent speeches by Lord Drayson, John Denham and Gordon Brown (again referenced in the THES article) are serious attempts to find a way to frame the right question. 

It does all remind me of those accountants.  The question is not really whether past research investment has been of value, it is all about the decisions we make going forward. And this is where it gets really difficult. When you are in a company with a defined theatre of operations, selecting areas for focus and judging progress towards impact is relatively easy. By the time you get to corporations, the search for diversification (which was popular in my day) means that any idea that could make money is worth evaluating. I was lucky enough to work in BP when they had the Venture Research Unit.  Their annual meeting was a smorgasbord of new ideas, and the people they funded among the more interesting scientists I have ever met. Many of the projects have gone on to be the bedrock of new areas of work, so I have real sympathy with the view that preselecting areas of research on the basis of existing products or processes can be limiting. 

Venture Research was quite cheap, but it opened doors to new processes and products. The cost of development was higher, as it is with all new developments. There is a rule of thumb in R&D that the cost scales up by factors of ten as the idea is taken through development, and that starting lots of things and selecting them on the basis of relevance/excitement/value at various stages can balance the need for real novelty with the need for delivery. Perhaps we should be asking not whether the front end of the process is focused, but whether the selection and development of all those potentially valuable ideas is appropriate for our needs?


Last updated on Wednesday 12 August 2009 at 15:20

  • Recent Comments

    Post your comment
  • r4 revolution |31/07/09 at 12:50 PM

    I don't believe in forgiveness myself..

    David Hughes|31/07/09 at 12:48 PM

    As usual, Dr Bott makes some interesting observations based on real practical experience. He suggests that the question is not really whether past research investment has been of value, it is all about the decisions we make going forward. He suggests that this is where it gets really difficult. But using the latest ideas of innovation management it does not have to be difficult. What we need to do is to get away from the idea that product development is an uncontrollable process. After all we now have six sigma manufacturing processes so why not expect similar performance from the new product development process? How might we do this? There is not room in a blog to go into details but here is a flavour of the key steps that the latest thinking on Innovation Management suggests: • Start at the customer end. For existing product markets identify the ‘job’ the customer is trying to do (rather than comparing existing products), what outcomes are required and how these are valued. Use this to seek out disruptive opportunities. For new markets think about ‘non-consumers’ -- for example as successfully demonstrated by Nintendo with the Wii which was aimed at non-gamers. • Recognise uncertainty in the business plan. What are the ranges of outcomes in sales/ development costs? This can be factored in to a financial evaluation using ‘option pricing’ techniques. Using such an approach also enables higher risk/ reward projects to progress to the next step as there may be a value to ‘keeping options open’ even if the project is subsequently abandoned. • Use a rigorous Project Management approach including independent phase gate reviews. At the phase gate also ensure that the latest business case assumptions are tested and verified independently. Don’t hesitate to kill projects that are not meeting their potential. All the above could well be applied to TSB projects because they should be in the applications phase not in the discovery research phase. If anyone is interested in exploring these ideas I can be contacted through the Business Innovation Group website ( )

Leave a Comment
*Required fields
Please enter the contents of the verification image. This is to help us prevent automated ‘spam’ comments.
Post comment
Copyright © 2013