home | strategy development | strategy skills | site index | access keys

Strategy Skills > Building an Evidence Base

Collecting data - Surveys

> in practice

Surveys are a means of developing a broad, representative understanding of a situation, social attitudes or prevalent behaviour.

It is helpful to first identify whether survey data is actually required or whether it is more appropriate to use data collected through other means such as focus groups, interviews with experts or practitioners or email discussion groups.

If a survey data is considered necessary, a search should be conducted for previous surveys that have been undertaken that could provide raw data required. The ONS's Guide to Official Statistics is a good starting point. If the data does already exist this would save considerable time and expense

If a survey is to be conducted, it may be necessary to commission a market research company to undertake the work. This can be particularly helpful if a large amount of data needs to be collected in a short period of time. The company will also have experience of what makes a good survey, and can feed best practice into its design. However, it will be expensive and will also take some time to tender for the job, design the survey and train the market researchers to conduct it successfully. This timing should be incorporated into the project plan.

Types of Survey Data

Most surveys contain cross-sectional data. This provides a snapshot at a point in time. A typical cross-sectional survey asks a random sample of the population the same questionnaire. As long as the sample is statistically representative, then it will give a clear guide to what answers the whole population would have given to the same questions. The larger the sample, the more confident you can be that the survey accurately represents the population's viewpoint.

Alternatively a longitudinal survey may be appropriate. These trace the same individuals over time. They may range from short-term panel studies, such as when the same people are asked the same questions before and after a big event, to comprehensive studies that track individuals - and even whole families or households - over a life-time, enabling causal links to be more confidently established than when based on one-off surveys. Longitudinal data can therefore be used to analyse the impacts of policy over time (for instance over an individual's lifetime or between generations) and also permit the analysis of how policy interventions may affect the future.

Things to consider when designing a survey

Designing a survey is a complex task and should usually be done in collaboration with a government social researcher or specialist market research firm. Before starting to design a survey, there are a number of questions that need to be considered:

• The purpose of the survey: a survey can either be descriptive or explanatory. A descriptive survey describes the distribution within a population of certain characteristics, attitudes or experience. An explanatory survey investigates the relationship between two or more variables. Explanatory surveys require that all variables that might be important are identified and measured during the data collection process.
• A structured or an unstructured approach: structured approaches are useful for hypothesis testing. Unstructured approaches are more useful for acquiring population data in an area where little research has been done.
• Quantitative or qualitative data: which type of data is more appropriate?
• The "population"  and "sub groups" to be studied: the sample to be surveyed needs to be carefully selected to ensure that the findings are similar to those found amongst your target population. There are three basic types of sampling:
  
  
1. Probability sampling. This includes random sampling, systematic sampling (similar to random sampling but some element of selection e.g. every 100th person in directory), and cluster sampling (e.g. pupils in a particular school). Consideration needs to be given to the 'sampling frame' - such as the voting register, telephone book etc. If the sampling frame is biased, such as richer people being ex-directory or poorer people avoiding the voting register, then this problem will be reflected in the sample.
   
   
2. Non-probability sampling. This can be useful when there is insufficient information about the population (i.e. there is uncertainty about how many people or events make up the population) or the population is intrinsically difficult to survey e.g. the homeless. Non probability sampling techniques include purposive sampling (e.g. the sample is handpicked) or snowballing (those identified for inclusion in the sample nominate others). Caution must be taken in generalising from such samples.
   
   
3. Stratified sampling. This involves dividing the sampling frame into segments and 'over-sampling' sections of the population. For example, a survey might deliberately over-sample young people or ethnic minorities in order to ensure that there are sufficient in the sample to make reliable statistical comparisons. Such samples can be 're-weighted' to give averages that are representative of the whole population. Stratified sampling is usually necessary for sub group analysis.

• Optimum sample size: the sample size needs to be an adequate size, in order to generalise from the survey's findings. Provided that the sample size is representative of the target population, the larger the sample size, the more confident you can be that the results are an accurate reflection of the population as a whole. The key factor is the absolute size of the sample, rather than the proportion of the population that gets included in the sample. Adequate samples can be estimated from the expected variation in the major variables of interest, and will therefore depend on the specific question or hypothesis to be tested. As a general rule of thumb, adequate samples will generally involve more than 30 events or people. Most market research companies use samples of around 1000-2000. However, other factors to consider when deciding on the sample size include the likely response rate, the desired level of accuracy, sub-divisions in the data etc. For example, if the survey seeks to discover not only the general attitude towards an issue, but also that of married men under 40, single parents etc, then a larger sample will be needed. Advice from a statistician or social researcher will help to ensure that the chosen sample size will yield reliable and relevant data.
  
  
• Data collection method: there are a variety of different methods for actually collecting survey data. Each has pros and cons:

1. self-completion postal questionnaires: this can be expensive and the typically low response rates, can result is a selection bias and hence doubt in the validity of the findings.
   
   
2. face to face interviews: market researchers may approach people in the street, or call at people's homes. On other occasions contact will be made in advance by phone or letter. Response rate is usually higher than for postal surveys but face to face interviews tend to be more expensive. Decisions will need to be made about whether the interviews are to be structured, unstructured or partially structured.
   
   
3. telephone interviews: these are quicker and cheaper than face to face interviews, but have the highest non-response rate because people are less inhibited about saying no over the phone.

Checklist for Designing a Survey or Questionnaire

1. Wording of the questions:

• Style of question should be suited to target group e.g. children or professionals. The table below provides some alternative styles.
• Respondents should only be required to answer about themselves, not others
• Avoid the use of leading questions
• Avoid asking the same question twice in different ways
• Avoid double barrelled questions
• Avoid double negatives
• Beware of ambiguous terms (e.g. lunch versus dinner)
• Make sure the wording is unambiguous and avoid jargon
• Keep questions short and straightforward

2. Are the questions in the right order?

• Getting the question order right will help the interview to flow.
• Remember that the nature of the previous question can affect answers.

3. Is the layout of the survey form/questionnaire clear?

4. Is the instruction to respondents clear?

5. Has a cover sheet been produced explaining purpose, return date, confidentiality, thanks etc?

6. Has access been granted from:

• appropriate authorities
• respondents

7. Has time been scheduled for:

• designing and production of an initial draft
• application for ethical committee approval and subsequent response
• piloting of an initial draft? Design of a subsequent draft
• the production of the subsequent draft
• numbering of questions
• respondents to complete the questionnaire
• pursuit of non-respondents
• collection and checking of questionnaires
• data preparation for analysis
• analysis of the results

Presentation of Survey Data

There are a number of tools that can help present survey data in a form that is easily understandable. They can be used to isolate important basic relationships, for example to understand any absolute differences in experiences of different population groups or sub-groups.

• Data can be presented in the form of a graph or table, for example a frequency table, block diagram, pie chart, frequency distribution or a histogram.
• Distribution and dispersion diagrams can be used to illustrate such concepts as the arithmetic mean and standard deviation.
• Descriptive statistics can be helpful in analysing data including the mean, maximum observation, minimum observation and other measures that describe how data looks.

Particular Types of Surveys

There are a number of survey types that are useful for public sector strategy work. These include:

• Customer Satisfaction Surveys
• Customer Priorities Surveys.

Customer Satisfaction Surveys

The level of satisfaction or dissatisfaction that results from an encounter between a service user and provider depends both on the user's expectations of the service they will receive and their perceptions of the service they have received. The leading model for thinking about satisfaction and perceptions of service quality focuses on whether the customer's expectations are "confirmed" or "disconfirmed" by their perceptions of the service they have received (see figure below). If a user's expectations are exceeded by their perceptions of the service they have received then the user is satisfied or even delighted. If their perceptions of the service fall short of their expectations then the result is dissatisfaction.

Expectations can be shaped by a number of factors:

• Personal needs. Each user of a public service will have individual needs that they expect to be met. This will vary from service to service and from customer to customer.
• Previous experience shapes expectations. For example, if someone has received excellent care from one GP they may have high expectations of another GP.
• Word of mouth and media communication. The experience of friends and family and the opinions of those in the media can be important in shaping expectations about the service.
• Explicit service communications. Printed material and statements from staff can have a direct impact on expectations. It is sometimes important to give a realistic assessment of the service the user might receive rather than raise expectations too high.
• Implicit service communications. For example, the physical appearance of buildings can be taken as a guide to the quality of services inside.
• Service reputation. The reputation of the wider service can raise or lower expectations about a single service encounter. Service reputation is determined by individual's perceptions of the their experience, the media and the reputation of the government.
• Personal beliefs and values. Expectations may also be shaped by people's personal values. For example, strong supporters of public services may be more forgiving of poor service.
• Nature of client group. It is thought that the social class, age and ethnicity of the client group tend to strongly influence people's expectations. For example, older people are consistently more satisfied with the health service, while richer people are less satisfied. It is thought that part of the explanation lies in the differing expectations of the better off and the elderly.

Similarly the perceptions of the service received by the user may depend upon a variety of factors including: access, aesthetics, attentiveness, availability, care, cleanliness, comfort, commitment, communication, competence, courtesy, flexibility, friendliness, functionality, integrity, reliability, responsiveness and security.

The 'in practice' example shows how the Communidad de Madrid conducted a gap analysis using this technique to drive service improvements.

Customer Priorities Survey

This approach enables satisfaction with different aspects of a service to be directly compared to the importance the customer attaches to each of them. By mapping satisfaction against importance areas of the service most in need of improvement can be identified. As can be seen in the figure below, the service provider can identify and focus action upon elements falling into the bottom right quadrant.

This approach can be applied at several levels:

• Inter-service priorities. To compare public priorities between different services.
• Intra-service priorities. To determine which aspects of a service are priorities for improvement. For example, existing surveys ask about importance of various factors. For General Practitioners the appointments system is one of the main areas of dissatisfaction mentioned to be in need of improvement.

Strengths

• A breadth of issues can be covered in a survey.
• Providing the sampling is sound, it should be possible to generalise the findings.
• Lends itself to quantitative data.
• Can assess how far the methods used are replicable (precise), accurate (approximate the true value of the quantity sought), and valid (represent the variable to be quantified).
• Gap analysis allows both individual aspects of a service encounter to be analysed separately and perceptions of the service overall to be measured. Thus individual aspects of the service (say, staff friendliness) can be isolated and singled out for improvement.

Weaknesses

• Data produced can lack the depth, detail and colour of, for instance, the case study approach.
• Difficult to check accuracy of responses or follow-up ideas, although cross-validation can be conducted (such as objective measures on a sub-sample).
• Causal inferences from survey (explanatory) research are generally less reliable than from experiments.
• Individuals are very different and may come to a service with very different expectations - for example more deferential people may arrive with lower expectations than those with more assertive personalities.
• In judging the overall service encounter different aspects of the service may differ in their importance for the consumer - for example in one service reliability might be more important than responsiveness, while in another reliability might be expected and therefore discounted by the service user. This can be handled through weighting different factors.

References

The Magenta Book on Policy Hub

Approaches to Social Research, Royce Singleton et al.

Denscombe M. (2003) The Good Research Guide, 2nd Edition. Open University Press, 0335213030

Research Design, Catherine Hakim

Statistical Methods in Medical Research, P. Armitage and G. Berry (Oxford: Blackwells; second edition 1987)

Buttle F (1995) SERVQUAL: review, critique, research agenda. European Journal of Marketing 30 (1) pp8-32

The Strategy Unit paper, Satisfaction with Public Services

^ top