Evaluating Impact - Education Research Paper No. 35, 1999, 262 p.

[Previous Page] [Table of Contents] [Next Page]

6. TOPICALITY VS SUSTAINABILITY

6.1 A consideration of project assessment: topicality vs sustainability
6.2 Topicality vs. sustainability in the evaluation of the South African Book Aid Project

6.1 A consideration of project assessment: topicality vs sustainability

Jeff Samuelson
Book Aid International
Sarah Harrity
Book Aid International

Book Aid International is an NGO based in the North with no overseas offices but with strong links with a wide variety of educational institutions, organisations and associations in the South. Book Aid International (BAI) works in partnership with these organisations, predominantly in sub-Saharan Africa, to support literacy, education, training and publishing, by providing books and other reading materials - which help people to realise their potential and to contribute to the development of their societies.

As part of its core programmes of book provision, BAI also manages a number of projects. Brief reference will be made here to two cases which will be used later to illustrate the points under discussion.

The first example, of a project now completed, concerns the purchase and subsequent distribution in 14 countries in Africa, of a booklet entitled Living with AIDS in the Community. The text was written by The Aids Support Organisation (TASO) in Uganda and was published by the World Health Organisation (WHO). The idea for this project originated with the Malawi National Library Service which was charged with the responsibility for developing an AIDS awareness campaign in Malawi through the network of libraries. The Director of the Library Service asked BAI, a long-standing partner, if it could provide appropriate materials and the project began to take shape when it rapidly became apparent that the materials produced in Britain would not be appropriate for Africa.

The second example which will be drawn upon in this paper is another project in which Book Aid International is a player, namely the South Africa Books Aid Project (SABAP), which aims to support local initiatives to improve the quality of basic education, including adult education, in three provinces. One of them is the Eastern Cape which is discussed by Cleaver Ota in the next paper.

2 Considering the distinction between

Before looking at the evaluations of these projects itself, it may be worthwhile spending just a few moments on defining specific terms and what they mean. The literature on monitoring and evaluation uses a number of terms about which there is often no universal agreement but, for reasons of clarity, it seems important at the outset to define what we mean by them for the purpose of this paper.

· Defining Impact

Impact, therefore, we shall define as the longer-term effects of a project or programme, effects which are brought about by change and which outlive the project. The evaluation of SABAP in Eastern Cape did not, by its own admission, seek to evaluate impact but focused rather on the short term achievements or outputs:

The domain of enduring change is a long-term process which implies that SABAP cannot be expected to produce the desired impact in the period of one year. (SABAP: 5)

· Defining outputs

Although there is no final definition of "outputs" it is here argued that if an evaluation focuses more on outputs, in addresses those specific achievements which the project design was supposed to guarantee. By their nature these may be described as questions of topicality rather than of sustainability. There are good reasons for this, having happened in the SABAP evaluation – the most important being that implementation of the project had only just been completed and it was not therefore possible at that stage to make a judgement about possible long-term effects. It was therefore a formative evaluation – a very useful one – and, to use Cleaver Ota's own words, 'it concentrated on guiding ideas, change in infrastructure, theory, methods and tools'. The purpose of the evaluation (which Book Aid International had helped to shape) was not only to find out what had happened in the SABAP project in the Eastern Cape but, just as importantly, to learn lessons which can be applied in the two subsequent phases of the project.

Three major implications of such an evaluation may be singled out:

· It is a health check – it provides an opportunity to assess the project in mid-stream and to see to what extent practice conforms to the theoretical design of the project.

· It can, and should, inform subsequent phases of the project. This is particularly true in the case of SABAP where implementation is being carried out on a sequential basis - Eastern Cape in Year 1, Mpumalanga in Year 2 and so on.

· It allows scope for change, for example to alter or amend the project outputs and activities.

The evaluation of SABAP in Eastern Cape concentrated, as we have seen, more on outputs than on impact but the evaluator was nonetheless able to conclude that 'the foundation of enduring change had been laid' (SABAP: viii).

This leads us to the question of sustainability and here again we attempt a definition of this concept. We believe that the meaning of sustainability comprises two aspects:

· The first is changed perceptions in individuals, the possibility of thinking differently, and perhaps more positively, about the situation that the project was designed to assist. This includes an understanding or appreciation that change and development are possible.

· The second is the extent to which the project activities will continue after the donor's financial support has been withdrawn.

4 Ensuring sustainability and measuring impact

What are the implications of trying to ensure project sustainability and measure impact?

4.1 Ensure stakeholder involvement

Firstly, is the need to ensure that all the stakeholders are actively involved in the project from the outset; not the point at which implementation begins but at the much earlier planning and design phase. The overall assessment of the AIDS awareness project was positive but at the same time one of the conclusions of the report¹ nevertheless drew attention to the fact that 'the project and its evaluation would have benefited from more detailed consultation with partners and local participation at the project design stage'. The need for inclusivity when thinking about evaluation at the design phase is illustrated from one of the lessons learned from this project:

An important issue is the level of consultation over new initiatives and active local participation in their design and planning. Informal consultation took place during the development of the project, but the project was not discussed formally with all partners until it was due to begin. Although those organisations that became directly involved in this project chose to participate, the origin of the project was not clear and it seems likely that many were not fully aware of what exactly would be required in its implementation and evaluation.... The lack of local ownership of the project is clearest in the evaluation, where input on the planning of the exercise was requested but very little received (Report on AIDS awareness project in Uganda).

4.2 Clarity of focus

Secondly, there should be absolute clarity about what is to be evaluated. The original intention of the evaluation of the AIDS project was to assess behaviour change as a result of the use of the booklet. However, it became clear that such an attempt (which would have been a true impact study), had to be scaled down to something more feasible but still useful. In short, a triumph of the pragmatic over the ideal. What was considered instead was the difference that the participating organisations felt the availability of the booklets had made to what they could do.

The same consideration applies to the core book provision programmes managed by BAI. works with a great variety and number of partner organisations all of which are selected against a set of criteria that has been developed over time and which were formally written up in 1996. The criteria concern matters of need, role, mission, objectives, access and use. BAI constantly monitors the activities and outputs achieved within these partnerships and is now recommending in an internal review of its monitoring and evaluation activities that a formal evaluation by an external evaluator should be carried out with one or more of the major partners. It would be perfectly feasible to evaluate the whole process of providing books (including the vital question of whether they were the right books) to, say, the Kenya National Library Service and the effect of the programme on the services delivered by the KNLS. Such an evaluation would be a formative one but, because of the close links and the fairly intensive monitoring activities, might not throw up many original findings.

It would be very much more difficult, time-consuming and expensive to undertake a true impact study, in other words to attempt to assess the impact of the provision of books on, say, the educational achievements of individual users of these services. One obvious difficulty, but only the first one, would be to disentangle the books provided specifically by BAI from others in the KNLS book stock. Notwithstanding these difficulties, BAI intends to attempt an evaluation that would examine not only outputs but impact as well.

4.3 The need for baseline data

Thirdly, it would seem difficult if not impossible to measure impact unless there is first a baseline study against which to measure the eventual outputs of the project The obvious implication of such an approach is that the timescale of the project would inevitably be extended unless only the most readily available statistics were used The disadvantage of such statistics is that they are usually quantitative in nature and do not reflect the much more complex situation that the project is trying to address.

For example, the goal of the SABAP project was 'to support local initiatives to improve the quality of basic education' and this reflects the emphasis rightly placed in most projects now on issues of quality rather than quantity. An extended timescale needed to address issues of quality at the stage of the baseline study, in its turn, implies a greater overall project cost.

4.4 Conceptualise the evaluation in the design phase

Fourthly, is the need to consider from the earliest stages of the project what is to be evaluated and, at the design phase, to build in appropriate indicators that will enable the measurement of impact All stakeholders should be involved in this process Both these factors – inclusivity (referred to in 4 1) and early planning - should help to ensure that all the stakeholders will be committed to the evaluation process and that the evaluation will be an integral part of the project and not a kind of appendix added on at the eleventh hour This will be important if, as should be the case, the evaluation is a means for all the participants to learn lessons from the exercise.

4.5 Consider other factors that might impact on the project

Fifthly, the complexity of measuring impact is increased by the necessity of evaluators gauging the extent to which the project intervention itself (as opposed to any number of external influences), has caused changes to happen The impact assessment therefore has to consider other factors including the political, social and economic context in which the project has been operating.

This calls for a different kind of health check from that mentioned in the context of a formative evaluation A project or programme may be good, if that description is permitted, within its own terms but may have unintended and perhaps what are perceived to be negative consequences elsewhere. For example, the core book provision programmes managed by BAI are designed to meet immediate needs and, judging by the feedback, they do serve this purpose According to a report issued this year by the Malawi National Library Service, more than half of the additions to stock in 1997 originated from BAl. The negative conclusions which could be drawn from this are:

· that a dependency culture is being created

· that the provision of these books from an external source is inhibiting the development of the local publishing industry.

In its reply, BAI agreed that supporting the local book supply chain was essential to sustainable book provision in Africa and highlighted some of its own work in that area At the same time it argued that books were not yet an affordable commodity in Africa and that, in order to develop a reading culture, short-term needs had to be met alongside investment in the local book industry It concluded that, at every stage, there was a need to develop imaginative and realistic ways of getting books to those who need them.

Even when the above implications of trying to ensure project sustainability and measuring impact have been taken into account, there still remains what is perhaps the greatest difficulty of all, which is knowing to what extent the project intervention itself, rather than any number of external influences, has caused changes to happen The impact assessment therefore has to consider other factors including the political, social and economic context in which the project has been operating A book, or the information contained in it, may be necessary to pass an exam for example, but it may not be sufficient Other factors, such as the facilities provided by the school, the quality of the teaching and the degree of parental support, might all be equally important in the process It therefore becomes extremely difficult to isolate the book itself from these other influences.

4.6 Serving multiple stakeholders

And finally, there is the extent to which possible demand for rigour and for proof of impact is linked to accountability to the donor rather than to project or programme development In the case of the AIDS awareness project, the evaluation was donor led As we have seen, this is not the ideal way of doing things but, on the positive side, useful lessons emerged from the experience which BAI has fed into subsequent work Can an evaluation serve both the needs of the donor who wishes to be assured that the money has been well spent and those of the other stakeholders whose needs are of a different order? As things stand, it has to serve both purposes and to be not only a demonstration of the goal achieved but also a learning exercise that will illuminate future activities beyond the project's life-span and inform comparable projects elsewhere.

5 Conclusion

While assessments tend to focus on outcomes rather that impact which outlives the project life-span, it is nevertheless possible to answer questions pertaining to issues of sustainability. The lessons learned as discussed in the previous section, are posed as suggestions for overcoming limitations imposed by assessments which would normally focus on the measuring of intended project outcomes.

Footnote

1. The project was evaluated with the Kenya National Library Service (KNLS), the Ghana Library Board and two local NGOs in Uganda, among others.

6.2 Topicality vs. sustainability in the evaluation of the South African Book Aid Project

Cleaver Ota
University of Fort Hare
South Africa

The new legislation and policies in education in South Africa firmly anticipate the establishment of self-managing schools. One of the projects which has the primary objective of assisting the development of self-managing schools was the Quality Schools Project.¹

The South African Book Aid Project (SABAP) was implemented in the Eastern Cape and was located in the Quality Schools Project. The strategy of SABAP was on whole school development. This included the establishment of school governing bodies, increased parental and community involvement in schools, and INSET for teachers and principals. An additional feature of the project was the creation of District Education Resource Centres (DERCs), whose main function was to provide library services for the surrounding communities and schools.

SABAP, a DFID-funded project, may be distinguished by its multi-partnered implementation. In the Eastern Cape, the partnership comprised the Eastern Cape Department of Education through the Provincial Libraries, and Information Service (LIS) Directorate. The project was managed in the UK by Book Aid International (BAI), and was implemented by the Institute of Training and Education for Capacity-Building (ITEC). ITEC, together with Read Education Trust (READ), were responsible for the training.

This paper considers the modus operandi surrounding the evaluation of this multi-partnered project. It then discusses the attainment of the outcomes that were defined in the logical framework. The paper finally concludes by asking why in the face of its successful attainment of outcomes, the project's prognosis for sustainability is not very positive.

2 Purpose of the evaluation

The purpose of the evaluation was to determine the extent to which SABAP was considered to be of value to key stakeholders and critical interest groups associated with the project. In order to do this, it was necessary to assess the extent to which SABAP:

· improved access to books and other materials

· provided effective training in library resource management

· created community-based structures that would be truly involved in the project

· created effective partnerships that would facilitate the transfer of relevant knowledge, skills and values.

2.1 Assessing impact of SABAP – ways of looking

The short-term objectives of SABAP were to ensure that there would be:

· better equipped primary school/resource centres/community libraries

· a trained and effective staff in those libraries

· an improved community understanding of the importance of books

· an enhanced capacity in the Provincial Libraries and Information Service, a directorate in the Eastern Cape Provincial Education Department, to manage the project after the pilot phase.²

· the relevance and suitability of the materials provided

· accessibility of the materials, including the system for distributing materials to the cluster of schools

· the use made of the materials

· the provision of training for the setting up and maintaining of systems to ensure administrative efficiency and the effectiveness and security of the materials

· training to stimulate an appropriate use of the materials

· the extent to which resources are shared and the community is involved in the project

· any aspect of the project that could contribute to its sustainability.

The evaluation guidelines contained in the terms of reference suggested important considerations. These included the necessity for the evaluation team to be balanced in terms of gender, for the evaluation enterprise to contribute to the building of capacity, and for the evaluation exercise to be as participatory as possible. In addition to the criteria specified in the terms of reference, the assessment was influenced by our understanding that an evaluation is

[It is] a type of disciplined inquiry undertaken to determine the value (merit and or worth) of some entity – evaluand – such as a treatment, program, facility, performance, and the like - in order to improve or refine the evaluand (formative evaluation) or to assess its impact (summative evaluation) (Lincoln and Guba 1989: 50).

2.3.1 Review of documents

A vast amount of project documentation had been accumulated during the years of implementation. It was necessary to select documents which would speak to our evaluative questions. We were able, from the various progress reports, to gain a sense of the progress and the timing of such progress that was being made in pilot schools. In addition, the records gave an indication of the gap between targets and achievements. The documentation also provided background data which enabled the SABAP intervention to be located within its context. We accessed documentation in the form of annual reports, minutes, progress reports and other literature which was relevant to the project.

A documentary study was considered to be the most appropriate way of attaining a sensitivity to the what had happened in the project. Thus, for example, minutes answered questions about processes and functioning while annual reports, and progress reports provided an understanding of the changes as they had occurred over time.

The documentation also provided an opportunity to conduct a cost analysis of the project.

2.3.2 Focus groups and one-on-one interviews

One-on-one interviews and focus group discussions were conducted. We selected qualitative methods which would enable us to explore the perceptions of the various players. In particular, the focus discussions and interviews were useful in enabling us to gain insights into the participants' perceptions of quality. This was considered pertinent insofar as it enabled us to assess the extent to which these critical interest groups considered that the project had or had not achieved the stated objectives.

In addition, the interviews were beneficial in this evaluation since they:

· enabled the interviewer to probe for more specific answers and repeat questions where it appeared that the questions had been misunderstood

· enabled the interviewer to observe non-verbal behaviour. The paralingual cues in the discussions often gave the researcher an indication of other dynamics.

· facilitated spontaneous responses.

The data obtained in the interview process were verified through the process of triangulating the data. This facilitated our understanding of which targets had been attained and which had not.

The discussion which follows under the headings listed here reveal how the findings are relevant to each of the specific outcomes:

· Better equipped primary school/resource centre/community libraries
· Effective training in library resources management
· Improved community understanding of the importance of books
· Enhanced Department of Education capacity for project management

3.1 Better equipped primary school/resource centre/community libraries

The following goals and critical project assumptions were met:

· the establishment of basic working libraries and systems at the District Education Resource Centres (DERCs)

· the purchase and delivery of books and other materials

· the establishment of effective security and maintenance systems

· the adequate and cost effective use of resources

· the achievement of reasonable borrowing levels

· the provision of relevant and suitable materials

· improved access and, in many cases, first time access to materials contributing to improved learning and teaching

· the introduction of mobile library system.

· Borrowing levels
The borrowing levels could have been higher if more schools had allowed learners to take books home.

· Access
While, in general, resources were adequate, there was considerable variation among schools in terms of access.

· The culture of reading
The evidence relating to the impact of books on the culture of reading, teaching and learning was ambiguous. While the majority of teachers and principals perceived significant improvements in the culture of reading, teaching and learning, a substantial number of learners claimed that the changes had not been significant.

The following targets and critical assumptions were met:

· Coverage
The coverage of training was good.

· Content
The content of training programmes was sound.

· Delivery
The delivery of training programmes was efficient and effective.

· Training
The training was relevant to work situations and planned functions.

· Staff turnover rates
These were low.

· Skills
The skills gained were utilised to improve job performance.

· The completion and thoroughness of training by DERC and individuals varied.

· Due to time constraints, training manuals that would have standardised training and assured quality were not developed. It should, however, be noted that training manuals are now being prepared for phase two of the project.

· There was limited on-the-job support to ensure the implementation of new ideas.

· Due to time constraints, the training of DERC facilitators was restricted to library resource management. It would appear that a broad range of development activities is taking place in some DERCs. This means that there is a need to extend the training of DERC facilitators to rural education facilitators.

· the establishment of representative community structures
· meaningful community involvement
· community capacity building
· voluntary contributions by communities to the project

· the limited social marketing of the project

· resistance on the part of a few DERC facilitators to the meaningful involvement of the community

· a lack of clarity about roles and functions on the part of some community-based structures' members

The following targets and critical assumptions were, on the whole, met:

· the creation of effective structures for cooperation

· the sharing knowledge, skills and competencies among the implementing and governmental agencies (this enhanced the Department's capacity to manage the project after the pilot phase)

· the establishment of mutually beneficial partnerships

· budget and time constraints
· the uncertainty resulting from the PLIS separation from the Department of Education
· communication problems caused by a lack of telephone and fax facilities in the rural schools
· a limited sense of ownership and control of the project by PLIS

While the findings outlined in the previous section speak volumes for the success of the project (insofar as it successfully achieved many of its projected goals), I nevertheless contend that it is difficult to make any prognosis about the sustainability of the project.

The issues of long-term sustainability relate first to the question of funding for the project after the donor funds have ceased to be allocated. The assumption was that the provincial government would take over the costs of running the libraries but, in the event, this did not happen. (An interesting development is that the project is now housed in one of the large teacher trade union's offices -as opposed to government offices.)

It is necessary when making a statement about project sustainability to take a variety of social, economic and political factors into account. The need for the provincial government to take over the running costs of the SABAP project was one of the prerequisites for sustainability. But this did not happen. When linking sustainability to context, it must be noted that the Eastern Cape is regarded as the poorest South African province – a province in which vast numbers of teachers are being retrenched as part of a rationalisation process. Numerous schools lack basic infrastructural needs (such as water, lights, toilets and desks). Against this background, the implications for funding are not clear. It is, however, possible that 'nice-to-haves' like books – in a poverty-stricken province – would be an unlikely government priority.

The second issue relating to sustainability relates to resolving tensions in the collaborative model of governance, management and delivery of education services. The inherent tensions in this model are (for example):

· democracy vs. professionalism
· organisational choice vs. professional choice

Sustainability implies the need for agreed definitions about requisite institutional capacity and how the project itself is defined. Sustainability also implies that the interests which promote and are affected by the project must at least be of 'one mind' if the project is to be sustained in the long term. This shared vision has to carry the project forward. In the arena of evaluation, assessment must also take into account the extent to which the project is able to impact on the ideas and interests of participating institutions.

6 Conclusion

Evaluation, as I pointed out above, is about determining the value (worth and/or merit) of a programme or project. There are two critical questions that relate to assessing the value of SABAP. Did SABAP do things correctly? This refers the degree of cost efficiency that was achieved in realising the specific objectives of the project. On the basis of the performance indicators and critical assumptions, the expected outputs were not only well achieved: they were also attained at a reasonable cost. The second and more critical question is: Did SABAP produce valued outputs? In answer to this question, a substantial number of the stakeholders answered in the affirmative on all the four outputs.

The foundations of enduring change have been laid because SABAP provided guiding ideas, theories, methods and tools and because they built teams from groups of individuals. The project has, in addition, developed skills and capabilities, and stakeholders are starting to see and experience the world differently. They are also beginning to form new beliefs and assumptions. Can this not be seen as the first step towards sustainability?

Footnote

1. It is pertinent that this paper contextualises the project being considered since the context is crucial to the question of sustainability.

2. The fourth criterion was not incorporated in the logical framework but was implied therein and has relevance for project sustainability.

[Previous Page] [Top of Page] [Next Page]