Evaluating Impact - Education Research Paper No. 35, 1999, 262 p.

[Previous Page] [Table of Contents] [Next Page]

4. RELATIONSHIP BETWEEN NATIONAL AND EXTERNAL

4.1 A note on a participatory impact study in Eritrea: exploring the relations between national and external researchers
4.2 The relationships between national researchers and external researchers
4.3 lmpact studies: the role of an insider/outsider
4.4 lmpact assessment in educational projects: some perspectives on the 'insider outsider' debate

4.1 A note on a participatory impact study in Eritrea: exploring the relations between national and external researchers

Tefsai Bariaghaber
SETER Project
Eritrea

My experience of the Eritrean project impact assessment and other local field research leads me to believe that a participatory research approach is the most effective way of assessing impact. The Eritrean Impact Study clearly demonstrated that the participation of local researchers and their consequential interaction with external researchers created benefits and learning opportunities for both local and external teams.

The two teams participated in the planning stages and jointly produced an appropriate research design. This activity was squarely based on our agreement about our central purpose, which was to ascertain whether or not the project had accomplished its aims. Although collaboration was one of the main factors that influenced the relations between the local/national researchers and their external counterparts, different perceptions and expectations of the project on the part of the two groups of researchers created different expectations about the ultimate aims of the project.

Because local and external researchers were somewhat at variance about what might constitute an optimal research design and adequate goals, and because both parties were motivated by differing perceptions and motivations, this variance of opinion and purposes inevitably influenced both the design of the instruments and the research agenda. This, however, in no way detracted from the benefits of collaboration. In contrast, one might argue that the differences actually enriched the ultimate research design as well as the final report.

2 Setting the scene

The arrival of external researchers for the assessment study conducted in Eritrea was met with enthusiasm and was welcomed by Eritrea's Ministry of Education. For the duration of the research exercise, the relationship between the national research team and the external researchers was harmonious, understanding and cooperative. This positive relationship between the two teams was evident from the moment the external team arrived. In their initial interactions, the external team orientated the local team about what they were hoping to achieve during their stay in the country by describing the technical details of the process. The local team responded to these briefings by outlining their hopes, expectations and plans for the project. After much discussion and a fruitful exchange of ideas, both teams were able to reach agreement about a framework for the project.

This initial orientation was crucial since it formed the cornerstone of a positive relationship between the two sides. The researcher considers such mutual interaction to be an the essential part of any process of this kind because it affords both sides the opportunities to exchange ideas about what and how things will be done. Although initial bonding between the two teams was strong, their harmonious relationship was unfortunately complicated in the latter stages of the process and this caused several difficulties. The difficulties that arose were the following:

2.1 Geographical separation

After our initial orientation, the local research team moved back to the sites of the research (i.e. back to the schools) while the external researchers returned to their home country. This geographical separation unfortunately created a break in continuity between the two teams of researchers. What the local team had hoped for was continuous cooperation and immediate feedback. Instead, feedback was delayed and the separation resulted in a number of associated problems.

2.2 Lack of communication

Another factor which is associated with geographical separation and which influenced the relationship between local and external researchers was that the two teams of researchers were not able to interact and hence were unable to exchange ideas about emerging problems and other factors that influenced the work of the project as it progressed. Thus, for example, when local researchers thought of new categories or ideas that they felt might enhance the research, they were unable to test such ideas with the external researchers. Certainly, the communications in Eritrea were such that it was not easy to incorporate any new components into the research process after the external researchers had departed.

2.3 Lack of capacity building

Local researchers in developing countries often tend to rely on external partners to assist with the development of local capacity. This was one of the expectations that the local team had entertained – but because of the separation, was not achieved. The local team, for example, felt unable to cope as well as they would have liked during the data processing stage of the project. By the time the project entered this phase, the external consultants had already left for home. The local researchers had recognised their need to develop the capacity for data processing – especially with regard to the tabulation of information and the categorisation of items during the data collecting process.

When problems began to emerge at this stage, the local researchers realised that the external researchers could have made a decisive contribution by helping to enlarge the skills base of the locals, and that this contribution might radically have affected the quality of the intended results. Because the national research team was limited in their understanding of the theoretical basis of data analysis, it would have made all the difference in the world if the British team had been able to stay longer in Eritrea If they had been able to stay, they could have helped local researchers to gain competence and confidence in the theory and practice of data analysis at this crucial stage This would have yielded better results and more meaningful recommendations in the final report, and the impact assessment might also have contributed to sustainability.

3 Conclusion

Although this paper outlines some of the problems inherent in the relationship between internal and external researchers, it is nevertheless clear that we, as local researchers, were able to experience many of the benefits that arise out of the process of participatory research - in spite of the difficulties engendered by the problems (the chief of which was the premature – in our view – separation of the two teams of researchers).

The participatory approach is extremely beneficial to national researchers because it gives them the chance to refine their knowledge of research methods and techniques Because it does this, it contributes to the development of local human resources On the other hand, the process surely also benefits visiting researchers because it enables them to gain easy access to the local context Indeed, it is my contention that this process of immersion in local culture (a consequence of participatory research and collaboration with the local team) benefits external researchers long after they have left the original site of research activities.

4.2 The relationships between national researchers and external researchers

Harvey Smith (in conjunction with Paul Bennell, Consultant)
Senior Education Adviser
Centre for British Teachers

Although an impact study of an aid-funded project may, in theory, be undertaken entirely by external consultants or entirely by researchers of the country where the project is taking place, such studies in practice are most likely to be undertaken by a mixed team. This paper considers some of the issues that arise as a result of collaborative relationships between local researchers and the external consultants.¹

The paper draws on the experience of the DFID-funded English Language Teaching (ELT) Impact Study in Angola and Eritrea. This study, undertaken in 1997 and 1998, looked at the impact of ELT projects funded by ODA/DFID in the two countries and involved teams of local researchers trained and supported by three external consultants. When the studies in each of the countries have been completed, a comparative analysis will be undertaken. In a briefing to the consultants, DFID staff emphasised that the study should not be a conventional evaluation based on project logical frameworks, and that local perceptions should be gathered from stakeholder meetings and research activities undertaken by the local teams.

Both Angola and Eritrea are countries which are deeply involved in violent conflict and they therefore experience special difficulties that may not by typical of the countries where impact studies are usually undertaken. Many of the educated nationals of both countries live abroad and there is a limited capacity within the countries - especially within the institutions and departments of the ministries of education - to undertake research.

Undertaking such studies has enabled the consultants to reflect on the nature and practice of impact studies, and a number of issues have been identified for discussion here. These have been divided into ones that may be labelled conceptual.

2 Conceptual issues

Conceptual questions raise questions about the nature of an impact study as an undertaking. They of necessity raise questions that relate to the practice or processes of undertaking such a study. The kinds of questions that were raised in the assessments of the aforementioned projects were as follows:

2.1 Ownership

Just as the success of externally initiated projects depends in part on the extent to which local ownership and commitment can be established, the same is true of an impact study (although there is often less time is available to achieve this). The ELT impact studies in Angola and Eritrea were initiated by the agency (ODA/DFID) which had established and funded the ELT projects in the two countries and not by stakeholders in the countries. The success of the studies therefore depends in part on the extent to which local motivation can be established and ownership achieved. This, in turn, depends in part on the ability and willingness (of the personnel of the projects whose impact is being studied) to raise the awareness of the ministry of education or other institutions² about the need for such a study and about the lessons which might be learnt from the results. If one considers the experience of the Angolan and Eritrean studies, one sees that the response of local institutions varied according to the extent to which they are able to identify ways in which the study might resonate with their own agendas.

2.2 Perceptions of the role of consultants

The role of the external consultants may be perceived differently by different stakeholders. Such perceptions may depend on many factors, such as:

· the nature of the involvement expected of the consultants in each of the main stages of the impact study³

· the extent to which the national researchers perceive themselves or are perceived as being able to act independently of the consultants

· the nature of the relations established between the consultants and the expatriate project personnel (if a project is an ongoing one)

2.3 Using a multidisciplinary team

A strength of the ELT Impact Study is undoubtedly attributable to the fact that it comprises a multidisciplinary team of consultants whose fields of expertise cover ELT, economic and social development (including gender issues). This has encouraged a broadening of the field in which the country studies look for evidence of impact. Although the external consultants perceive themselves as a team, and in spite of the fact that they have made individual visits to each country, they may not be perceived locally in the same way. Moreover, it was not possible for the local players to replicate our notion of a multidisciplinary team since they were unable to field people with experience in investigating socio-economic issues.

2.4 Baseline data

It is difficult for local researchers to assess impact where little or no baseline data are available. Baseline studies would have contributed to the measuring of the post-project situation by enabling comparisons to be made against pre-project data. In both studies, the external consultants had access to documents compiled by DFID, which describe the pre-project situation in outsider terms. (Local researchers would not normally have access to such documents.) Equivalent descriptions in insider terms are unlikely to exist. Local researchers may not have been around before the project started or they may themselves have been involved in a very specific part of the project and so be unaware of the wider picture.

2.5 Levels of impact

This issue refers to the balance between looking narrowly (in the case of an ELT impact study) for impact on ELT activity and looking for wider socio-economic impact. This is especially problematic when local researchers are drawn from a narrow group in the ELT field or from a ministry of education. In such cases there may be a desire to focus largely or entirely on looking for evidence of impact in the ELT classroom and even a reluctance to go beyond the direct outputs of the project. This can cause tensions between local and external researchers.

2.6 Training researchers while simultaneously undertaking research

The role of the external consultants includes training the local researchers and assisting them with the design of the research, the development of the instruments, the analysis and interpretation of the data and the presentation of the findings. How does one balance these roles? Although the training role is essential to building on local capacity, this may cause the external consultants to perceive this training as being more important than their roles as facilitators and managers of the research. They may therefore not consequently fulfil their terms of reference.

2.7 Remuneration and responsibility

The relationship between external and local researchers is also influenced by whether or how the local researchers are remunerated. Ownership of the study by a local institution would imply that the institution remunerates the researchers (even where an external agency assists with the budget for this), and the external researchers are not seen as buying the research. In one of the countries where an ELT Impact Study was undertaken, local institutions were unable to compensate local researchers. In this case, they were paid a fee by the external consultants. The perception created by this was that individual researchers were employed by the outside consultant agency and that institutional (and hence also local ownership of the study) were therefore limited.

3 Practical issues

While conceptual questions raise questions that relate to difficulties arising from the research process, there are several logistical or practical issues that impact on both the research enterprise and the relationship between the local and external researchers.

3.1 Location

A problem that we experienced while managing the ELT Impact Studies for Eritrea and Angola was that, in both instances, the management and coordination was located outside the countries concerned. The management was therefore perceived as driven by the client (DFID) - at a time when engendering a sense of local ownership was seen as critical to the success of the studies.

3.2 Number of institutions involved

In both studies we experienced difficulties with coordinating research and ensuring cooperation because in both cases more than one institution was involved in assessing the impact of the respective projects. This was particularly noticeable in those cases where there was a mixture of government and autonomous or semi-autonomous institutions, such as a ministry of education and a university. If there is no institutional hierarchy, the person who is appointed as team leader or as research coordinator may find it difficult to secure the necessary collaboration. In such cases, requirements such as obtaining agreement about deadlines (to name but one issue) may prove to be extremely difficult. When this happens, external consultants might find themselves assuming a coordinating role by default.

3.3 Selection of researchers

There are a number of practical issues which are relevant to the way in which a local team is selected and managed, and to what the role of external consultants might be in this process. These include the extent to which external consultants can influence the size and composition of the research team, their gender balance, the level or mix of research skills, the identification and inclusion of disinterested parties, and the commitment or level of involvement of the team members. Also relevant are issues such as whether or not researchers are released from other duties for the period of the study and whether or not remuneration is necessary. In the Eritrean and Angolan studies, there was reluctance on the part of participating institutions to have people from outside those institutions in the research teams. The result of this was that the teams did not have the insider/outsider balance which the external consultants would have wished to see. Such differences can lead to tension between external consultants and institutions.

3.4 Timing

The timing of assessments is also a significant issue which needs to be considered. In Angola and Eritrea, the funding agency (and not the local players) decided when the impact assessment should take place. In both cases, the process started while the projects were still being implemented and while expatriate project staff were still in their postings. This posed difficulties for both the internal and external researchers. For the former, the timing of the investigation increased the difficulties that local researchers experienced in assuming a detached attitude to the actual project - and this meant that they were less able to look for impact beyond the current project activities. The consequences for the external consultant were that they experienced difficulties (in the limited time available) in setting up the research programme and convening stakeholder meetings immediately after they had arrived in the country.

3.5 Local realities and external deadlines

Progress in the research depends on the extent to which local researchers are able to undertake the work without disruption from unforeseen events (the ELT Impact Study in Angola and Eritrea had to compete with wars and strikes) or are able to work at their tasks without being asked to undertake other activities. Since impact studies generally have to be completed in a very short time-scale, any delays resulting from unforeseen local events (such as in war-torn countries) will have a greater effect, as there is less capacity to absorb them.

4 Conclusion

In essence, the issues above relate to the difficulty of finding the right sort of balance between local ownership and achieving the externally imposed terms of reference. If an impact study is to be locally owned and is to provide a ministry of education, or another local institution with information about the effect that a project has had, then the study must meet local needs and must be undertaken in a manner that is locally acceptable.

If the investigation is to be an external activity that enable an external funding agency to find out how effectively its projects have contributed to development (assuming that it is possible to attribute evidence of impact to the external funding), then some way must be found of making externally imposed criteria and expectations acceptable to those who are locally involved. The key question then becomes, What sort of compromise is possible under the circumstances?

Footnote

1. Following the brief given to the writers, the paper will deal solely with the perspective of the external consultants.

2. This would need to be done well before the actual study is started.

3. This includes consultation with stakeholders, design, data collection, data analysis and report writing.

4.3 lmpact studies: the role of an insider/outsider

Mohammed Melouk
Mohamed V University
Rabat, Morocco

Assessing the impact of an educational project is not an easy matter, especially when the aim of the exercise is to determine exactly, in both quantitative and qualitative terms, the exact nature and extent of change which is being sought by the project. This complexity may present an insurmountable hurdle - whether the evaluative enterprise is carried out by people directly involved in the project (the project designers and implementers) or by outsiders or external experts (or a combination of both).

I contend that the sometimes baffling complexities of impact assessment are caused by the fact that education and educational change are situated in a complex matrix of causes and effects that include cultural, socio-economic, psychological, material and administrative processes. Impact assessments, by their very nature, require a diverse number of role players, including, among others, decision-makers, funding organisations, project designers, administrators, inspectors, teachers, students and pupils.

Quite apart from this, the unique geographical, social, economic, political and educational conditions that gave rise to the project in the first place cause project evaluators to become enmeshed (whether they like it or not) in a complex of feelings and attitudes that invariably impact strongly on the assessment process. Educational projects which are designed for developing countries are far too often designed without any input (and in the absence of) those for whom the project is intended. This inevitably gives rise to a variety of attitudes on the part of the local population to the project. Attitudes may range from enthusiastic adherence (the rarest!) to qualified but sullen acceptance, indifference, or simple outright a rejection -at least at the psychological level.

Although the emergence of such attitudes among the target population may be attributed more to negative emotional reactions rather than reasoned intellectual objections, the effect on the project itself is nearly always emasculating or debilitating. This, sadly, is inevitably the case when the local target population have not been involved in the initial stages of the project and do not fully understand its methods, aims and objectives - and hence, of course, cannot appreciate its potential benefits. In such cases, an accurate assessment of the impact of the project cannot be obtained without considering the potential effect of such attitudes on the outcome of the project. The assumption here is that the contribution of insiders, who are outsiders to the project but insiders to its situational context (hence the term insider/outsider), may be able to contribute insights of crucial importance.

2 The rationale for an insider/outsider in an impact study

In the light of these considerations, it is necessary to discuss the insider/outsider dimensions of impact studies. The discussion is based on conclusions drawn from personal experience in a collaborative impact study entitled The Moroccan Item Banking Project.¹ In this paper I have attempted to highlight the importance of local human abilities in any endeavour to assess the impact of an educational project, without, of course, completely ignoring the legitimate concerns of objective educational evaluation or of funding bodies.

Given the nature of the educational enterprise, the implementation of any educational project is bound to generate a dynamic of its own, which is often not anticipated in the original design or sufficiently catered for or monitored during the implementation stages. In addition, any project involving many people will engender varying levels and degrees of involvement. The people directly concerned or those for whom the project has been designed may take a route (in terms of objectives and priorities) slightly different from the one originally intended. They may also have a hidden agenda