This snapshot, taken on 03/08/2004, shows web content selected for preservation by The National Archives. External links, forms and search boxes may not work in archived websites.
|
Evaluating Impact - Education Research Paper No. 35, 1999, 262 p. [Previous Page] [Table of Contents] [Next Page] 7. IMPACT ASSESSMENT AND SUSTAINABILITY
7.1 Background to the MAPP evaluation 7.1 Background to the MAPP evaluationCarew B W Treffgarne
1 Background to MAPP British support for this event in Zacatecas/Pachuca/Tijuana/Merida/Monterrey today arises from our involvement in the Advanced Professionalisation Programme for Mexican University teachers since 1991. The purpose of this training scheme for upgrading University teachers is capacity building in the widest sense, ie not just training of trainers, but English curriculum development and institutional development through the Schools or Departments of Languages and Language Centres in the State Universities. It is part of the Mexican Government s commitment to raising standards in English teaching. In consequence the programme builds on the Certificate for Overseas Teachers of English (COTE) scheme, introduced by SEP in collaboration with the British Council. It is also linked to the complementary SEP/British Council programme for developing Self-Access Centres (SACs) for language teachers and students. When I first visited the Programme 18 months ago, the evident signs of project impact exceeded our expectations. We were encouraged to find that the Director of Higher Education, Dr Arrendondo, shared our interest in ensuring that the evaluation of this exercise should be both participatory and formative. In this way, it is hoped that the Mexican Government, the British Government and each participating University will gain in-depth insight into the value of this training from the point of view of each institution that has benefited. For DFID the Advanced Professionalisation programme is unique from several stand points: · Firstly, it represents a unique experiment in terms of scale. Five different British Universities have been involved in 9 projects on courses designed to upgrade teachers from 31 public universities. In terms of quantitative impact, there has so far been a 75% success rate with 96/124 teachers gaining their undergraduate or postgraduate qualifications.For us it also represents a significant experience - not only in terms of low unit cost, but also in terms of cost sharing. The figure used to calculate the cost/benefit of training in Mexico versus training in UK does not take into account the financial contribution of SEP, the participating Universities and the individuals concerned. SEP have contributed to the participants transport and accommodation costs. The host University for each taught module has been generous in its allocation of seminar rooms and administrative support, and has contributed to the accommodation in Mexico of visiting British lecturers. And in many cases, the teachers have had additional expenses that they have met themselves. DFID paid tuition fees, book presentations, air fares for British tutors and management costs. The fourth aspect of the programme that represents an innovation for DFID is the scope and scale of this Impact Study. The original plan was to undertake a traditional approach to evaluating the programme using a team of external consultants. However I was convinced by the presentations that I observed at Puerta Vallarta of the Western Masters and the Central Universities Diploma programmes in January 1996, that no external evaluation study could do justice to the impact of the programme in the way that those who have directly benefited from the training can do. The survey of personal involvement in institutional change I conducted among a sample of 34 teachers (from 25 Universities) in February 1997 confirmed this conviction. 2 Assessing impact This Impact Study is participatory because we hope that since February, the teachers in the scheme will have been engaged in researching the different areas where they think the MAP programme has affected their institution. It is participatory because the exercise should also have involved colleagues from the same Department or Language Centre (who have not necessarily undergone training). The exercise will be formative because we anticipate that the following outcomes will emerge from the presentations: They will demonstrate the value of what teachers have been doing as a result of their training to their own University authorities (and to others in the region). They will highlight the quantitative and qualitative impact of their training. We realise that it will be impossible to extrapolate the effect of the Advanced Professionalisation Programme per se. Hence the impact exercise takes into account the cumulative effect of COTE, advanced training and SAC programmes funded by SEP and the BC. Most Universities have been affected to a greater or lesser extent by all three, and some teachers have been involved in all 3 as well.
The process of researching quantitative and qualitative impact will have helped to enhance the professionalism of those involved, and will hopefully encourage the Departments or Language Centres to set up procedures and mechanisms for monitoring the impact of curriculum development and teacher training in a more systematic manner. Our concern as a participating funding agency is that the qualitative impact of the training is clearly demonstrated in terms of · new teaching responsibilities and skillsThis needs to be recognised. For many teachers this means an improvement in status, and this should lead to an increase in salary. In consequence, we urge all university and ministry representatives to redouble their efforts to ensure that the British certificates awarded by the scheme are recognised in each participating teachers place of work/host university, and of course nationally. Impact in a programme like this can be read in four ways. In the February workshop at the British Council we explored the distinction between impact on the individual, and impact on the institution. There is also the question of impact on relations between the Universities and SEP. Finally there is the question of impact on the external funding agency, DFID. 3 Conclusion To summarise what I have already identified as unique or innovatory, this programme demonstrates to DFID: · the value of organising the training in Mexico (rather than in UK). I should add that, in addition to lowering the cost substantially, more women have been able to benefit;There is considerable interest in what we are doing this week in Mexico in London, because this particular approach to impact studies is in itself an innovation. We are here to listen and learn. I am sure that we shall not be disappointed. 7.2 Sustaining Impact: the Mexican Advanced Professionalisation ProjectKeith Morrow
1 Introduction The Mexican Advanced Professionalisation Project (MAPP) was set up with the support of ODA/DFID in 1991 In its simplest sense, it was a scheme to upgrade the professional qualifications of teachers working in university schools/departments of languages, and language centres. It also incorporated the broader aim of capacity building in the widest sense. In this sense, Morrow understands capacity building to mean not just the training of teachers: he defines it to include English curriculum development and institutional development. From 1991 to 1997 five British universities were involved in nine separate projects which ran courses for teachers from 31 public universities. Towards the end of the project, it became clear that many of the intended outcomes were not susceptible to evaluation of a conventional summative nature. While certain of the outcomes (for example, the number of participants, their success rate in obtaining target qualifications and the costs incurred) could be measured in straightforward terms, it was clear that much of the impact of the project was not easy to measure since, if it was known at all, it was known only to the individuals concerned. It was in fact unlikely that even they understood the more subtle implications of the impact since they had never been accorded any formal opportunity within the framework of the project to articulate or explore what the impact on themselves might have been. It was therefore decided, with the active encouragement of the Mexican government, to undertake a participatory evaluation which would draw directly on the experience of the participants. Furthermore, it was decided that the evaluation should be formative in nature. It was also decided that the evaluation would not only include attempts to uncover in retrospect the impact that MAPP had made, but that it would also include an exploration of ways of effectively disseminating this impact in the institutions in which participants were based. It was felt that an impact evaluation of this kind would contribute to institutional development. 2 The formulation of a participatory approach It was decided to design an approach that would enable participants to articulate the impact that the project had had on them as individuals. This impact evaluation therefore comprised the following three elements:
2.1 Objectives of the participatory impact evaluation The evaluation had three main objectives: · Firstly, we wanted to know how the training which participants had received in the project had brought about change for them as individuals, and how it enabled them to contribute to change in the institutions in which they worked. This was direct impact evaluation.2.2 Contribution of the approach to sustainability Our review of what we achieved highlighted three issues of particular relevance to the evaluation of impact. The close interdependence of the three areas outlined above Sustainability (which is perhaps the key issue for the funding agency) was enhanced by an impact evaluation which involved participants and helped them to articulate the changes which had taken place in their professional lives as a result of the project. This articulation is crucial since, without it, the impact of the project may have remained hidden - even to those who participated in it. We also realised that sustainability is enhanced when participants help to disseminate information about impact. But because effective dissemination requires specific skills and procedures, we realised that participants needed to be taught such skills if they did not already have them. Far from being self-indulgent proselytising, the teaching of such skills to participants should be viewed as a crucial aspect of sustainability. The initial workshop was therefore much more concerned with exploring ideas about change and development on both a personal and an institutional level. It was also concerned with providing a framework in terms of which participants (1) might identify changes which had taken place in their own professional context as a direct result of the training they had received or (2) be able to say how such a training had enabled them to contribute more effectively. Some general categories were developed to group the changes identified by participants. These included: · new teaching responsibilities and skillsDiscussion about ways in which participants could become proactive in bringing about change in their work contexts was another important feature of the workshop. On one level this involves the development of dissemination and implementation skills. It also involves the identification of appropriate action areas where sustainable local initiatives can have important consequences in building and strengthening the institution. Sustainability involves helping project participants to set targets for the future. The concept of benchmarking for ensuring the achievement of outcomes and the achievement of sustainability is a crucial one that was unfamiliar to participants. A lot of time at the initial workshop was therefore taken up in exploring the notion of target-setting, and in exploring the different ways in which targets could be identified and set for different aspects of the work of their institutions. One of the most important outcomes of the workshop was a set of benchmarks for developing the curriculum and delivering the four different types of Licenciatura in ELT courses. A striking feature that emerged during this process was the degree of difference between different institutional contexts, and hence the differences in specific targets set by participants from different institutions. In spite of this, the sharing and discussion of categories within which benchmarks could be established was a major benefit of the initial workshop. This once again emphasises just how necessary the participatory approach is. Apart from being likely to be either wrong or irrelevant in content in individual settings, externally imposed global benchmarks also fail to involve the project participants as stakeholders in their implementation. 3 The need for different documentation for different audiences This may seem an obvious or even a trivial point, but it is extremely significant. Traditionally, the recipients of a project report are the funding agencies, and the data they require are largely global in nature. However, it is essential, in a participatory and formative evaluation, that documentation be prepared for the participants and that such documentation relate to their individual experiences and needs. In the case under discussion, we prepared a report at the end of the initial workshop and we circulated this report to all participants. It was essentially an aide-mémoire which described the stages of the workshop, the activities which we undertook and the rationale which supported them, the outcomes in terms of revealed impacts, and agreed benchmarks. It also suggested strategies which participants might use for working towards attaining these benchmarks in their own institutions. Although the report was compiled by the external consultant who had been leading the workshop, it was, in a sense, the property of those who had taken part in the workshop, and was meaningful to them in a way which an externally generated impact evaluation study could never be. This focus was stressed in the introduction:
3 Conclusion The overall focus of the workshop, and of the three-stage evaluation framework, was to help the participants to develop the skills they needed to foster institutional growth through their own professional development. Setting up a framework which provided the opportunity for participants to carry out research into qualitative and quantitative impact (but which placed the responsibility for the research on the participants themselves), enhanced the professionalism of those involved. It was in this way that the process of evaluation contributed to the aims of the project. 7.3 Assessing the impact of sector wide, institutional and policy outcomesKora Basich
1. Introduction Our university is situated in the north-western region of Mexico – in an area which, over the last twenty years, has been transformed from an agricultural to a primarily industrial zone. This relatively new socio-economic characteristic of our region, as well as its geographical location on the border with the United States, makes English Language skills an important part of any training programme. English Language Teaching in Mexico has been enormously professionalised during the last five years. At the university where I teach, the Universidad Autónoma de Baja California in Mexico, two major ELT training programmes were offered between 1992 and 1997. The first programme is the Bachelor in Philosophy of Education degree (B Phil Ed), which is offered by the University College of St Mark and St John (Marjons) in association with Exeter University, and in which 23 teachers participated. Eleven of these were from my own university. This programme was financed by the ODA (now DFID), which put up the finances for the College. Our travelling expenses and subsistence were funded by the Mexican Ministry of Education. The financial management of the project was undertaken by the British Council in Mexico. The second training programme is the Certificate for Overseas Teachers of English (COTE), which is offered by Cambridge University, and which to date has trained more than fifty teachers in three state-wide programmes. The programmes were all undertaken at my University, and we received financial support (as has been mentioned) from the Mexican Ministry of Education and organisational support from the British Council. 2. Anticipated and unanticipated benefits When a training process begins there are some clearly defined anticipated benefits. Our institution aimed to train teachers (in both the programmes mentioned above) so that we in future would be in a position to implement an English Teaching Programme of our own with the support of graduates from the B Phil Ed ELT course. I believe that these were clear and accomplishable goals. When we performed an assessment of impact, we were greatly surprised when we encountered a number of unexpected consequences that had arisen out of the training programmes. Although some of these had not even been anticipated even in our long-term plans, they have proved to be enormously beneficial for both our institution and our region. 3. The approach which we used in our research impact The methodology which we utilised to identify impact was eclectic, and our aim was to gather not only quantitative results, but also qualitative data of the kind that would enable us to identify which actions had produced the greatest impact. We designed different types of research activities in order to obtain this information. 3.1 Reflection workshop Firstly, we planned a reflection workshop. This was organised so that we could bring together all the individuals who had participated in the training programmes so that they could collectively reflect on the type of work they had been doing prior to the intervention. They were required to reflect on where they were working and at what level they were operating. They were also expected to reflect on the procedures which produced change. Thus, for example, they had to think about the extent to which their work had changed and at what stage they believed these changes to have occurred. We then asked them to reflect on what they were doing at present, how their views had changed, and what their expectations of themselves and their institution were. In the process we induced them to think about what their training and development aims were and how they were working to accomplish these aims. This particular exercise produced a much greater quantity of useful information than we had expected. Indeed, participants used the time allocated to thinking about themselves and the opportunity to share their experiences with others so well that they identified, in the process, many of personal benefits which, until then, they had not even considered. This process actually therefore strengthened and reinforced the aims and objectives of the project. This meant that the actual research process contributed to the enhancement of the initial project goals. The research process empowered the further growth and development of participants both in terms of their personal roles and their personal satisfaction and gains. But it also contributed to the enhancement of institutional and regional improvement in their areas of expertise. 3.2 The use of a questionnaire We followed this exercise by giving each participant a questionnaire to take home for one week. Once again the length of time given to participants to consider their responses to the issues raised provided them each with opportunities for profound reflection. We believe that these opportunities were crucial factors in reinforcing changed practice. 3.3 Documentary research Since we had agreed that it was necessary to obtain certain baseline data, we conducted documentary research by going through our own institutional database in order to find changes in programmes that were offered and changes in student and teacher characteristics. At the same time we examined our own relationships with other institutions, including educational, governmental and private institutions, in order to arrive at an understanding of how our own department had changed in terms of activities and responsibilities. We also gathered data about our own responses to change, the problems which we encountered and the way in which our aims were accomplished. We consider that one of the most important benefits of the training provided is an awareness of change and how change may be managed. We had learned a good deal about how processes could be analysed by using the Review-Plan-Act-Review-Plan-Act Cycle. 4. Benchmarks Another most important benefit which accrued from the impact analysis exercise that DFID initiated early in 1997 was that it provided us with guidelines for organising planning. Once we had conducted our baseline investigations and had acquired an adequate amount of baseline information, it was then much easier for us to see where we had been, how training had effected or promoted certain important changes, and where we could realistically hope to go from the point at which we had arrived. The benchmarks acted as guide. They located us in a context and enabled us to plan our own future development and growth. At the same time they offered us the opportunity to control quality and implement the programmes which we had planned. Benchmark planning was an activity that (we found) conferred most tangible benefits. 5. Conclusion Our University is deeply grateful for the aid which we received from the Department for International Development and for the support which was given to us by the British Council in Mexico. We are also profoundly indebted to those British universities which involved themselves so enthusiastically in the training programmes. The effort which was invested in these particular programmes has been enormously beneficial for the individuals who were trained, for their institutions and for regional development in Mexico. As I have indicated in this paper, the benefits which have accrued go far beyond what we initially anticipated. This in itself is a testimony to both the impact and to the sustainability of the project. 7.4 Determining the unanticipated outcomes and using these as benchmarks for future projectsJorge Anguilar Rodriguez
1 Introducation The Autonomous University of Sinaloa (hereafter referred to as UAS) is located in the state of Sinaloa in Northwest Mexico. It has a population of 95 000 students. 11 630 of those students study at the four Language Centres of UAS in Los Mochis, Guasave, Culiacan, and Mazatlan. There are two different programmes in the language centres, the regular course for young adults and adults (which has an enrolment of 8 480 students), and the Saturday Children's Programme, with an enrolment of 3 150 children with ages ranging from 8 to 14 years old. Both the regular courses and the children's programmes offer general English courses that teach the kind of communicative competence that children need and that adults need to make them effective communicators in their personal, academic and professional lives. Methodology In order to ensure the success of the impact evaluation and to have a clear framework for our evaluation, we decided to conceptualise this investigation as a process consisting of the following steps: 1. Formulation of aims2.1 Impact assessment as contributing to project sustainability It is significant to note that although the above outline is similar to the conventional stages of all research enterprises, it was applied in such a way that the data gathered would be useful for the enhancement of project sustainability. Because the assessment needed to give as much attention to gauging the anticipated outcomes of the project as it did to gauging the unanticipated outcomes, each stage of the research design was considered for the what it would reveal about unanticipated outcomes. Thus, for example, stage 6 of the research outline, dwells on the importance of the establishment of new goals, new benchmarks and new projects. These arise from the uncovering of unanticipated outcomes in stage 5. For the same reason, stage 5 devotes a considerable amount of time to identifying what positive unanticipated benefits might be ascertained from project players. The subsequent phase refers to ways that such benefits could be mainstreamed so that new benchmarks might be formulated and new projects considered in stage 6. 2.2 Aims We decided, prior to beginning the impact study, that we needed to formulate the aims of the investigation so that we could undertake more focused research. The aims specified at that time (April 1997) were: · to investigate the extent to which individuals who have received training have contributed to the development of our institution2.3 Description of practice and its effectiveness After we had formulated the aims of our impact evaluation, we felt the need to describe our practice and programmes at the language centres prior to the implementation of the Mexican Advanced Professionalisation Scheme (MAPS). We did this so that we could familiarise ourselves with our teaching and management practices. We believed that such a description would lend weight to the evaluation and increase its validity and reliability. A team of teachers, administrators and academic co-ordinators were therefore subsequently involved in the process of describing practices and determining how effective they might be for meeting the needs of students, teachers, the institution and the community. 2.4 Focus of investigation In order to be properly focused and avoid generalisations, we decided to determine, by way of analysis, the key areas that needed investigation. It was decided, after analysis, that the following areas needed assessment: (1) Knowledge, skills and attitudes · syllabusIn addition to these, it was necessary to gauge the expected and unexpected benefits as well as positive and negative effects of the project. This of course had implications for the choice of the research instruments. 2.5 Research instruments Once we had agreed on the focus of investigation, we analysed various research instruments in order to find those that would be appropriate for assessing the areas to which we had assigned priority (those listed above). We found that practical and easy-to-implement instruments seemed to us to be the most appropriate. In order to make this part of the process more valid and reliable, certain contextual factors were taken into consideration. The research instruments which we ultimately chose were interviews, questionnaires, surveys, group activities and documentary evidence. 2.6 Data collection We then decided to interview a few teachers on an individual basis in order to arrive at an understanding of personal involvement in institutional change. We planned, in this way, to collect data about individual and institutional change. · Teachers were firstly asked to comment on their performance and on how they had viewed themselves before, during and after their training. To our surprise, we found that the unanticipated benefits and outcomes of the training that teachers received seemed directly related to the degree of their involvement.2.7 Data analysis and interpretation Once the data had been collected, we met to analyse and interpret all the data. We were both surprised and gratified as we identified, at the meeting, more evidence of both anticipated and unanticipated outcomes and benefits. This process also made us more aware that we needed to create the conditions which would maximise the potential inherent in the training, attitudes and willingness of teachers to participate more actively in the development of new programmes and projects in our institution. We may say, by way of summary, that the data analysis and interpretation stage made us aware of the hitherto unrealised potential of our situation, and this motivated us to embark upon new attempts to professionalise English language teaching. Such attempts were necessary if we hoped to improve the quality of the service given to the community - not only in our institution but also in other institutions, both public and private. 2.8 Analysis of anticipated and unanticipated benefits The unanticipated benefits, which were discerned in the process, were: · greater professionalism3 New goals, benchmarks and projects The research process played an important role in identifying the benefits of both the anticipated and unanticipated outcomes. The determination of unanticipated outcomes in particular played a profoundly significant role in shaping how the programme would be conducted in the future. The positive unanticipated outcomes illuminated possible ways of taking the project forward and also suggested other, related possible projects. It was for this reason that the assessment focused heavily on the implications of the unanticipated outcomes in our context and institution as well as on what these outcomes meant for the project and for sustaining practice. The positive outcomes (both anticipated and anticipated) were treated as explicit benchmarks and goals for our current and future projects and programmes – and hence for the sustainability and further enhancement of the project. As a result of the identification and definition of expected and unexpected benefits, we initiated several changes in our current programmes and projects and gave every encouragement to those who would have to carry them out. All these factors had far-reaching effects on project sustainability. 4 Dissemination of findings We then attended a conference in Tijuana, Mexico, to present, describe and share the data which we had collected, the anticipated and unanticipated benefits, and our perceptions of the impact which the changes had effected. During this same conference, representatives from universities in north-west Mexico presented the impact which the Mexican Advanced Professionalisation Scheme has made in their institutions. In addition, we organised a meeting in which we shared all details of our process as well as the expected and unexpected outcomes which we had identified and which are the subject of this paper. 5 Conclusion We should like to state, in conclusion, that there are implicit and explicit benefits which arise out of impact evaluation studies. The most important of these, in my view, are the following: · Impact studies raise awareness of the potentials and weaknesses of an institution.We have become aware that English language skills are an essential asset for the development of the Universidad Autonoma de Sinaloa and for the whole of Mexico, and that our role as teachers and/or teacher-trainers is of paramount importance for the development of these skills. Finally, we should like to affirm that the benefits of the findings for project sustainability were immeasurable.
|
