Evaluating Impact - Education Research Paper No. 35, 1999, 262 p.

[Previous Page] [Table of Contents] [Next Page]

2. THE ROLE OF BASELINE STUDIES IN IMPACT ASSESSMENT

2.1 School focused baseline assessments as a catalyst for change
2.2 A general framework for evaluating educational programmes
2.3 Issues to consider when planning a baseline study

2.1 School focused baseline assessments as a catalyst for change

Carol Moloney
Link Community Development Education Programme
South Africa

In 1994, when LCD first began conducting baseline surveys of the inservice training programmes which they facilitated, they used the data they obtained primarily for providing benchmarks against which future change could be assessed. There was a need to assess the impact of their programmes both for the participants themselves and for funding agencies who required proof that resources were being effectively utilised. With experience, LCD has recognised the value of participatory baseline surveys as developmental processes in themselves. This paper begins by outlining how a baseline study can serve as a catalyst for change.¹ In these instances, baseline studies are intended to show where there may still be room for improvement. (See cartoon drawings in section 3 for a depiction of the process.)

LCD stresses the importance of undertaking a school-focused baseline assessment of the situation in schools before an intervention programme begins. The study involves both

· confidential interviews with representatives of all school stakeholders, and
· classroom and school observations which lead to the development of school profiles.

Because the baseline process deals in depth with an analysis of needs it inadvertently also deals with an analysis of unforeseen issues and difficulties that frequently arise. To ignore these is to ignore the reality of beneficiaries' lives. Since beneficiaries, as Escobar (1995: 107) points out:

... are socially constructed prior to the agent's (planner, researcher, development expert) interaction with them..... This does not deter the agent or institution from presenting the results of the interaction as facts, that is, true discoveries of the real situation characterising the client.

2 Baseline surveys as a precursor to an intervention

Participatory baseline surveys endeavour to ensure that recommendations for change are based on a shared perception of the reality of the classroom. Teachers are often viewed as passive agents in the change process. Dalin (1990) suggests that such a philosophy rests on the following assumptions:

· Schools are seeking to improve and will recognise inputs as being beneficial to them.

· Technocratic issues take precedence over ideological questions.

· The teacher will mechanistically implement the changes produced for him/her by others further along the chain.

3 Collaboration as essentiel to the baseline process

One of the greatest strengths of a participatory baseline process is that its success is dependent on the collaboration of various parties. In the LCD process, a wide number of stakeholders are consulted (LCD 1997). Thus, for example,

· teacher unions are consulted for permission to appraise teachers in the classroom.

· departmental officials work closely with teachers who are engaged in assessing the teaching-learning environment.

· principals are requested to allow teacher evaluators who are elected by their peers to assess the situation in their schools.

· teachers themselves assess their peers and ratify their perceptions in collaboration with the external agents and the rest of the teaching staff.

STAGES OF A BASELINE

A DAY IN THE LIFE OF A BASELINE

This degree of intense collaboration in which various parties not only actively but are respected for their contributions, is an extremely valuable practice in itself, especially in the South African context. Sectors such as teacher unions, the education department and teachers in South Africa, have historically viewed each other with suspicion. The LCD baseline survey facilitates their working together on a joint venture. This shared process allows each party greater insight into the viewpoint and reality of the other and is important for establishing the trust needed for effective collaboration. According to Fullan (1991: 79), after such a process,

teachers and others know enough now not to take change seriously unless central administrators demonstrate through actions that they should... the policy maker on the one hand and the local practitioner on the other hand...to the extent that each side is ignorant of the subjective world of the other, reform will fail.

4 An application of Vygotsky to baseline studies

Vygotsky's (1934) theory of the Zone of Proximal Development (ZPD) can be drawn on to explain LCD's approach to evaluation. Vygotsky recommends that teachers work in the learner's Zone of Proximal Development. Vygotsky postulates that, through structured interactions known as scaffolding the teacher is able to facilitate the development of the learner's understanding and ability to perform a task which s/he would not have been able to do alone but will be able to perform independently after the interventions take place. It is believed that the learner's performance is, in part, a result of the setting and type of support offered.

The ZPD is not an attribute of an individual... but rather the attribute of an event.... Actual and potential levels of achievement are never just a reflection of an individual's cognitive potential and learning strategies, but are always also a measure of the strength of the cultural frameworks that supports that learning (Mercer 1994: 103).

5 Using the baseline study to introduce changed practices

One of the important policy changes of the education department under the democratically elected government in South Africa is that it has addressed the problem of poor teacher practice. Several policy innovations have been introduced to enable this. The policy requires that teaching shifts, from being content-driven and teacher-centred, to being outcomes-based and learner-centred. How this paradigm shift will be successfully implemented on a large scale remains an enigma.

Enabling teachers to shift from their current content focus towards an outcomes orientation within the broader system of a country's education presents an enormous challenge. If the reconceptualisation of philosophy and the practical changes required are to be sustained – and if they are not going to be superficially adopted as token elements of the 'new order', then teachers have to be given opportunities which allow them to internalise what this shift might mean. They will also have to be given opportunities to try out new approaches in a safe environment. The focus on specified learner-outcomes and on the achievement of critical outcomes² in accordance with South Africa's OBE curriculum is central to LCD's conception of the baseline assessment.

The inclusion of this alternative philosophy in LCD's baseline studies enables those involved in the evaluation to reflect on their understanding of the curriculum shift and to test out the skills required to implement changed practice. Consequently, as well as enhancing ownership of proposed project interventions, participatory evaluations can also serve as an educational tool. LCD thus works on the premise that through including teachers in the baseline, the new skills attained and the reflections induced go some way to facilitating the paradigm shift that teachers are required to make within the new dispensation. Hence LCD argues that participatory baseline evaluation procedures have potential long-term benefits. In more immediate terms, a baseline evaluation provides the space for educators to reflect, expand their repertoire cognitively and practically, and be supported during this process.

Collaboration is not the panacea of educational reform. To undertake such a process leads educationists along a sharp incline of learning, along a path filled with potholes. One cannot simply sail to the end of the road and hope to encounter sustained success. One of the reasons for this is that people are not accustomed to collaborating. Participation is a skill that disadvantaged South African teachers need to learn. They need to learn to be transparent with colleagues and with themselves, to share, and to listen. These are stepping stones which should be acknowledged and planned into the process. Failure to do so can create frustration and resentment on all sides. Having personally erred by not having built in these skills, I have become very aware of the pitfalls of participatory ventures. I will draw on the following account of the Soshanguve³ baseline to illustrate lessons to be heeded for future ventures.

5.1 CASE STUDY of a baseline study undertaken in Soshanguve

Perhaps it is because I come from a teaching background myself that, in this project, I had far greater empathy with the teachers involved than with the Department officials who were to participate. Consequently, when undertaking this project, I took far more care to ensure that the participating teachers were encouraged to contribute to the process and that their contributions were valued. I also took care to ensure that that they understood what was required of them, and saw to it that they felt sufficiently confident of the support system we provided. In retrospect, I realise that this same consideration was not offered to the seven departmental subject advisors who were included in the evaluation team. If participation is to work effectively, all participants should feel comfortable with the approach. We at LCD recognised that, for the team which was brought together for this assessment process, this level of inclusivity constituted quite a radical change in methodology: teachers were accustomed to receiving knowledge from the Department and the latter was used to transmitting this. Such an arrangement ensured that no one really had to deal with the situation on the ground. This in turn meant that neither the Department officials (espousing reforms) nor the teachers on the receiving end felt confident about implementing new innovations in the classroom. The practical nature of the preparatory baseline workshops where, for example, indicators of effective teaching were being developed, expected all participants to demonstrate their understanding of the reality in the classroom. Naturally on 'home ground', the teacher-evaluators felt quite confident with the task and made relevant and creative contributions. It was notable, however, that despite encouragement, the majority of the Department officials declined to take part. This marked the start of a pattern which was to re-emerge on different occasions during the two-week workshop. For example, the District subject advisors would constantly retreat to working and talking amongst themselves - even though they were paired with teachers for various activities. Another shortfall of the Soshanguve baseline was that the intensity and pace of the assessment process, once underway, did not really allow for reflection-in-action (Schon 1983). Hence their energies were devoted to reflecting on the product, the findings in schools and on creating accurate pictures of typical scenarios. They were not concerned about the process of going about change. Fullan (1991) points out that change agents who fail to reflect critically on how they go about change, lose out on improving their situations and on a lot of learning. Having had the opportunity to reflect, I realised retrospectively that LCD had erroneously moved too quickly from the process of preparing the evaluation team to undertake the school-based baseline assessment to the formation of a joint forum (of teachers and department officials) once an agreement and initial contacts had been signed. This rapidity did not allow the Department officials to work through their understanding in the relative comfort zone of the Department. It is possible that, because teachers felt less was expected of them, we noted only minimal hesitation on their part to express reservation or confusion. Some of the officials, on the other hand, felt a great reluctance to expose gaps in their knowledge or skills and contributed with excessive caution. It was quite evident at that time how disempowered the process left the subject advisors with regard to the new curriculum reforms with which they themselves were grappling while they were, at the same time, being expected to train teachers. What we were indeed failing to do, in terms of Fullan's theory, was to allow the subject advisors the time, space and support required to develop their own meaning about the changes that the participatory baseline assessment would imply for their own practice.

The difficulties encountered by the participants could be explained by the following quotation taken from Fullan's (1991: 31) citing of Marris (1975):

When those who have the power to manipulate changes act as if they only have to explain, and when their explanations are not at once accepted, shrug off opposition as ignorance or prejudice, they express a profound contempt for the meaning of lives other than their own. For the reformers have already assimilated these changes to their purpose, and worked out a reformulation which makes sense to them, perhaps through months and years of analysis and debate. If they deny others the chance to do the same, they treat them as puppets dangling by the threads of their own conceptions.

The failure to do just this became evident when, as with Vygotsky's theory of ZPD, the learner – in this case the Department officials – attempted to enact the baseline process independently. Although the Department coordinator thought that she was replicating a participatory evaluation approach, the lack of internalisation of the concept of participatory was evidenced by the authoritarian mode that she proposed. Principals and teacher unions from the District rejected the process and requests were made for the LCD approach to be followed. This highlights how a participatory baseline survey cannot be viewed as an isolated event, but rather as one step in an on-going developmental process. Fullan (1991: 92) cogently sums this up by indicating that 'ownership in the sense of clarity, skill and commitment is a progressive process. True ownership is not something that occurs magically at the beginning but rather is something that comes out the other end of a successful change process.'

6 Conclusion

In conclusion, it must be restated that the approach used by LCD in undertaking baseline studies is predicated on the premise that reflection has to be grounded in a philosophy which prioritises collaboration and learner-centredness and which is procedural. Although this approach provides a modus operandi for doing baseline studies, it still fulfils its expressed intention of informing the proposed intervention. The inclusive, participatory activity requiring the collaboration of various stakeholders is crucial to developing a sense of ownership among all stakeholders. It also helps to encourage a collaborative mode which is engendered by the participants having to work through the various stages of the baseline assessment. It is here argued that the baseline, both as a modus operandi for doing baseline and as well as a research approach for informing proposed interventions, is a process which is necessary for ensuring the sustainability the proposed project. Collaboration and sharing cannot be underestimated in the South African context. Should they be underestimated anywhere at all?

Footnote

1. It is important to note at this point, that since LCD would like to see itself as a learning organisation it has built in a component dealing with baseline studies as part of its teacher development programmes.

2. The critical outcomes are internalised via the teaching-learning process: they deal with learners being able to solve problems, work in a team, collect, evaluate and communicate information etc.

3. Soshanguve is a township north of Pretoria in South Africa. It was a township developed during the apartheid era as a black urban settlement area The township is disadvantaged and a large portion of the population live in squatter and informal settlements The name of the township is an acronym formed by taking the first letters of the different ethnic groups living in the area (The Sotho - So; the Shangaans - Shang and the Venda - Ve.) This was characteristic of the 'creative names' used by the apartheid regime.

2.2 A general framework for evaluating educational programmes

Samir Guha Roy
Indian Statistical Institute

Efforts which engage the community in intervention programmes intended to improve the delivery of education are relatively new, but since the introduction of the District Primary Education Programme (DPEP), there has been a growing interest in the scientific evaluation of such programmes. To develop a systematic and sustainable framework for evaluation, a wide range of people at different levels needs to be involved in the creation of an evaluation culture. Because of this, certain activities relevant to human resources development need to be initiated with a view to evaluating the mid-term and end-of-project impact. Because of the community focus of DPEP, training needs to be undertaken at the most local level (i.e. teacher and village level). If this is done, then qualitatively upgraded human resources can become effective partners in sustainable programmes.

The general principles of programme evaluation apply in the field of education (as they do in other fields). These include:

· defining the objectives of the programme

· selecting the criteria by which performance can be judged and defining the methods of measuring them

· deciding on the logic or design of evaluation

· collecting and analysing data (such as test scores and socio-economic background information)

· providing interpretations of the findings to the programme administrators

As in other fields, an intervention programme may be evaluated in terms of a hierarchy of objectives. A programme is usually conceived as having an ultimate objective. From this objective, a series of subsidiary objectives is derived. Each of the sub-objectives (or programme execution objectives) is a means of achieving the objectives at the next higher level, and these objectives may be termed programme impact objectives. This type of conceptualisation makes the programme evaluation process more orderly and sensitive.

Evaluations seldom address the difficult issue of impact because many factors in addition to programme activities may be responsible for influencing change. This problem may be overcome by using:

· Control groups

One possible approach to overcoming this problem is through the use of classically designed action and control groups. If the vagaries of social and economic changes unrelated to the programme are to be properly accounted for, it is necessary to introduce replication and use several control and experimental areas.

· Baseline data

Another way of overcoming this problem is by concentrating on obtaining firm baseline data before the programme is initiated and periodically thereafter so as to detect any trends.

· Factorial concept of experimentation

A more effective approach is the factorial concept of experimentation in which all possible combination of factors are investigated. Assessment of students' academic achievements is an important component of impact study. The concepts of item bank and test equating may be utilised to locate the learners on the same scale tested by different sets of tests in different regions over time.

What follows now outlines the various ways of classifying designs for evaluation.

31 Distinguishing forms of assessment

· Formative – summative
The distinction between formative and summative forms of assessment is aptly illustrated by an example by Robert Stake (1976) who indicates that 'When the cook tastes the soup it is formative evaluation, and when the guest tastes the soup it is summative'. Which form should be used? The evaluation team for the Indian state of Andhra Pradesh District Primary Education Programme (APDPEP) recommended both types of evaluation. Other distinctions need to be considered. They are:

· Formal – Informal
Formal evaluation is more operationalised and less personal. It must pass the tests of reliability, validity, credibility and utility.

· Case Particular – Generalisation
Evaluation research may be done either to find the worth of the particular programme or the worth of the general approach.

· Product – Process
A study of the product is expected to indicate the pay-off value while a study of the process is expected to indicate the intrinsic values of the programme. Both are needed, however, to find the worth of the programme.¹

· Classical Design for Impact Study

Classical Design for Impact Study

Measurements
Time	Project area	"Control" area
0	X0	Y0
	Programme treatment	No Programme treatment
1	X1	Y1
2	X2	Y2
·	·	·
·	·	·
·	·	·
·	·	·

(X₀ @Y₀)

In the above scheme, x's and y's are any educational measurements. A valid estimate of programme impact at the end of time², say, will be

Impact = Ix₂ - y₂ - Ix₁ - y₁l

provided the two areas are exposed to the same exogenous factors.

4 Issues identified for assessment in the Andhra Pradesh District Primary Education Programme (APDPEP)

The key issues identified for assessing the impact of the APDPEP are:

· the state of capacity building for programme implementation
· levels of community participation
· the nature of equity focus
· the development of classroom processes
· the effectiveness of teacher training

Students' mean score

	No programme package	Programme package implemented	Mean	Response to programme
Community participation (CP)	X	Z	(X+Z)/2	Z-X
No or little CP	Y	W	(Y+W)/2	W-Y
Mean	(X+Y)/2	(Z+ W)/2
Effect of CP	X-Y	Z-W

4.1 Evaluating the accomplishment of a programme

Answers to the following questions provide the background against which the accomplishments of a programme may be evaluated.

· What are the specific changes being sought? What are the conditions in the programme area to which a programme must be adjusted if it is to attract the active support of the people?

· What are the channels of communication for the effective flow of education from project administrators to the people and for a flow of the attitudes and responses of the people to those responsible for the project?

· What are the barriers that must be overcome if the project is to achieve its objectives?

4.2 The development of an item bank for pupil assessment

· To start with, a number of tests may be constructed following anchor test design applicable to a particular level or class. Collected data will be analysed and items will be calibrated on a common scale. These items will be the initial deposit to the item bank.

· Similar item banks may be established for different classes or grades with common items between them. As a result, common scales may be framed for the entire target group (vertical equating).

· Any number of parallel tests may be constructed without further cost or delay. Moreover, post facto analysis of the test data and removal of poor items can avoid pretesting on every occasion.

· Maintenance of test security will also not be vitiated even if all the items are made known. This is because mastery of the items and mastery in the subject concerned are almost synonymous.

In this paper, Roy points out that programmes that engage the community in the actual intervention to improve education delivery are relatively new. While the participatory nature of DPEP has gained ground through the areas of intervention, this paper argues that there are nevertheless a number of limitations on non-scientific approaches when attempts are made to assess impact.

The author begins by drawing attention to the hierarchy of objectives of intervention programmes. He suggests that evaluators usually steer away from addressing the difficult issue of impact which may be caused by many factors which are quite distinct from programme activities, and he points to the difficulties of distinguishing between possible activities which could be responsible for influencing changes. The paper suggests various ways of controlling the investigation to enable the impact of project activities to be evaluated. It argues that there is a growing interest in the domain of project assessment in the scientific evaluation of such programmes. The paper concludes with a proposal of how impact can indeed be assessed.

Footnote

1. The other common dimensions that Stake (1976) mentions with regard to the classification of evaluation designs are: Descriptive – Judgmental; Preordinate – Responsive; Holistic – Analytic; Internal – External.

2.3 Issues to consider when planning a baseline study

Tony Luxon
Institute for English Language Education
Lancaster University

Over the past decade, the methodology and the philosophy of baseline research for projects in all types of social and educational contexts has evolved a great deal - although not to the point where there is an agreed model for baseline studies. The principle reason for there not being a single agreed-upon model is that because projects are context dependent, they are all different. They vary not only in their objectives and activities, according to scale of resources available and area of focus, but also in the social, cultural and educational environment in which they are carried out. Thus, what is appropriate for projects in countries of the former Soviet sphere of influence, might be inappropriate in, for example, the African continent, where there is a different educational tradition, and a different socio-economic organisation.

It would nevertheless be wrong to assume that nothing that is achieved in one context cannot allow us to learn lessons for other situations, or that the methodology for carrying out baseline studies in different contexts is completely different. Although there are differences, there are also inevitable commonalities. Baseline studies invariably give some form of 'snapshot' of the project environment before its activities are under way -and they usually have an evaluative and developmental function. Also, there is usually some form of survey of stakeholders and potential beneficiaries of the project.

A survey of baseline studies in which the IELE at Lancaster University has been involved shows this clearly, and, while a totally reusable template does not exist, there may well be issues which need to be considered each time an activity of this kind is undertaken. In the sections which follow, I suggest what some of these issues are and why they might be important for someone contemplating baseline research regardless of where it is carried out. How some of these issues are dealt with is still a matter for debate, but they at least need to be considered.

2 Issues pertaining to the needs of the project implementation team

Most of the baseline studies in which the IELE has been involved have been carried out by the project implementers. Whether this happens or not depends on the capacity of the project team. The issue here is whether the exercise will be one of capacity identification or capacity building. The following three issues deal with suggestions pertaining to the team carrying out the project.

2.1 Where possible, the project team should carry out the baseline study

In many of the projects in which IELE was involved in Eastern and central Europe, for example, the capacity in the area of ELT and linguistics was already very strong, as was the research tradition. Therefore, the main task was to find a combination of the most suitable people to constitute a project team. In these cases then, technically, there was no reason why the project implementers could not carry out the baseline study for their own project. The question of insiders' disinterestedness and objectivity towards the project environment is one which is regularly discussed in the literature of projects and project evaluation.

Even if we could agree on the parameters of objectivity, it cannot be automatically assumed that an outsider to the project is by definition more objective than an insider. As Alderson (1993) points out, outside evaluators bring their own agenda to the exercise, their own beliefs about evaluation, education and about the project environment. Outsiders may have to spend much of their time trying to understand the environment, and it is possible that their understanding will at best be partial and at worst wrong. Because of their outsider status, they may be less prone to influence from the variety of players connected with the project (although this cannot be assumed). At any stage of the exercise, when they do not have first hand knowledge of the project or the project environment, they may have to make decisions on whose judgement is reliable. They may well therefore be influenced precisely because of their 'outsiderness'.

Time is also an issue that impinges on decisions about whether the baseline study should be an insider- or outsider-led exercise. It might be difficult for an outsider, precisely because of time and money constraints, to stay within the project environment for two or three months. This needs to be borne in mind if it is agreed that the minimum time that it generally takes to carry out a baseline study for a large-scale three year project is approximately three months. If this were to be the case, then the cost of maintaining an outside consultant in the project environment could well be prohibitive.

In the case of projects in St Petersberg and Ukraine, for example, the baseline studies were very extensive reports with a wealth of data that could only have been carried out by a team of people with access to information about testing, classroom performance and interested stakeholders. It simply would not have been feasible for one or even two outside evaluators to have carried out this exercise in anything like this kind of breadth and depth.

· Insider-led baseline studies

In the case of these insider-led baseline studies, Lancaster played a consultative and training role, as and when needed. As mentioned above, the capacity was already more than sufficient in these contexts, but there was a perceived need on the part of the implementers themselves for consultancy in this particular kind of 'real-world, research. Although many of the team members had been involved in research prior to the project, they usually came to the project with no experience of working in a project or of the type of research experience such as, for example, classroom observation, instrument development or data analysis. Where it was possible to call on people in the host institutions for consultancy or training in any particular area, this was done¹.

· Outsider-led baseline studies

At this point it is worth considering the few examples of baseline studies which were not carried out by project implementers (or, at least, not entirely so). In the case of the Philippines, where the total management of the project, including policy, administration and budget was managed by Filipinos, a consultant from Lancaster carried out a very small-scale qualitative baseline study with the aim of determining needs and not for use as an evaluative instrument. It was felt that a 'new pair of eyes' introduced into the situation might reveal things which had not clearly been hitherto seen.

In Nicaragua, the baseline research was carried out by the British TCOs for the ODA funded ELT Project because it was not clear at the outset what type of project was needed (no research had as yet been undertaken). This also meant that there was no Nicaraguan implementation team. With no team, the only identified project implementers, the British TCOs, were required to do the research.²

· Joint insider-outsider baseline studies

In the case of a baseline study carried out in Cambodia, there were actually two reports. One was produced by the project team and was based on a qualitative exercise which focused on the activities in the schools, and the other, which concentrated on the project framework indicators of achievement, was produced by the outside consultant. Because the baseline study was seen as an important step in establishing ownership, it was important that it was not simply a case of the outsider consultant taking over the writing of the report with the assistance of the implementers. Furthermore, whereas the implementers neither regarded the project framework as very important nor felt that it would be particularly useful to them to base their report on the indicators, the project managers felt that the indicators might be important to the project sponsors. For this reason, the two reports were produced: one to do with evaluation and one to do with development and capacity-building.

This first consideration, that of who carries out the baseline study, is one of the most fundamental questions about which a decision needs to be made. Once it is made, other issues then become significant.

The baseline study, if considered necessary, needs to be written into the project and have resources allotted to it. If this is not the case, then the team carrying out the exercise might run out of time, money and the stakeholders' patience. Research is often seen as about something rather than being an integral part of that something.

2.3 Consider what the project team might need in order to carry out the baseline study

It is necessary to give consideration to the type of training that might be necessary for participants involved in conducting a baseline study. At this stage, the kind of communication system necessary to link members of the team is also an important consideration, especially if they do not belong to the same institution. For example, it may be necessary, as was the case with the project in Ukraine, to facilitate communication between cities in different parts of the country by e-mail. Assistance from the British Council was sought to facilitate the introduction of this mode of communication. It was also necessary to meet periodically (it is crucial to make sure that such meetings are arranged and funded at the outset of the research).

3 Methodological considerations

The following seven suggestions pertain to the way in which baseline research is approached. They refer to methodological issues such as the development of instruments and the types of data that baseline studies should seek to capture.

3.1 Be aware of both evaluation and development issues. Take advantage of the capacity-building/identification and communications opportunities which arise through the process.

If the project implementers themselves are to carry out the baseline study, then this is an extremely valuable opportunity to develop capacity for the ensuing project. Indeed, it is here argued that the baseline research process should be seen as the first activity in the implementation of the project rather than as a pre-project exercise.

It might be the case that project members do not know each other well. Training seminars and workshops, cooperative working on research and the writing of the report are all opportunities for team building. Training sessions give the first opportunities to see how well, or badly, teams work together.

If the baseline study is treated as a mini-project in itself, then there are valuable project or innovation skills to be learned from carrying it out. Team members may well be professional academics in ELT and Applied Linguistics, but it cannot be assumed that they have the necessary skills to deal with other agendas. Buchanan and Boddy (1992: 28), refer to three agendas in an innovation context: the content, control and process agendas. These can be described as follows:

· The content agenda:

The project manager is expected to be technically competent and experienced with respect to the substance of the changes being implemented. Thus, for example, he is expected to be competent with the hardware and software of a networked management information system.

· The control agenda:

The project manager is expected to be familiar and competent with a range of planning, scheduling, budgeting, resourcing and monitoring techniques, with setting and meeting deadlines and targets - the staple fare of project management courses.

· The process agenda:

The project manager is expected to be competent in communications and consultation, in team building, in influencing and negotiating skills and in the management of enthusiasm and resistance.

In the Cambodia baseline study referred to above, the team which carried out the research was the actual inspection team for ELT. Their duties during and after the project were to visit schools, talk to teachers, students and school principals, observe classes and look at test results. All these activities were included in the baseline research, and so, as well as being a valuable piece of research, the team also went through a process from which they could learn a great deal about their future responsibilities.

Similarly, in the report produced after the Ukraine INSETT baseline study, the team members referred to what they had gained through the process:

The challenge to the team For the majority of team members this was the first experience of research work of this nature, involving the close study of situations and attitudes in the real world. While the team acknowledges mistakes were made due to lack of experience, we can identify several gains made as a result of work on this study: · All members of the team have gained experience in a mode of research which is very new to Ukraine, namely research which is centred around professional practices and carried out by practitioners. · Team members have a clearer grasp of the issues surrounding approaches to ELT. · Team members have become familiar with a method of observing lessons which allows the observer to assess all aspects of a lesson in much more detail than traditional observation techniques allow. · The groups of secondary stakeholders who are rarely, if ever, consulted have been given the opportunity to consider some of the issues surrounding ELT and to express their own views on the current state of ELT in Ukrainian schools. · Secondary stakeholders have been made aware of the possibility of change.

Source: Baseline Study Report, Ukraine INSETT Project- 1998

The baseline study, both through the process and its product, can be an important means of establishing credibility with the project beneficiaries and the stakeholders. As mentioned previously, the Nicaraguan ELT project baseline study was carried out by the British project coordinators, principally because there was not, at the time, a locally-based project team. The research process enabled the coordinators to meet teachers throughout the country and to familiarise themselves with the ELT situation in schools throughout the regions. They also met most of the stakeholders of the project during this process. They found that this provided an invaluable opportunity for making themselves known to all involved. In fact, the research was valuable because it uncovered many relevant issues pertaining to ELT in Nicaragua. This happened because the project was the first national ELT research investigation undertaken in the country after a period in which English had been, in the words of the Director of Education, 'abandoned'. This made it possible to say things about ELT which were not mere assertions and also to provide the rationale for project activities from then on. It is fair to say that, without the baseline study, it would have been much more difficult to carry out the activities of the project.

In all these instances, the opportunities for establishing communication, for team building, capacity identification/building and the opportunity to establish the credibility of the project implementers would have been lost had the baseline study been carried out by an evaluator who had not been part of the project. As Weir and Roberts (1994: 218) point out:

While we know that the collection and analysis of data should meet the standards of feasibility and accuracy, we have also learnt that positive interpersonal and institutional relationships must underpin technical adequacy, and are at the heart of effective evaluation: this is because relationships of commitment and trust enable the involvement of players in the evaluation process, and the utilisation of findings. We have learnt that the importance of these relationships must be taken into account from the very outset.

Although it is difficult to estimate, three months may be considered to be the average length of time during which a large-scale project is likely to spend on a baseline study. If we are considering the implementation of a project with a lifespan of three years, this will account for less than ten percent of the project time. In view of the developmental value of the process and its influence on the future activities of the project, this is not excessive.³

3.3 If possible, collect both qualitative and quantitative data

Often stakeholders prefer quantitative data because of its potential to account for things in a countable way. However, not everything is quantifiable. Nor is it possible to gain an in-depth understanding of a situation without also using qualitative methods of data collection. If both qualitative and quantitative methods are employed, a more comprehensive and holistic understanding of a situation is possible.

There are nevertheless aspects that can be counted, as, for example, the number of teachers trained through a teacher training project, or the number of books distributed through a textbook/materials writing project. And such methods are useful for calculating differences in examination results.

Changes in teachers' attitudes and behaviours are more difficult to quantify as are predicated on the types of interaction in a classroom. It is possible to quantify the quality of experience in some ways if one uses custom-made instruments. This means that the quantifying classroom behaviour can tell a story as vividly as a prose description. There are many benefits to processing qualitative data in a way that permits it to become countable. The IELE attempted this in the Nicaraguan baseline study. Instruments were developed specifically for the project. This enabled data from classroom observations to be quantified in terms of, for example, teacher talking time, the amount of Spanish used in relation to the amount of English used in class, the kinds of interactions which took place between teachers and students, and between students and students.

These results were then compared with the data of teacher interviews which were more qualitative in nature and were concerned with why they taught the way they did rather than what they did when they taught. It may be argued that qualitative and quantitative methods of analysis do not look at the same kind of thing and so 'true' triangulation is not possible. This may be so. Nevertheless, the sheer accumulation of complementary qualitative and quantitative data makes for a comprehensive and, for many, a convincing picture of what is happening.

3.4 Collect data from a variety of sources so as to allow for a variety of perspectives

The issue of triangulation of data has been given much attention in the literature on project evaluation. Triangulation helps to counteract the Rashomon effect, as Fanselow (1987) calls it: which is the effect of a variety of perceptions. There are clear epistemological implications in research of this nature, but data collection from a variety of sources seems, to me, to be the most logical way to deal with the issues regardless of one's perspective. It is possible to take a non-realist, relativist point of view, and yet accept that as long as perceptions are recognised as such, what is reported may be considered to be valid. Likewise, a critical realist might believe that the truth is out there, and regard a multi-faceted approach as one of the best ways of gaining access to it. However, as there are so many stakeholders with varying views of the project environment, it is necessary to report their perceptions as a matter of record.

3.5 Consider which data already exist in documented form and which data need to be collected by using instruments

The kind of data which need to be collected may already be available in one form or another. It is therefore unwise to attempt to re-invent the wheel. A useful beginning would be to try to survey relevant reports carried out by international organisations, such as UNICEF and UNESCO, or local government organisations and NGOs. Certainly statistical information, which might be obtained through the ministry of education and triangulated with other sources, can be used as valuable contextual data. If the information has been collected through a reputable agency, this has the added advantage of increasing the credibility of the report.

Documentation which relates to the curriculum, or to teaching and learning philosophies, for example, may already exist. Even if the documentation is not as accurate or as comprehensive as it might be, this is in itself useful to know.

Data on what happens in classrooms will probably need to be collected by visits and observations, and instrumentation may be developed to collect this data. It is possible that other research may have already collected relevant classroom data and, if this is so, it may be usefully incorporated into the study. However, there is no real substitute for the team going out and investigating the situation themselves!

3.6 Whatever the type of project, always visit the classroom

This suggestion may seem to recommend what is blindingly obvious, but it is an issue that is often overlooked. Very often baseline data consist of quantitative data on book distribution, for example, but they might not give any idea as to how the book is used in class by teachers and students, something which it is vital to know if the supply of books is to prove effective in the classroom. Similarly, while it is important to know about desertion rates, it is also useful to know how students react to what they are learning in the class as this may be a factor which contributes to desertion rates.

Whether the project implementers be insiders to the target situation or outsiders, only visits to the classroom will enable them to gain an understanding about what happens in the teaching-learning situation. It may well be the case that project members are or have been teachers themselves, but it is surprising what they may discover about how much they do not know.

3.7 Consider the possible uses and audiences there might be for the baseline study, and allow for new uses discovered through the process.

It is important to realise that the results of the baseline study may be read in a variety of forms by different audiences and may also be used in ways that were not envisaged at the outset. The Nicaragua baseline report was used in the following ways:

· It was used as reference material for the Ministry of Education and the universities which, until that time, had done no formal research into ELT. It was also used as a reference for anybody else, Nicaraguan or foreign, who wished to carry out research into ELT in the future. A number of researchers from North America used the baseline study for their own research, as did four Nicaraguan project members, who wrote dissertations on ELT in Nicaragua. The baseline was therefore a major stimulus to much needed further research.

· It was used as a briefing document for consultancy and ODA monitoring visits. Having this kind of information available made these visits more effective in a shorter time. Other aid organisations working in ELT and in the secondary and tertiary systems in general also used the baseline study to inform their own work.

· It was used as an aid to the overseas training institution in order to provide appropriate training for project trainers. Overseas training is not always appropriate to the local situation. The baseline study, along with visits from the UK training institution, helped to make the training relevant and appropriate.

· As mentioned earlier, the baseline report was used as the basis for the diffusion process. It enabled the dissemination information about the project to as many teachers, directors and officials as possible. It also contributed to the establishment of an ELT communication network.

The baseline report offers benefits to many of the stakeholders. For this reason, the findings of the study should be transmitted to its many audiences in appropriate ways. The following are suggestions of ways to enable this:

4.1 Allow for a variety of channels through which the findings of the baseline study might be transmitted to its audiences.

It may be argued that all stakeholders and beneficiaries should have equal access to the report and that decisions taken by the producers of the report or any other concerned stakeholder could disempower those who cannot take those decisions. I would hold this as a valid theoretical position, but would want to say that reality must intervene at some point. Depending on how wide a target grouping of beneficiaries is, it is not likely that so many would actively want to read such a report. Furthermore, the distribution of such a report among, for example, 5000 teachers, would be prohibitive in terms of cost and logistics.

Copies could be made available in resource centres (if they exist) or at regional ministry offices, if they are ever visited by teachers. The report should be made available as widely as possible but I would suggest that it is not realistic to expect that many people will actually want to read a lengthy report, which is unlikely to reflect the kind of information that automatically grips one's attention on every single page.

Who should receive a copy is not necessarily a question of power, but rather of real accessibility. It may have more to do with the preference for quicker and more striking ways of reading the results. Although it may be necessary for a certain type of audience (academics for example) to see the whole report, this is not necessarily the case with all audiences. In Nicaragua, a shorter, more graphic report was produced for people who did not want to, or have the time to, read the report in full. However, had anyone wished to read the complete report, it could easily have been made available. The shorter report was more easily accessible and, because of its graphic nature, the results were more clearly shown. Policy makers and other beneficiaries who were concerned with the central findings, but not with the details of the main report, seemed to prefer this report.

4.2 Keep stakeholders and target groups as Informed as possible throughout the process so that they know what kind of report to expect

If, as indicated above, it takes on average three months for the research on a national scale to be carried out, and then another relatively lengthy period while drafts are written and findings are discussed. In the duration, it is important for stakeholders and beneficiaries to know what kind of report is being produced and what will be covered in it. Our experience suggests that there should not be any great surprises in the report, and that people need to be given a chance to add their contributions to the report before it is finally produced. An interim report can be of immense value in encouraging input from stakeholders.

5 Conclusion

Finally, it must be stated that all of the above issues have been addressed by others in one form or another in baseline study exercises in various parts of the world, and in studies undertaken in diverse social and educational conditions. It is certainly not the contention here that this paper presents a universal set of measures for dealing with these issues in any conditions. Projects are of necessity context-sensitive - as should be the research on which projects are based. It is suggested rather that these issues will need to be addressed by those who carry out project baseline research. What needs to be considered afresh in each baseline study is Who should be involved? and What is the most appropriate approach for proceeding in this context? I would argue that if this initial process can be successfully dealt with, then the possibilities for the success of the ensuing project are increased enormously. It is well worth putting effort into the baseline study. It is, after all, the first step on the journey of a thousand miles.

Footnote

1. In the case of data analysis, for example, although none of the team had used SPSS, they were able to contract somebody in their institution to enter the data into the package and assist them with the analysis.

2. The situation was different at the end of the project when the impact assessment involved a team of 22 project implementers who had worked together throughout the project.

3. It is recognised that projects differ in scale and this rightly should affect how much time and effort is spent on baseline research.

[Previous Page] [Top of Page] [Next Page]