The role of evaluation in ensuring excellence in CIT programmes

As a result of the problematic features of CIT programmes, outlined in the previous section, their evaluation makes special demands on evaluators. In this section I will put forward an approach to the evaluation of CIT programmes which is largely derived from my own experience, but also draws upon the evaluation literature.

During the last 25 years considerable research has been done to develop robust strategies for evaluating programmes. In an early study, House (1980) deﬁ nes eight separate approaches including ‘systems analysis’, ‘behavioural objectives’ and ‘case study’. He does not recommend any one approach, but concludes (ibid., p. 256) that, ‘Public evaluation should be democratic, fair, and ultimately based upon the moral values of equality, autonomy, impartiality, and reciprocity.’ Ultimately evaluation is about making value judgements but doing so on the best possible evidence. After 30 years experience, Cronbach came to the conclusion that:

The evaluator should almost never sacrifi ce breadth of information for the sake of giving a defi nite answer to one narrow question. To arrange to collect the most helpful information requires a high degree of imagination, coupled with the fl exibility to change plans in midstudy.

(Cronbach 1982, p. xii)

Currently, there is a preoccupation with the tension between the educative and the accountability purposes of evaluation that I referred to in the ﬁ rst section of this paper (Greene 1999). McEldowney (1997, p. 176) refers to these more explicitly as the ‘control’ model and the ‘helping’ model of evaluation. He explores the concepts of ‘deadweight’, ‘displacement’ and ‘additionality’ as means of judging the value for money of a programme, but concludes that although this is a useful approach:

The role of evaluation 137 (It is) often without regard to the difﬁ culties and limitations that exist in

measuring these concepts (and is often at the expense of ) a consultative style where a transfer of learning should occur based on timely feedback to program managers about the effectiveness of patticular interventions or measures and about the achievement of the objectives set for the program in question.

(McEldowney 1997, p. 186)

Those who see evaluation as primarily an educative process place emphasis upon its utilisation (Patton 1986). Others such as Stake (1998, pp. 203–4) see their role as ‘ﬁ nding and understanding quality’ and warn against becoming ‘collaborators in redevelopment’. The tension between these two approaches is now one that evaluators frequently need to address, since currently there is considerable interest in approaches to evaluation which take into account the needs of multiple stakeholders (Somekh et al. 1999).

Some of the most interesting recent work has reported on various approaches to evaluating multi-site programmes. In the case of European Social Fund programmes, evaluators have the opportunity of looking at similar initiatives implemented in different countries, with the possibility of determining the effects of adopting different strategies. Barbier and Simonin (1997, p. 396) look at two levels of imple- mentation: ‘the political, administrative and ﬁ nancial level’ and ‘the street level’ but cite Hall (1996) in concluding that comparisons are difﬁ cult as ‘all actors involved have objectives of their own; the more numerous they are, the more objectives and points of view there are’. In the case of CIT projects funded by the EU each partner normally contributes in different ways to each work package and takes responsibility for different deliverables. Nevertheless, there is potential for much greater depth of understanding when working with more than one partner. Large-scale multi- site evaluations require good organisation and good communication. The numbers of partners involved at meetings can pose problems, hence some evaluators have developed tools for structuring discussion and consultation (Beywl and Potter 1998). The Kellogg’s Foundation’s ‘cluster evaluation’ approach is of particular interest. It has four key characteristics:

1 It looks for common threads and themes across sites.

2 It seeks to learn why things happened as well as what happened.

3 It is highly collaborative, encouraging all players to participate.

4 It maintains conﬁ dentiality between the external evaluators and the projects (i.e. project teams had enhanced control over the release of information beyond their project).

(adapted from Worthen and Schmitz 1997, p. 303)

Evaluations of CIT in education fall into two broad categories: evaluations of funded programmes and evaluations of general uptake as a result of on-going policies and accumulated spending on CIT within the education system. An important evaluation of the general uptake of CIT in the US education system was the 1998 national Teaching, Learning and Computing (TLC) survey (Becker 2000), which

138 Research methods for ICT in education provided rich contextual information on teacher beliefs and attitudes, usage in

teaching and the location of equipment, as well as information on the numbers of computers in schools. It achieved a response rate of 75 per cent. A very early evaluation of a computers in education programme was House’s analysis of the PLATO programme which began in the USA in 1960. According to House (1974, p. 188), the political imperative for PLATO to succeed was so great that the reality of the technical problems, poor-quality ‘lessons’ and excessive time demands upon the teachers developing materials were not acknowledged, and it continued to receive funding for a number of years. During the 1990s the focus of CIT evaluation in the USA was on meeting the requirements of the 1993 Government Performance and Results Act (GPRA) for performance measurement (Greene 1999, p. 161). This led to the production of materials to assist administrators/managers in evaluating initiatives in-house, such as the ‘framework of 7 inter-dependent dimensions’ for planning, implementing and evaluating initiatives, produced by the Milken Exchange on Education Technology (1998); and the ‘evaluation tool kits that can be used to support local and national studies of the educational uses of computers, video and telecommunications’, produced by the Flashlight project supported fi rst by the Annenberg Foundation and later by the American Association for Higher Education (Ehrmann 1997). To ensure quality in a fi eld increasingly open to newcomers, the Joint Committee on Standards for Educational Evaluation, sponsored by 15 national associations of researchers, teachers and education offi cers, including the American Evaluation Association, produced Program Evaluation Standards setting out ‘how to assess evaluations of educational programs’ (Sanders 1994) and these were updated and re-titled, Guiding Principles for Evaluators, in 2004 (see below).

In the UK, a large number of projects were funded in higher education within the Teaching and Learning Technology Programme, and many of these included funding for an evaluation. Data for section two of this article is partly drawn from my own work as evaluator of a project funded to develop courseware to teach accountancy in Phase One of TLTP (Somekh 2001). A typical example of a more recent TLTP evaluation is given by Hall and Harding (2000), who report on a project implementing the use of computer-based teaching in a university History Department. The shift from the development of materials to in-house support for implementing their use is partly the result of the recommendations of the 1996 evaluation of TLTP by Coopers and Lybrand et al. (op. cit.) which found that uptake of TLTP resources had been disappointingly low. A year later, the report on IT and Teaching and Learning in Higher Education (ITATL), commissioned by the Dearing Enquiry into Higher Education, was inconclusive in its ﬁ ndings, mainly because the sponsors demanded

a large element of cost–beneﬁ t analysis for which the data were not available from universities (Boucher et al. 1997). Within the school system in the UK, strong political interest in Success Maker and other Integrated Learning Systems (ILS), as ‘solutions’ to under-achievement, led to a series of three major evaluation studies (NCET 1994a, 1996; Wood 1998). The conclusions of these studies discouraged any major national investment in ILS. Other evaluations during the 1990s also had

a signiﬁ cant impact on policy. The Superhighways Initiative (EDSI), an innovative initiative to encourage commercial sponsorship for networked technology projects,

The role of evaluation 139 incorporated a government-funded evaluation involving ﬁ ve teams of evaluators and

a synopter (Scrimshaw 1997; Somekh et al. 1999). The EDSI fi ndings led directly into a major programme to establish the National Grid for Learning and provide system-wide training for teachers in ICT. The evaluation of the Multimedia Portable Computers for Teachers pilot project, funded by the National Council for Educational Technology (NCET) (Harrison et al. 1997), has been important in confi rming the effi cacy of giving teachers powerful portable computers for their own use. A major national initiative to provide all teachers with this technology resulted.

The role of evaluation in ensuring excellence in CIT programmes