1. Introduction

Much work has now been done on the design of CSCW systems, and particularly the integration of methods that incorporate social knowledge into the design process (Hughes et al, 1994). However, considerably less work has been done on the evaluation of CSCW. Such authors as have considered this acknowledge the problem to be a difficult one (Grudin, 1988; Ross et al, in press). The problems are partly to do with methodology - are the methods created for the evaluation of single-user systems likely to work with multi-user systems and if not, how can they be modified; and partly to do with goals - are we aiming to evaluate the usability of the systems, as in Human Computer Interaction (HCI), or do other factors become relevant?

It is my contention that existing methods are mostly inadequate, and that we need to consider new ways of evaluating CSCW, to take into account issues of individual, group and organisational effects as well as questions of usability. The aim of my PhD is to explore the space of CSCW evaluation, and to produce some methods that go some of the way to solving these problems.

A semantic note - the word "system" is used very loosely. The working title of this project is "Evaluation of cooperative systems", and this is taken to imply that evaluation is not just of a computer system but of the social system within which it is embedded. Thus the work is inherently socio-technical.

1.1. What Is Evaluation?

A good starting point might be to ask exactly what is evaluation. I take as my starting definition that of Elliot Stern (quoted in Sommerlad, 1992):

Evaluation is any activity that throughout the planning and delivery of innovative programmes enables those involved to learn and make judgements about the starting assumptions, implementation processes and outcomes of the innovation concerned.

Stern is concerned with the evaluation of educational, social and organisational programmes rather than of computer systems. However, the extent to which computer systems are embedded in, and shaped by, the social systems within which they are situated (cf. the work on "socio-technical systems" at the Tavistock Institute) means that such an approach will be of considerable utility to this sort of evaluation. To put it more bluntly: evaluation is no good if it just considers the computer. The situation is also up for evaluation.

The traditional divide of evaluation into formative (contributing to the redesign of the system) and summative (considering the system for purchase) seems not particularly useful in this light. Evaluation can occur at the following points:

while a system is being built
once a system is built, but before it is installed in any organisation
while a system is being installed in a 'test' organisation
once a system has been running in a 'test' organisation for some time
while a system is being installed in a wider setting
once a system has been running for some time

All of these can contribute to the redesign of the system; all can contribute to decisions by IT managers etc. as to whether the system is good enough to be purchased, by general managers as to what effect the system is having on their organisation, by trade union officials as to what effect the system is having on their workers, and so on. The differences in the utility of the evaluation for such purposes will depend partly on the used and partly on the information provided to the different stakeholders.

1.2. Structure Of This Report

This report begins by examining such work as has been done within CSCW on evaluation, both methodologically and in terms of case studies. This is then broadened out to consider pertinent work from other areas relating to the evaluation of computer systems, notably the usability testing of single-user systems in HCI, and the more organisational perspectives in the body of work known as information systems. A broader perspective still is taken with an all-too-brief summary of the work in the evaluation of social and educational programmes, which has considerable pertinence for CSCW.

These sections constitute a review of the existing literature, and lead into a discussion of the inadequacies of these methods, the kinds of relevant criteria for and stakeholders in CSCW evaluation, and thence to a description of one solution to the problems, the PETRA framework I have developed with Susi Ross. Finally, a plan for further work is presented.

There are also two appendices to this report, presenting issues that are of interest to me and on which I have conducted work during this year, but which are not in tune with the general tone of the paper: the emancipatory role of the fieldworker, and organisational memory. Finally, the two papers I have written with others during the year are also appended. The first of these, discussing PETRA, due to appear in Interacting with Computers, is extensively discussed in this document elsewhere. The second, "What are workplaces studies for?" (Lydia Plowman, Yvonne Rogers and myself, published at ECSCW 95), is not discussed here further, although some of the reading carried out for it is summarised in the section on CSCW evaluation.

Next Chapter (2) / Contents / References

Go to the Evaluation of Cooperative Systems home page

Cooperative Systems Engineering Group | Computing Department | Lancaster University

Magnus Ramage 10 October 1995