First, there is the problem of Method in general. A lot of people have spent a lot of time developing methods that are designed to be the One Best Way (cf. Taylorism) to evaluate or design computer systems. This particularly affects the software engineering methods which are designed to be followed to the letter, regardless of the situation. The problems with this approach in other disciplines have been noted by several authors in philosophy and sociology (Feyerberand, 1975; Gadamer, 1975; Morgan, 1983): any one method, based in a particularly disciplinary background, can only consider a certain part of a particular situation. As CSCW systems involve an effect on such a wide range of people and organisations in various ways (see the next section on criteria for some of these), this is to leave out a considerable part of the interesting things about the situation. Critiques of the rationalist approaches on this score are common in CSCW (e.g. Suchman, 1987), but the same charge can be levelled at any approach. The solution advocated below is the combination of methods.
The other main problem with many CSCW and related evaluations is a naivety about who are relevant stakeholders in the determination of the appropriateness of a computer system. As mentioned above concerning information systems evaluation, too often the only relevant stakeholders are held to be senior management and computer programmers, so that the only criteria under consideration are financial or technical: does the system run without crashing and fast enough, and is it sufficiently cheap? To ignore the question of individual and group effects is to court disaster, as Grudin (1988) long ago pointed out: both from the point of view of those stakeholders whose needs are not considered, and for the organisation which will end up with an inadequate usage of its system. A solution to this problem is the participation of stakeholders at all levels in the evaluation process.
Go to the Evaluation of Cooperative Systems home page