In some senses, I view this list as a hierarchy: as the list goes on, the issues become more complex and affect larger numbers of people. There is also a gradual widening of the scope of the evaluation as the focus moves from the internals of the computer through its interface and to effects on larger and larger groups of people (cf. Grudin, 1990). This distinguishes CSCW evaluation from HCI evaluation, which typically considers only the first three points, although it is not necessary to do this there: single-user systems have individual, organisational and social implications too (e.g. Webster, 1990).
One final point on criteria is whether these are specified in advance, or emerge once the system has been built/implemented. These will partly depend on which stakeholders have power within the evaluation: managers will tend to privilege organisational effects, trade unionists will concentrate on workforce effects, and so on - see the next section for further discussion.
It is also a methodological question. An ethnomethodologist looking at the above list would probably be inclined to vilify it as privileging certain categories (the ones mentioned) above others. I would thus be accused of imposing my analysis of the situation over that of participants. This might be a valid point if the list were to be used exclusively, as the only source of evaluation criteria. However, I would suggest that the use of participatory evaluation methods will avoid this to some extent, by feeding in participants' concerns and issues to the evaluation.
This question has been addressed under the title of 'goal-free evaluation' in the evaluation literature. Michael Scriven, who coined the term, writes: "the evaluator is not told the purpose of the programme but enters into the evaluation with the purpose of finding out what the programme is actually doing without being cued as to what it is trying to do" (1981, p.68).
Go to the Evaluation of Cooperative Systems home page