Evaluation of cooperative systems project

This page describes the project "Evaluation of cooperative systems" conducted as PhD work by Magnus Ramage, supervised by Michael Twidale and Ian Sommerville, and funded by the EPSRC and Digital Equipment Corporation under a CASE studentship. This work has now (1999) finished and the author has been awarded the degree of PhD. The completed thesis is online.

Introduction

Much work has now been done on the design of CSCW systems, and particularly the integration of methods that incorporate social knowledge into the design process (Hughes et al, 1994). However, considerably less work has been done on the evaluation of CSCW. Such authors as have considered this acknowledge the problem to be a difficult one (Grudin, 1988; Ross et al, in press). The problems are partly to do with methodology - are the methods created for the evaluation of single-user systems likely to work with multi-user systems and if not, how can they be modified; and partly to do with goals - are we aiming to evaluate the usability of the systems, as in Human Computer Interaction (HCI), or do other factors become relevant?

It is my contention that existing methods are mostly inadequate, and that we need to consider new ways of evaluating CSCW, to take into account issues of individual, group and organisational effects as well as questions of usability. The aim of my PhD is to explore the space of CSCW evaluation, and to produce some methods that go some of the way to solving these problems.

A semantic note - the word "system" is used very loosely. The working title of this project is "Evaluation of cooperative systems", and this is taken to imply that evaluation is not just of a computer system but of the social system within which it is embedded. Thus the work is inherently socio-technical.

A good starting point might be to ask exactly what is evaluation. I take as my starting definition that of Elliot Stern (quoted in Sommerlad, 1992):

Evaluation is any activity that throughout the planning and delivery of innovative programmes enables those involved to learn and make judgements about the starting assumptions, implementation processes and outcomes of the innovation concerned.
Stern is concerned with the evaluation of educational, social and organisational programmes rather than of computer systems. However, the extent to which computer systems are embedded in, and shaped by, the social systems within which they are situated (cf. the work on "socio-technical systems" at the Tavistock Institute) means that such an approach will be of considerable utility to this sort of evaluation. To put it more bluntly: evaluation is no good if it just considers the computer. The situation is also up for evaluation.

Relevant questions to this work include:
How has evaluation been done before?
When do you do evaluation?
What are the criteria for evaluation?
Who are the stakeholders in CSCW evaluation?

Much of this work is concerned with methodology. I advocate multiplicity of method, theory and perspective. My bias is towards situated and qualitative methods, as these seem to capture more readily the various stakeholders' perceptions of the situation. 

Funding

The work was funded by the Engineering and Physical Sciences Research Council (EPSRC) and the Digital Equipment Corporation, under CASE award 94563908.

Papers and publications

Papers arising from this work to date (all available on request from Magnus Ramage):

Related Documents

Other relevant Web documents (mostly by Magnus Ramage):
Cooperative Systems Engineering Group | Computing Department | Lancaster University
Magnus Ramage11 May 2000