Of course, the use of multiple methods in support of evaluation or design is by no means uncommon. A recent example is an evaluation conducted by Mark Ackerman (1994) of his own system, which used "multiple data collection procedures including an initial questionnaire, usage data at the 'mouse-stroke' level, critical incident interviews, final questionnaires, and field observation and interviews ... a concerted effort was made to use the qualitative and quantitative data together" (p.246-7). Similarly, researchers have often modified difficult methods for design, producing notions such as "quick and dirty ethnography" (Hughes et al, 1994).
The striking thing about the CSCW evaluation literature discussed above is that most designers and evaluators use more than one method in combination: even those who seem the most relentlessly devoted to laboratory methods will also use heuristic evaluation by the designers during the building of the system.
The point, however, is that this is not made clear in the literature: mixing methods, in either of the ways Patton describes above (i.e. running parallel methods at the same time or mixing parts from different methods), is regarded as unscientific and an inappropriate breaching of one's own discipline. Curiously, although inter-disciplinary and multi-disciplinary projects are common, individuals within those projects often feel the need to maintain clear boundaries between themselves and others in terms of disciplines - individuals doing both ethnography and programming are few, for example (although there are exceptions).
This seems to be part of the modernist imperative to split up everything into categories (Latour, 1993), seen elsewhere as the Taylorist drive towards division of labour. To reject it and use multiple methods leads to three possibilities: a postmodernist value relativism, where all methods are equal and none can be judged to have primacy over another; an all-embracing meta-method, a grand theory of the worst kind that would encompass all phenomena and explain none of them (Mills, 1959); or a rejection of the notion of the disciplinary divide and the embracing of multiple methods within a single approach. This last is my preferred solution.
My basic suggestion here is that of the theoretical jackdaw: snapping up glittering ideas from wherever they occur, without any particular concern about their origins or that they fit together at a deep level, just that they fit the situation. I would thus encourage the ad-hoc over the established, the situational over the pre-formed category. This has been the cry of ethnomethodology for a long time and suffers from the flaw of that framework: it ignores the fact that researchers go into situations with a lot of pre-formed theoretical ideas and that it is simply not possible to ignore them. Well so be it, but it is at least worth trying to take other ideas into account as explanations of what is happening.
As an example of this kind of jackdaw approach, consider the theoretical basis of my analysis of ShrEdit in use (Ross et al, in press). My basic consideration was shared understanding (or intersubjectivity), which has arisen in different forms in many different forms of social science but which for me came out of ethnomethodology. However, to this was added the analysis of breakdowns (which comes from the philosophy of Heidegger via Winograd and Flores) and various use of the idea of the propagation of represenations from distributed cognition; as well, of course of folkloric CSCW concepts such as surreptitious monitoring.
Once again, I claim no special feature to my work in using all these different areas - this is done by lots of researchers in their description of situations - but I would argue that this is too often seen as something of shame, to be hidden or not thought about, whereas to me it is the very basis for the full understanding of situations of study.
Our way of doing this was to use a redesign session to find out the kind of system (and particularly the interface) preferrable to the participants, but also to elicit more directly the critical issues that occurred to the users as problems with the ShrEdit system. Thus the result is not so much a redesigned interface (as might be the case in a standard participatory design session) as a set of issues that integrate into the evaluation.
The advantages of involving users of systems in the evaluation process have been stressed by many of the works under discussion in this report. We do, it should be said, take a rather simplistic notion of 'users' here, taking just this one group: as the study was conducted of a pre-formed program in a semi-situated environment, the full panoply of stakeholders discussed above did not really come into play. In a situated evaluation study, or even in a somewhat wider semi-situated study, the role of the different kinds of intererested parties such as managers and workers' representatives need also to be considered.
Go to the Evaluation of Cooperative Systems home page