Thus the first issue we must consider is that of paradigm. In general terms, evaluators tend to fall either into the camp labelled by Burrell and Morgan (1979) as "functionalist", or into the "interpretive" camp. That is, most evaluators are not in fact bomb-throwing radicals out to break the system but rather tinkerers who want to see if the system is working out the way it ought (whatever that means) and to suggest how it might work better. The division comes in how to determine whether or not it is working: either one uses the methods of laboratory science, with controlled experiments to determine quantitative data on which one performs statistical tests that reveal The Truth; or one uses the methods of ethnography, performing interviews, hanging around and using open-ended questionnaires to try to understand the situation and draw conclusions about it. That this relates closely to the division in CSCW between (broadly) psychologists and ethnographers will be clear - both come from the same theoretical debate.
The definition of "the way the programme ought to work" is also a subject of some debate. One can take either the perspective of those who designed the programme (ie their objectives) or are paying for it; or that of those who are working on it, using it or affected in some other way. This split is entirely separate from the above, but it tends to be the case that functionalism and objectives-based evaluation go together, while interpretivism and user-based evaluation go together.
These complicated divisions are rather often summed up under the banners of 'quantitative' and 'qualititative', which are usually referred to as methodologies rather than paradigms. That this is a gross over-simplification seems self-evident: it is entirely possible to conduct a qualitative evaluation of programme objectives, or a quantitative evaluation from the users' perspective, or an evaluation that mixes some or all of these things together. But certainly a broad world-view can be seen.
Patton (1980:28) has summarised the difference as follows, relating to the use of questionnaires: "Quantitative measures are succinct, parsimonious, and easily aggregated for analysis; quantitative data are systematic, standardised, and easily presented in a short space. By contrast, the qualitative measures are longer, more detailed, and variable in content; analysis is difficult because the responses are neither systematic nor standardised. Yet the open-ended responses permit one to understand the world as seen by the respondents."
The other distinction that is widely made within evaluation is between formative and summative evaluations. Michael Scriven coined the terms in 1967 - he described them later as: "Formative evaluation is conducted during the development or improvement of a programme or product (or person etc.). It is an evaluation which is conducted for the in-house staff of the programme and normally remains in-house. ... Summative evaluation of a programme (etc.) is conducted after completion and for the benefit of some external audience or decision maker. ... The [distinction] has been well summed up in a sentence of Bob Stake's: 'When the cook tastes the soup, that's formative; when the guests taste the soup, that's summative.'" (Scriven 1981, p.63 and p.150)
This division is often held not to be very useful: I have argued above (section 1.1) that a more useful division is one of when the evaluation occurs. Wadsworth (1991) emphasises that evaluation should be considered a part of the day-to-day working process. She identifies two types of evaluation - "open inquiry" and "audit review" (p.34), which resemble formative/summative but can happen at any time. She holds the cycle of evaluation to be Reflection-Design-Fieldwork-Analysis-Conclusions-Feedback-Planning.
Go to the Evaluation of Cooperative Systems home page