Research conducted under the banner of IS/IT generally tends to derive from management schools rather than computing departments, but also from departments of informatics, where that term is in use. Evaluations of IT in broad ways have also been carried out by sociologists (eg Webster, 1990) and people from other social sciences. While this work has often been influenced by the programme evaluation literature (see section 2.3), it also often reproduces similar results and conclusions independently. Once again, there is a considerable body of work, which I shall only summarise in the briefest of ways here.
Farbey (1995:207-8) writes that the purposes of IT evaluation are typically: as a basis for decision-making, control or accountability; legitimisation of a decision already taken, for example on strategic grounds; to gain and retain committment from stakeholders; as a learning process for the organisation and its members; and as a starting point for negotiation and collective decision-making.
A familiar strain from this area is the need to consider the organisational context of use, and to take into account many perspectives - calls which are also heard elsewhere in this report. Neils Bj¿rn-Andersen writes: "much of our Information Systems research has failed to consider the dynamic nature of our environment and the demands placed on society and individuals" (1984:1). Similarly, Peter Checkland argues that "uniformity of perspective cannot be imposed upon autonomous human beings" (1984:17).
By contrast, Targett (1995:203) writes that "evaluation has too narrow an interpretation in many organisations, being taken to refer to the quantification of well-defined benefits with the sole objective of gettting the project over organisational fences". Likewise, Blackler and Brown (1985:1) write "whilst social and organisational factors may be crucial for the successful implementation of [IT], most evaluation models fail to emphasise or even to include such factors" while Farbey et al (1994:44) write that "in practice few managers feel comfortable with anything other than Return on Investment (ROI) calculations, supplemented by a verbal description of the 'soft' or 'intangible' benefits".
Moving towards appropriate methodologies to use, Symons and Walsham (1988:120) write that "real management design or decisions are generally unstructured or 'wicked', and formulating them is itself a problem. The most suitable systems of inquiry are Kantian (multiple models), Hegelian (dialectical, conflicting models and Singer-Churchmanian (learning systems); these have been almost totally neglected in favour of the Lockean (data based) and Leibnitzian (model based) systems beloved of 'hard' operational research and management science". They talk of an analysis where they wish to use a combination of four conceptuals models: formal-rational (economic), structural (considering organisations as rule systems), interactionist (negotiations between groups) and political (conflicting needs of groups).
Serafeimidis and Smithson (1994:135) argue that IT applications have gone through three phases, each with associated evaluation methods: automation (to the end of the 1960s), where the evaluation concerns were purely cost-based; information (1970s and early 80s), where predetermined outcomes were considered as the main focus, such as in the use of cost-benefit analysis; and transformation (1985 onwards), where evaluation is based on the intangible benefits from IT. Drawing on various authors, they present three layers of evaluation with associated methods and questions: content, where evaluation goals are considered with metrics and cost questions but also intangible results; process, which considers the way of performing the evaluation, particularly as a learning activity; and context, which considers issues such as stakeholder involvement. They quote Gilbert et al (1988) with the following list of stakeholders: owners, the financial community, activity groups, customers, customer advocate groups, trade unions, employees, trade associations, competitors, suppliers, government and political groups. That this list has a somewhat different emphasis from my list in section 3.3 will be rather readily apparent.
Extending this last point on the importance of stakeholders, Jurison (1994) argues that the benefits accruing to different stakeholders (eg customers, employees, other organisational collaborators), while among the most important features of an IT system's effects, may be ignored in evaluations due to their not showing up in balance sheets. To only consider indicators such as ROI will "capture only the objectives of a narrow groups of stakeholders" (p.251).
Even where cost-benefit analysis is appropriate, argues Sassone (1987), the method used is often not justifiable. The standard economic model in IS cost-benefit evaluation is called Time Saved Times Salary (ie the use of the direct salary cost to the organisation as a measure of the value). Through a mathematical analysis, Sassone says this is inadequate in two respects: it assumes "a person's value is equal to his or her cost to the organisation" and it "takes no account of how saved time is used" (p.280). He presents an alternative economic model "wherein the value of a good or service is represented in terms of the value of the salient characteristics of that good or service" (p.280).
Walsham (1993) has written extensively on interpretive perspectives in IS design and evaluation. He argues that IS evaluation should consider the issues of content, social context and social process. Under the first heading, important questions include the purpose for which the evaluation is being carried out and what factors are included in it. Concerning social context, he argues for the consideration of who are the stakeholders in the situation, what their needs are and how to resolve conflict between those needs. Finally, he suggests that IS evaluation be seen as a multi-stage process occurring at several points, in different ways, during the product life-cycle. It is important he says, to consider evaluation as a learning process for all involved - "questioning is acceptable, all assessments are legitimate in the evaluation discourse, everybody is a learner during the evaluation process, and moral issues can be debated" (ibid., p.185).
An interesting design framework from IS that is worth mentioning here for its relevant ideas to evaluation is Multiview (Avison and Wood-Harper, 1990). The authors argue for an approach to design that is contingent within the framework, rather than contingent between frameworkers, and say that "the techniques and tools of Multiview are chosen and adjusted according to the problem situation" (p.13). This stands in stark contrast to traditional design methodologies such as SSADM, where "the steps are prescribed in great detail and are expected to be followed rigorously in all situations" (p.13). This contingent approach to design is of great interest, and is substantially based on the notion of multiple stakeholder perspectives that need to be considered (hence the name). It partly derives from the interpretivist Soft Systems Methodology of Peter Checkland (Checkland and Scholes, 1990), where a socio-technical 'system' is recognised as a construct of the analyst and not an objective entity, which means that the Weltanschauungen (world-views) of the participants and the analyst need to be taken into account in considering the system.
Webster (1990:124) gives a lengthy study of word-processor operators. Her conclusions are interesting: "the wider technological, organisational and managerial context within which these various organisations operate ... have had an important influence in maintaining continuities in working patterns in automated offices ... divisions of labour themselves arise from previous divisions and working practices, which in their turn arise from managerial and political processes within organisations". That is, past practice cannot be ignored in evaluating the effect of computer systems: the slate can never be cleaned off (contrary to the aims of business process reengineering).
Finally, and leading on to the next section, there was a large evaluation (Guy et al, 1991) carried out by the universities of Sussex and Manchester of the Alvey programme for IT research & development, sponsored by the British government between 1983 and 1988, and involving over three hundred projects from academia and industry (mostly collaborative). The report begins with lessons for four stakeholder groups: industrial participants, academic participants, programme administrators and policy makers. It is constructed as a summative evaluation, but is based on a series of formative evaluations that provided feedback throughout the project to the Alvey directorate, Alvey project members and the public through meetings, presentations and articles. A series of different methods were used, during the lifetime of the project - surveys, interviews, technical reviews and databases of financial and performance indicators. Some individual evaluation exercises also used direct observation and techniques such as network analysis. The main evaluation issues were efficiency of implementation, impact & effectiveness, and appropriateness.
While the Alvey evaluation, being written for central government, is rather based in financial indicators (of the cost-benefit kind), the heterogeneity of methods used in it are interesting. It is also one of the largest-scale IT evaluations conducted, running as it did for five years alongside the programme itself.
Go to the Evaluation of Cooperative Systems home page