SESL - Systemic Evaluation for Stakeholder Learning
SESL is a methodology for evaluating cooperative systems. I have elsewhere (Ramage, 1996a) defined a cooperative system as:
a combination of technology, people and organisations that facilitates the communication and coordination necessary for a group to effectively work together in the pursuit of a shared goal, and to achieve gain for all its members.
As a prologue, I would emphasise
the point I made in my paper on evaluation as learning (Ramage, 1996b),
that all stages of this methodology are intended to be seen as
part of an ongoing learning process. I hope it is not the case
that a user of this framework will see themselves as an outside
observer prodding and poking and studying for a while, and then
issuing a scientific and dispassionate evaluation document at
the end of their study; but rather as a participant in the system
themselves and as a part of an ongoing organisational learning
process.
The following diagram summarises
the five stages of the methodology. I must stress that this is
not intended to be overly prescriptive of the order in which things
are done. It is my view that the first three items need to be
performed to some degree before the study/analysis cycle can be
commenced; but this may be in a fairly sketchy way, with the details
filled in later. And, of course (as the dotted arrows indicate),
it is possible to go 'backwards' at any stage.
A. Determining the nature of "the system"
As SESL is a methodology for evaluating
cooperative systems, it is vital to decide what system one is
evaluating. [It must be noted here that a "system" is
a constructed thing - in the eye of the beholder - and so one
person's view of what is the system will differ from another's.]
I would in general suggest that in a CSCW setting the main focus
of interest will be socio-technical. We know from lots of studies
that how a groupware program is used depends on the context in
which it is used, the organisational and cultural situation and
so on. So to study a CSCW system only as technology seems to be
folly.
However, one could in theory do that;
and likewise, one could in theory study only the "social
system" - the context within which the technology is embedded,
without reference to the technology. This is also foolish, as
it ignores the fact that the technology changes the organisational
structure and culture (and not least its processes) - and this
change is one of the selling points of the technology. However,
it is sometimes interesting to make the focus of one's
study primarily the social or the technical system (remembering
that this is simply a statement about what one is studying, not
about the inherent nature of the system). This is what I mean
by determining what is "the system under study".
Checkland and Scholes (1990:31) give
some helpful advice on how to be clear what system one is studying,
with the caveats that it is very important one is clear that properly
one will end up with many different systems to study, and that
the system(s) chosen are simply one way of viewing the situation.
They suggest that two basic kinds of system can be identified:
the primary-task system: "some organised purposeful
action which could be reflected in the choice of a notional human
activity system whose boundary would coincide with the real-world
manifestation" (that is, the boundaries of an organisation);
and the issue-based relevant system, which considers the
working out of a particular conceptual issue (e.g. conflicts over
resource allocation) as its boundaries. To assist in formulating
these systems, they also suggest the use of metaphors to describe
the relationships between different key parties within the system
- e.g. master/slave, husband/wife, brother/sister, organism/virus
etc.
For Checkland and Scholes (p.36), these systems can be usefully described in terms of a well-structured model that they call a "root definition". An example, for someone painting a fence, might be:
A householder-owned and manned system
to paint a garden fence, by conventional hand painting, in keeping
with the overall decoration scheme of the property, in order to
enhance the visual appearance of the property.
This kind of highly-structured formulation
is presented only for illustration, not for use in this methodology.
However, I do find it helpful to reflect on what is the content
of the system that one is evaluating, and to be aware that the
obvious answers of "the program" or "the program
and the people who use it" may not be the best ones; or they
may be, as long as one is aware that one is making an explicit
choice in using those answers rather than any others.
B. Identifying the evaluation as one of the Five Types
Consider the following two questions:
1) Are you looking at the effects or objectives of a system?
2) Are you looking at the way things
are now, or their potential?
The answer to these questions leads us to a 2x2 matrix that concerns itself with the evaluation of socio-technical systems:
[NB. The Buying type of evaluation
could be argued to straddle both Potential Objectives and Potential
Effects, not least because Potential Objectives as a category
implies two levels of being-in-the-future that are a bit confusing
to think of at once!]
The four elements of this matrix,
together with one more, are those which I have identified in an
earlier chapter as the five key types of CSCW evaluation. The
fourth type, People-Focus, is not included in the matrix, because
its focus is not on the socio-technical system but rather on the
'social system' (it would fit into the Actual-Effects quadrant).
As mentioned above, such a focus may seem odd for a CSCW evaluation
- but they do happen sometimes. Other kinds of evaluation of a
social system, especially of its potential state, are usually
called management consultancy. It may be the case that considering
which one or more of the five types an evaluation fits into will
assist in determining the appropriate answers to the above questions.
Neither the four quadrants of the
matrix, nor the five types, are intended to be exclusive or complete.
No doubt there are others - this taxonomy is an open one to be
changed, presented for the provocation of thought and for heuristic
use. A further caveat is that these are ideal types (in Weber's
sense): I would in practice expect to see them appear in combination
and in modified form.
C. Identifying stakeholders and their viewpoints
The next question to ask is, who
are the stakeholders in the system and what are their interests?
A typical definitions of stakeholders is: "any identifiable
group or individual who can affect ... or is affected by the achievement
of an organisation's objectives" (Freeman and Reed, 1983).
The verb "affect" could also be replaced by "depends
on" or "influences" (again in both directions).
This gives one set of ways of looking at who are the stakeholders.
Another way is to take a list of
typical stakeholders for the relevant domain and see to what extent
they apply to a given system. Typical stakeholders in CSCW are
those listed above: those who use the software, their colleagues
and managers, the software developers and retailers, the Information
Systems department of their organisation (if appropriate), and
perhaps the customers of the organisation. Wider groups (such
as trade unions, parent companies, employers' associations, shareholders
and governments) may also be stakeholders.
A third way, that I find particularly
helpful, is to have a representative group collectively construct
a stakeholder map: a diagram showing who are the key people relating
to the system by showing the system in the centre of the map and
then the distance of others from the centre, showing their importance
to it. This diagrammatic form can help to identify groups who
might otherwise have been forgotten. The following is the stakeholder
map that was drawn up by myself and the project team during a
study I conducted of a research project. Note how proximity of
the stakeholders is used to show closeness to one another (even
to the extent of overlapping) and to the centre; and how three
groupings were ultimately identified.
Researchers on the use of stakeholders, especially Mason & Mitroff (1981:94-103) have described several more methods for identifying them, including:
Another significant question (which
is assisted by the process of stakeholder mapping) is who are
the key stakeholders: whose views need most to be met? Who will
the system break down without the involvement of? It is important
to make it clear who you think these are, as this will alter how
much weight to give to different aspects of the evaluation. It
is also useful to be clear what are the perspectives of these
key stakeholders upon the system.
It must be stressed that the lists
of stakeholders, their interests and relationships gained by these
methods are subjective: different methods (and different participants)
will produce different lists, and they will also change over time.
As Bateson (1972:449) wrote, "the map is not the territory".
For this reason, it is greatly desirable to produce them collaboratively,
preferably involving both insiders and outsiders to the organisation.
D. Analysis: key questions to ask
Now we move on to the study and analysis.
The type of study conducted (ethnography, usability lab experiment,
semi-situated study where your colleagues try out the software,
etc.) does not affect this methodology directly: use whatever
comes to hand, according to your skills, experiences and resources.
The analysis of the system will depend
upon the exact study method used and the kind of evaluation being
conducted. A lab experiment will have a very different relationship
between the study and the analysis from an ethnography. Again,
it will depend on the theoretical background of the evaluator
- ethnomethodology and distributed cognition, for example, will
provide two very different ways of conducting an ethnography and
thinking about the results. However, in general I assume here
that there is some kind of study cycle, where observation and
analysis are intertwined: one does not just observe once and then
go back to one's ivory tower to analyse, but observes and analyses
back and forth.
Here I will offer a few key questions that can usefully be at the centre of that study/analysis cycle (as shown in the diagram at the start of this chapter). To determine which to use, use the matrix in part B above as discussed there. The relationship to the five types is to answer the following questions for each type:
Remember that all except People-Focus
relate to the socio-technical system; that relates to the social
system.
1. ACTUAL EFFECTS.
What are the system's effects upon...
a) the work of the group using the system?
b) the life of the group?
c) the life of the people in the group?
d) the life and work of the people outside the group?
e) the organisation(s) of which the group is a part?
f) society?
(Which of these questions will be
held to be most important depends on the key stakeholders identified
in part C. However, I suggest that all of the categories need
to be considered to some extent.)
2. ACTUAL OBJECTIVES:
What are the system's objectives
(from the different perspectives of the various stakeholders)?
To what extent are these being met?
3. POTENTIAL EFFECTS:
What are the potential effects of the system upon:
a) the work of the group using the system?
b) the life of the group?
c) the life of the people in the group?
d) the life and work of the people outside the group?
e) the organisation(s) of which the group is a part?
f) society?
(The same caveat as in question 1
applies here.)
4. POTENTIAL OBJECTIVES:
What are the objective for this new
system objectives (from the different perspectives of the various
stakeholders)? How well are they likely to be met?
E. Results
Communication is vital in evaluation:
unless one tells people what one has determined in the evaluation
it's not much use to anyone. So that is the first part of the
feedback process - telling the various relevant stakeholders what
has been determined in the evaluation.
The next part is assisting them in
decision-making. It's not the evaluator's job to make decisions
about what should be done with the system, nor is it their job
to decide who should make decisions. However, it is their
job to give sufficient information to all those who might make
decisions (which is not necessarily the same as those obviously
in power), that they can do so.
Finally, perhaps the most important
role for an evaluator is to act as a facilitator for organisational
learning. This may be implicit in their attitudes, or it may be
explicit, and involve taking a view that evaluation is not so
much about making judgements as about assisting in the learning
processes of all those involved in a project (I have written elsewhere
on this issue). Think of an evaluation, not as a process of decision-making,
but as a process of greater understanding.
References
Mason, Robert and Mitroff, Ian. "Challenging Strategic Planning Assumptions", Chichester: Wiley, 1981.
Checkland, Peter and Scholes, Jim. "Soft Systems Methodology in Action" Chichester: Wiley, 1990.
Bateson, Gregory. Form, Substance and Difference. In Steps to an Ecology of Mind, Chandler, 1972, pp.448-464.
Freeman, R. Edward and David Reed (1983). Stockholders and Stakeholders:
A New Perspective on Corporate Governance. California Management
Review, 25 (3): 88-106.
Cooperative Systems Engineering Group
| Computing Department
| Lancaster University
Magnus Ramage
22 February 1997