Some Experiences of Applying Ethnography and Discourse Analysis in Real Software Development

Helen Sharp*, Mark Woodman**, Fiona Hovenden**, Hugh Robinson**

*Centre for HCI Design, City University, Northampton Square, London EC1V 0HB,
**Computing Department, The Open University, Walton Hall, Milton Keynes MK7 6AA

We have been using a combined approach of ethnography and discourse analysis to study ‘non-technical’ factors affecting software quality management systems (SQMSs) through a project called SoFEA (Hovenden et al., 1994). This involved us in visiting five companies of different sizes to investigate stories about their SQMSs. Three of us are software engineers, the fourth is a social scientist. In this paper, we summarise the project, and describe some expectations and experiences of software engineers, as researchers and as subjects, in applying these social science techniques in the context of real software development.

Our approach: combining ethnography and discourse analysis

One of the criticisms of ‘classical’ ethnography is that it fails to feed back into practice (Hammersley, 1992). To address this, others have developed modifications to the basic ethnographic approach (for example Ball & Ormerod (1999), Hotlzblatt & Beyer (1993) and Meyer (1992)).

In our project, we used ethnography to provide a framework for studying the culture of a setting, and for uncovering the knowledge, ideas, beliefs and values which inform activity. We used discourse analysis to focus on the structures of text to explore people’s interactions and interpretations, and uncover the ways in which knowledge, ideas, beliefs and values are transmitted, discussed, developed and perpetuated. Combining ethnography and discourse analysis in one approach means that you do not divorce the texts from their cultural setting, but use the cultural setting to provide enhanced insight and understanding.

The SoFEA project

We began the project in 1994 because of our concerns that technical innovations did not appear to be producing better quality software systems. Our particular focus was on the effectiveness of quality initiatives such as accreditation to ISO9000, and the social factors affecting their success, their adoption and evolution.

Initial contact with our collaborators was made through a software quality management conference which resulted in a series of visits to different companies and one one-week long study. In the end, our interlocutors were one person from each of three companies, three people in one company, and all members of a department involved in software development in another company. The two main techniques used were informal interview and participative observation based on ethnographic theory and practice. We used discourse analysis to study the transcripts of the interviews as well as the ethnographic observations. The interviews were semi-structured insofar as we had a list of ‘core’ topics that we wished interlocutors to cover, but we made the tone as informal and conversational as possible and did not adhere to a pre-determined order.

The data collected consists of audio and video recordings, items from noticeboards, technical documentation, marketing brochures, observations of physical environment and employees’ habits, etc.

Results from the project fall into two categories: the factors we found to be influencing the effectiveness of SQMSs, and observations on the use of the chosen techniques, from ourselves as software engineers trying to apply them, and from our collaborators as software engineers being ‘subjected’ to them. We do not elaborate on the former in this paper. Influences we observed included organisational factors, such as the nature of the company’s business, the customer base, market pressures, and individual factors such as a strong community culture in ‘Quality’, maverick developers, pressures on individuals. More detailed results on this aspect of the project are reported elsewhere (e.g. Hovenden et al, 1998; Sharp et al., 1998).

Expectations and experience from a software engineer’s perspective

We used discourse analysis to uncover some of the tacit ways in which messages are communicated. This can reveal beliefs and values within a culture. However, the speaker is also communicating facts, and as software engineers we are far more used to collecting, analysing, interpreting and using facts. At times, it was difficult to ‘see the wood for the trees’ and know how to deal with the facts, and how to deal with the discourse.

We were concerned to allow our collaborators to ‘speak for themselves’, i.e. we did not ask pre-determined questions aimed at eliciting specific kinds of answer. Our collaborators, particularly, were surprised by this.

Throughout the study, and subsequent analysis, we maintained a non-judgmental position, and treated everything as ‘strange’. From our experience we believe that this approach allowed our collaborators to express their own viewpoints more fully, and allowed us to gain a better inside view of the cultures and influences at play. It is important to maintain this stance, and to treat the collaborator as the expert. Maintaining this perspective, from a researcher’s point of view, is not easy.

A related aspect is the lack of hypothesis to test. It is impossible to suspend entirely who you are, your training, your experiences and your expectations. Pragmatically, the best that can be done is to be aware of them and to question how you interpret what you see and hear. Having no hypothesis means that it is easy to feel uncertain of what you are looking for. Of course, you are looking for everything; all comments, events, glances, etc. are significant.

Our collaborators had expected us to be judgmental and comparative. They wanted us to be experts, consultants.They regarded us as ‘one of them’ because we shared a common background, being software engineers with knowledge of quality management issues. This allowed us to enter into informed discussion, but our observations needed to remain impartial, and our ability to enter into such discussion may have led them into thinking that we would provide different feedback.

Conclusion

We have been applying social science methods (ethnography and discourse analysis) in real software development environments to uncover how quality procedures are actually applied by practitioners. In this role, we have found the techniques to be a powerful tool. The main improvement we would suggest is that the non-judgmental approach be maintained only until the analysis of data is complete. After this, the specialist expertise of the researchers, e.g. in process improvement or software engineering, should be brought to bear to provide the kind of comparative feedback our collaborators would have appreciated.

We have found that software engineers, both as researchers and as collaborators, have some uncertainties about these methods, and hence are hesitant to embrace them as useful tools. Exposing these uncertainties and discussing ways to overcome them will, we hope, lead to their acceptance by the software community.

References