Ethnography and Software Testing


"Program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence."
Dijkstra's famous quote from his 1972 Turing Award lecture.


Investigators

Professor Ian Sommerville

Dr Mark Rouncefield

Dr David Martin

Dr John Rooksby

Jo Mackie


Focus

This project addresses customer focused testing. Customer focused testing is usually based around the notion of test scenarios. These scenarios represent typical situations of use of the software, and from these a set of software tests can be derived. However, the use of test scenarios as a systematic approach to testing is still immature. There is, of course, a body of work on scenario-based testing, but most of this focuses on deriving tests from scenarios rather than the development of these scenarios.


Research Objectives

The research has the following three objectives:
  • To investigate how ethnographic analyses may be used to systematically derive customer focused test scenarios.
  • To develop test scenario structures that may be used for the systematic generation of customer focused tests.
  • To investigate how the knowledge embedded in test scenarios may be presented to software testers and reused in other systems form the same application domain.

    • Research Activities

    The approach that we will use to drive this research involves a number of activities:
  • Field studies in real settings where we will collect information that will be the basis of deriving test scenarios
  • The re-analysis of existing patterns of interaction and vignettes by a software engineer to assess what is missing from them in terms of test scenarios.
  • The development of guidelines for test case derivation from scenarios and test scenario guidelines that identify the information required from system testing and that can be used as a guide by ethnographers involved in field studies.
  • The extension of our existing patterns collection with test scenarios and further patterns.
  • Extensive evaluation using real software systems that are being developed or deployed in actual working environments.

    • Relevance

    The work is particularly relevant to two areas of software testing: Testing by developers of software to assess how effective the software is in meeting the needs of customers of the software; and testing by software procurers to assess the suitability of some system for their needs.


    Workshop

    In March 2006 we held a workshop on "Ethnographies of Code". This workshop focused on the wider issues raised by our work on Ethnography and Software Testing. It featured twenty six papers from researchers and professionals working in a variety of disciplines, and attracted wide international participation.


    Publications


    Martin D, Rooksby J, Rouncefield M, Sommerville I (2007) ‘Good’ Organisational Reasons for ‘Bad’ Software Testing: An Ethnographic Study of Testing in a Small Software Company. Accepted for ICSE (International Conference on Software Engineering) 2007.

    Mrtin D, Rooksby J, Rouncefield M, Sommerville I (2008) Cooperative Work in Software Testing. Proceedings of the 2008 International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), 13 May, Leipzig, Germany. 93-96.

    Martin D, Rooksby J, Rouncefield M (2007) Users as Contextual Features of Software Product Development and Testing. Proceedings of the 2007 International ACM Conference on Supporting Group Work (GROUP), 4-7 November, Sanibel Island, Florida, USA. 301-310.

    Martin D, Rooksby J, Rouncefield M, Sommerville I (2007) "Good" Organisational Reasons for "Bad" Software Testing: An Ethnographic Study of Testing in a Small Software Company. Proceedings of The 29th International Conference on Software Engineering (ICSE), 20-26 May 2007, Minneapolis MN, USA, Pages 602-611.

    David Martin, Mark Hartswood, Roger Slack, Alex Voss: Achieving Dependability in the Configuration, Integration and Testing of Healthcare Technologies. Computer Supported Cooperative Work 15(5-6): 467-499 (2006)

    John Rooksby (2008) Work Practices and the "Economics and Psychology" of Software Testing. Paper Presented at the 2008 Psychology of Programming Interest Group Work in Progress Workshop (PPIG-WIP) 21-22 Feb 2008, University of Sussex, UK.

    John Rooksby (2007) Diagnostic Work in Computer Programming. Paper Presented at the Workshop on CSCW, Technology and Diagnostic Work. 26th Sept 2007. Limerick, IE.

    Rooksby J, Martin D, Rouncefield M (2006) Reading as a Part of Computer Programming. Proceedings of the 18th Annual Workshop of the Psychology of Programming Interest Group (PPIG2006), September 7-8th 2006, Brighton UK. 198-212.

    Martin D, Rooksby J (2006) Knowledge and Reasoning About Code in a Large Code Base. TeamEthno-online issue 2, June 2006, 3-12

    John Rooksby (2006) Debugging in Software Development. Paper Presented at Ethnographies of Technoscience, 2nd June 2006, Lancaster University, UK.

    Martin D, Rooksby J (2006) Knowledge and Reasoning About Code in a Large Code Base. TeamEthno-online issue 2, June 2006, 3-12

    Rooksby J, Martin D, Rouncefield M (2006) Reading as a Part of Computer Programming. Proceedings of the 18th Annual Workshop of the Psychology of Programming Interest Group (PPIG2006), September 7-8th 2006, Brighton UK.

    Links

    Computing Department

    Lancaster University

    Department Publications

    Chameleon Project

    Dependability IRC

    PoInter Project

    Ethnographies of Code




    Lancaster University. Funded by EPSRC.