A Testing Approach for Program Families

These days a popular - because cost efficient - approach to software development is 
the deployment of, so called, program families or product lines. A program family 
signifies that over the years from a successful program not only improved or adapted 
maintenance versions, but also variants for other platforms and application areas are 
developed. The main characteristic of a program family is that its members have a 
great number of properties in common. So object-oriented framework technology is 
especially suited for the implementation of program families.

In practice, the efficient testing of a program family member remains a problem. 
Commercial test tools which promise to accomplish this task, are mostly of minor use 
for testing them. Often based on some capture/replay facility which manipulates the 
user interface, they are very sensitive to changes in the user interface. Therefore in 
practice attempts to use them across different family members usually fail. As a 
consequence of this, new tests for every member have to be developed, often leading 
in practice to inadequately tested software. Even worse is the fact that this 
specification-based, also called black-box, strategy tests the program only 
superficially not considering its internal implementation. 

So the aim of my work is to develop a more efficient testing process for the members 
of a program family. The main model for my approach are test benches that are used 
in the classical engineering disciplines, e.g. when a new engine is developed, 
engineers develop in parallel the necessary test bench for trying it. The principle idea 
to transfer this approach to object-oriented software development is to equip the 
frameworks with infrastructure, for example interfaces to a separate test environment, 
to improve test execution and automation. The main problem with this approach is the 
lack of approved testing methods for object-oriented software. On the one hand there 
are the classic structural testing techniques which have their roots in imperative 
programming. But based on the sequential execution of imperative programs they are 
mostly inadequate for testing the random sequences of method calls in object-oriented 
software. On the other hand, there are various techniques, mostly coming from 
abstract data type theory, for testing individual classes for correctness. While they are 
useful for this purpose, they do not permit to test the interaction of collaborating 
classes. In general there are no approved ways for testing the collaboration of two or 
more classes.

Currently I am investigating a way for testing, if classes, that collaborate in a known 
scheme, are correctly implemented by developing test cases from their intended 
behaviour. For example considering the MVC pattern, there are test cases to prove 
that the participating classes for model, view and controller implement the expected 
MVC behaviour. An example is a state change to a model object which requires 
corresponding updates to its observing view objects. Such a change to the model in 
combination with checks for the expected updates depicts one test case. A product of 
this approach would be the inclusion of built-in tests into the MVC-framework which 
allows to test automatically the correct implementation and usage of such 
collaborations. This means on one hand the integration of dedicated classes which 
manage and execute the predesigned test cases, on the other hand the classes under 
test need also to be augemented by means, primarily additional inspection methods, 
for checking their responses.