Combinatorial Testing of Dynamically Adaptive Software Systems
(2026)
Files
Thesis_MARTOU.pdf
Open Access - Adobe PDF
- 1.84 MB
Details
- Authors
- Supervisors
- Abstract
- Dynamically adaptive software systems can change their behaviour at runtime, resulting in large configuration spaces. Features and constraints provide a convenient representation of such variability, but the number of possible configurations grows exponentially with the number of features, making exhaustive testing impractical. This thesis investigates how system-level testing can be used to efficiently find and localise faults in dynamically adaptive software systems. To address faults that arise not only from feature interactions within configurations but also from transitions between configurations at runtime, we propose Combinatorial Transition Testing (CTT). It provides a comprehensive testing process that covers the generation of test suites, the automatic detection of transition faults, and their subsequent localisation. To generate test suites efficiently, we propose an optimised greedy row-by-row generation algorithm. Building upon CTT test generation, we introduce a differential test oracle capable of detecting transition faults without prior knowledge of the system's expected behaviour. We further propose a fault localisation approach that combines differential testing, divide-and-conquer, and SMT solving to isolate faulty transitions from a large set of suspects. Our empirical results show that the size and cost of CTT-generated test suites are linearly correlated with those generated by Combinatorial Interaction Testing (CIT), as is the additional effort required to use the test oracle. Together, these results provide empirical evidence that both test generation and test oracle effort grow logarithmically with the number of features. For fault localisation, the experimental results indicate approximately linear growth in both the number of executed configurations and SMT solver calls on the evaluated feature models, despite the exponential growth of the configuration and suspect spaces. Finally, the proposed approach is illustrated and evaluated on three dynamically adaptive software systems: a Smart Messaging Application, a Risk Information System, and a Medical Appointment Manager. These artefacts provide variability models, representative faults, and supporting code fragments for evaluation and reproducibility. In addition, a large collection of feature models from the S.P.L.O.T. repository is used for experimental validation. All data and source code are freely available as an open-source research tool to support the reproduction and further development of CTT.
- Affiliations
Citations
Martou, P. (2026). Combinatorial Testing of Dynamically Adaptive Software Systems. https://hdl.handle.net/2078.5/281452