Mutation testing research has often used the number of mutants as a surrogate measure for the true execution cost of generating and executing mutants. This poses a potential threat to the validity of the scientific findings reported in the literature. Out of 75 works surveyed in this paper, we found that 54 (72%) are vulnerable to this threat. To investigate the magnitude of the threat, we conducted an empirical evaluation using 10 real-world programs. The results reveal that: i) percentages of randomly sampled mutants differ from the true execution time, on average, by 44%, varying in difference from 19% to 91%; ii) errors arising from using the surrogate correlate with program size (ρ = 0.74) and number of mutants (ρ = 0.76), making the problem more pernicious for more realistic programs; iii) scientific findings concerning sampling strategies would have approximately 37% rank disagreement, indicating potentially dramatic impact on experiment validity. To investigate whether this threat matters in practice, we reproduced a seminal study on Selective Mutation (widely relied upon for more than two decades). The impact is stark: an inconclusive scientific finding using the surrogate is transformed to an unequivocal finding when using the true execution cost.
Fri 13 NovDisplayed time zone: (UTC) Coordinated Universal Time change
08:00 - 08:30 | |||
08:00 2mTalk | Baital: An Adaptive Weighted Sampling Approach for Improved t-wise Coverage Research Papers Eduard Baranov Université Catholique de Louvain, Belgium, Axel Legay Université Catholique de Louvain, Belgium, Kuldeep S. Meel National University of Singapore, Singapore DOI | ||
08:03 1mResearch paper | Cost Measures Matter for Mutation Testing Study Validity Research Papers Giovani Guizzo University College London, UK, Federica Sarro University College London, UK, Mark Harman University College London, UK DOI Pre-print | ||
08:05 1mTalk | Developing and Evaluating Objective Termination Criteria for Random Testing Journal First Porfirio Tramontana Department of Electrical Engineering and Information Technologies, University of Naples Federico II, Italy, Domenico Amalfitano University of Naples Federico II, Nicola Amatucci Department of Civil, Architectural and Environmental Engineering, University of Naples Federico II, Italy, Atif Memon Apple Inc., Anna Rita Fasolino Federico II University of Naples | ||
08:07 1mTalk | Efficient Binary-Level Coverage Analysis Research Papers M. Ammar Ben Khadra TU Kaiserslautern, Germany, Dominik Stoffel TU Kaiserslautern, Germany, Wolfgang Kunz TU Kaiserslautern, Germany DOI Pre-print Media Attached | ||
08:09 1mTalk | Efficiently Finding Higher-Order Mutants Research Papers Chu-Pan Wong Carnegie Mellon University, USA, Jens Meinicke Carnegie Mellon University, USA, Leo Chen Carnegie Mellon University, USA, João Paulo Diniz Federal University of Minas Gerais, Brazil, Christian Kästner Carnegie Mellon University, USA, Eduardo Figueiredo Federal University of Minas Gerais, Brazil DOI | ||
08:11 1mTalk | Selecting Fault Revealing Mutants Journal First Thierry Titcheu Chekam University of Luxembourg (SnT), Mike Papadakis University of Luxembourg, Luxembourg, Tegawendé F. Bissyandé University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg, Koushik Sen University of California at Berkeley | ||
08:13 17mTalk | Conversations on Testing 3 Paper Presentations Chu-Pan Wong Carnegie Mellon University, USA, Eduard Baranov Université Catholique de Louvain, Belgium, Giovani Guizzo University College London, UK, M. Ammar Ben Khadra TU Kaiserslautern, Germany, Porfirio Tramontana Department of Electrical Engineering and Information Technologies, University of Naples Federico II, Italy, Thierry Titcheu Chekam University of Luxembourg (SnT), M: Marcel Böhme Monash University, Australia |