Write a Blog >>
Fri 13 Nov 2020 08:03 - 08:04 at Virtual room 1 - Testing 3

Mutation testing research has often used the number of mutants as a surrogate measure for the true execution cost of generating and executing mutants. This poses a potential threat to the validity of the scientific findings reported in the literature. Out of 75 works surveyed in this paper, we found that 54 (72%) are vulnerable to this threat. To investigate the magnitude of the threat, we conducted an empirical evaluation using 10 real-world programs. The results reveal that: i) percentages of randomly sampled mutants differ from the true execution time, on average, by 44%, varying in difference from 19% to 91%; ii) errors arising from using the surrogate correlate with program size (ρ = 0.74) and number of mutants (ρ = 0.76), making the problem more pernicious for more realistic programs; iii) scientific findings concerning sampling strategies would have approximately 37% rank disagreement, indicating potentially dramatic impact on experiment validity. To investigate whether this threat matters in practice, we reproduced a seminal study on Selective Mutation (widely relied upon for more than two decades). The impact is stark: an inconclusive scientific finding using the surrogate is transformed to an unequivocal finding when using the true execution cost.

Fri 13 Nov
Times are displayed in time zone: (UTC) Coordinated Universal Time change

08:00 - 08:02
Talk
Research Papers
Eduard BaranovUniversité Catholique de Louvain, Belgium, Axel LegayUniversité Catholique de Louvain, Belgium, Kuldeep S. MeelNational University of Singapore, Singapore
DOI
08:03 - 08:04
Research paper
Research Papers
Giovani GuizzoUniversity College London, UK, Federica SarroUniversity College London, UK, Mark HarmanUniversity College London, UK
DOI Pre-print
08:05 - 08:06
Talk
Journal First
Porfirio TramontanaDepartment of Electrical Engineering and Information Technologies, University of Naples Federico II, Italy, Domenico AmalfitanoUniversity of Naples Federico II, Nicola AmatucciDepartment of Civil, Architectural and Environmental Engineering, University of Naples Federico II, Italy, Atif MemonApple Inc., Anna Rita FasolinoFederico II University of Naples
08:07 - 08:08
Talk
Research Papers
M. Ammar Ben KhadraTU Kaiserslautern, Germany, Dominik StoffelTU Kaiserslautern, Germany, Wolfgang KunzTU Kaiserslautern, Germany
DOI Pre-print Media Attached
08:09 - 08:10
Talk
Research Papers
Chu-Pan WongCarnegie Mellon University, USA, Jens MeinickeCarnegie Mellon University, USA, Leo ChenCarnegie Mellon University, USA, João P. DinizFederal University of Minas Gerais, Brazil, Christian KästnerCarnegie Mellon University, USA, Eduardo FigueiredoFederal University of Minas Gerais, Brazil
DOI
08:11 - 08:12
Talk
Journal First
Thierry Titcheu ChekamUniversity of Luxembourg (SnT), Mike PapadakisUniversity of Luxembourg, Luxembourg, Tegawendé F. BissyandéUniversity of Luxembourg, Luxembourg, Yves Le TraonUniversity of Luxembourg, Luxembourg, Koushik SenUniversity of California at Berkeley
08:13 - 08:30
Talk
Paper Presentations
Chu-Pan WongCarnegie Mellon University, USA, Eduard BaranovUniversité Catholique de Louvain, Belgium, Giovani GuizzoUniversity College London, UK, M. Ammar Ben KhadraTU Kaiserslautern, Germany, Porfirio TramontanaDepartment of Electrical Engineering and Information Technologies, University of Naples Federico II, Italy, Thierry Titcheu ChekamUniversity of Luxembourg (SnT), M: Marcel BöhmeMonash University, Australia