Write a Blog >>
Thu 12 Nov 2020 08:09 - 08:10 at Virtual room 2 - ML Testing 2

With the increasing adoption of Deep Learning (DL) for critical tasks, such as autonomous driving, the evaluation of the quality of systems that rely on DL has become crucial. Once trained, DL systems produce an output for any arbitrary numeric vector provided as input, regardless of whether it is within or outside the validity domain of the system under test. Hence, the quality of such systems is determined by the intersection between their validity domain and the regions where their outputs exhibit a misbehaviour.

In this paper, we introduce the notion of frontier of behaviours, i.e., the inputs at which the DL system starts to misbehave. If the frontier of misbehaviours is outside the validity domain of the system, the quality check is passed. Otherwise, the inputs at the intersection represent quality deficiencies of the system. We developed DeepJanus, a search-based tool that generates frontier inputs for DL systems. The experimental results obtained for the lane keeping component of a self-driving car show that the frontier of a well trained system contains almost exclusively unrealistic roads that violate the best practices of civil engineering, while the frontier of a poorly trained one includes many valid inputs that point to serious deficiencies of the system.

Thu 12 Nov

Displayed time zone: (UTC) Coordinated Universal Time change

08:00 - 08:30
08:00
2m
Talk
DeepSearch: A Simple and Effective Blackbox Attack for Deep Neural Networks
Research Papers
Fuyuan Zhang MPI-SWS, Germany, Sankalan Pal Chowdhury MPI-SWS, Germany, Maria Christakis MPI-SWS
DOI
08:03
1m
Talk
Machine Learning Based Test Data Generation for Safety-critical Software
Paper Presentations
Ján Čegiň Faculty of Informatics and Information Technologies Slovak Technical University
08:05
1m
Talk
Machine Learning Testing: Survey, Landscapes and Horizons
Journal First
Jie M. Zhang University College London, UK, Mark Harman University College London, UK, Lei Ma Kyushu University, Yang Liu Nanyang Technological University, Singapore
08:07
1m
Talk
Machine Translation Testing via Pathological Invariance
Research Papers
Shashij Gupta IIT Bombay, India, Pinjia He ETH Zurich, Switzerland, Clara Meister ETH Zurich, Switzerland, Zhendong Su ETH Zurich
DOI
08:09
1m
Talk
Model-Based Exploration of the Frontier of Behaviours for Deep Learning System Testing
Research Papers
Vincenzo Riccio USI Lugano, Switzerland, Paolo Tonella USI Lugano, Switzerland
DOI
08:11
1m
Talk
PRODeep: A Platform for Robustness Verification of Deep Neural Networks
Tool Demos
Renjue Li Institute of Software at Chinese Academy of Sciences, China, Jianlin Li Institute of Software at Chinese Academy of Sciences, China, Cheng-Chao Huang Institute of Intelligent Software, China, Pengfei Yang Institute of Software at Chinese Academy of Sciences, China, Xiaowei Huang University of Liverpool, Lijun Zhang Institute of Software, Chinese Academy of Sciences, Bai Xue Institute of Software at Chinese Academy of Sciences, China, Holger Hermanns Saarland University
DOI
08:13
1m
Talk
Testing Machine Learning Code using Polyhedral Region
Visions and Reflections
Md Sohel Ahmed National Institute of Informatics, Japan, Fuyuki Ishikawa National Institute of Informatics, Mahito Sugiyama National Institute of Informatics, Japan
DOI
08:15
15m
Talk
Conversations on ML Testing 2
Paper Presentations
Fuyuan Zhang MPI-SWS, Germany, Ján Čegiň Faculty of Informatics and Information Technologies Slovak Technical University, Mark Harman University College London, UK, Renjue Li Institute of Software at Chinese Academy of Sciences, China, Shashij Gupta IIT Bombay, India, Vincenzo Riccio USI Lugano, Switzerland, M: Shin Yoo Korea Advanced Institute of Science and Technology