Write a Blog >>
Thu 12 Nov 2020 08:09 - 08:10 at Virtual room 2 - ML Testing 2

With the increasing adoption of Deep Learning (DL) for critical tasks, such as autonomous driving, the evaluation of the quality of systems that rely on DL has become crucial. Once trained, DL systems produce an output for any arbitrary numeric vector provided as input, regardless of whether it is within or outside the validity domain of the system under test. Hence, the quality of such systems is determined by the intersection between their validity domain and the regions where their outputs exhibit a misbehaviour.

In this paper, we introduce the notion of frontier of behaviours, i.e., the inputs at which the DL system starts to misbehave. If the frontier of misbehaviours is outside the validity domain of the system, the quality check is passed. Otherwise, the inputs at the intersection represent quality deficiencies of the system. We developed DeepJanus, a search-based tool that generates frontier inputs for DL systems. The experimental results obtained for the lane keeping component of a self-driving car show that the frontier of a well trained system contains almost exclusively unrealistic roads that violate the best practices of civil engineering, while the frontier of a poorly trained one includes many valid inputs that point to serious deficiencies of the system.

Thu 12 Nov
Times are displayed in time zone: (UTC) Coordinated Universal Time change

08:00 - 08:02
Talk
DeepSearch: A Simple and Effective Blackbox Attack for Deep Neural Networks
Research Papers
Fuyuan ZhangMPI-SWS, Germany, Sankalan Pal ChowdhuryMPI-SWS, Germany, Maria ChristakisMPI-SWS
DOI
08:03 - 08:04
Talk
Machine Learning Based Test Data Generation for Safety-critical Software
Paper Presentations
Ján ČegiňFaculty of Informatics and Information Technologies Slovak Technical University
08:05 - 08:06
Talk
Machine Learning Testing: Survey, Landscapes and Horizons
Journal First
Jie M. ZhangUniversity College London, UK, Mark HarmanUniversity College London, UK, Lei MaKyushu University, Yang LiuNanyang Technological University, Singapore
08:07 - 08:08
Talk
Machine Translation Testing via Pathological Invariance
Research Papers
Shashij GuptaIIT Bombay, India, Pinjia HeETH Zurich, Switzerland, Clara MeisterETH Zurich, Switzerland, Zhendong SuETH Zurich
DOI
08:09 - 08:10
Talk
Model-Based Exploration of the Frontier of Behaviours for Deep Learning System Testing
Research Papers
Vincenzo RiccioUSI Lugano, Switzerland, Paolo TonellaUSI Lugano, Switzerland
DOI
08:11 - 08:12
Talk
PRODeep: A Platform for Robustness Verification of Deep Neural Networks
Tool Demos
Renjue LiInstitute of Software at Chinese Academy of Sciences, China, Jianlin LiInstitute of Software at Chinese Academy of Sciences, China, Cheng-Chao HuangInstitute of Intelligent Software, China, Pengfei YangInstitute of Software at Chinese Academy of Sciences, China, Xiaowei HuangUniversity of Liverpool, Lijun ZhangInstitute of Software, Chinese Academy of Sciences, Bai XueInstitute of Software at Chinese Academy of Sciences, China, Holger HermannsSaarland University
DOI
08:13 - 08:14
Talk
Testing Machine Learning Code using Polyhedral Region
Visions and Reflections
Md Sohel AhmedNational Institute of Informatics, Japan, Fuyuki IshikawaNational Institute of Informatics, Mahito SugiyamaNational Institute of Informatics, Japan
DOI
08:15 - 08:30
Talk
Conversations on ML Testing 2
Paper Presentations
Fuyuan ZhangMPI-SWS, Germany, Ján ČegiňFaculty of Informatics and Information Technologies Slovak Technical University, Mark HarmanUniversity College London, UK, Renjue LiInstitute of Software at Chinese Academy of Sciences, China, Shashij GuptaIIT Bombay, India, Vincenzo RiccioUSI Lugano, Switzerland, M: Shin YooKorea Advanced Institute of Science and Technology