Deep Learning Library Testing via Effective Model GenerationACM SIGSOFT Distinguished Paper Award
Deep learning (DL) techniques are rapidly developed and have been widely adopted in practice. However, similar to traditional software systems, DL systems also contain bugs, which could cause serious impacts especially in safety-critical domains. Recently, many research approaches have focused on testing DL models, while little attention has been paid for testing DL libraries, which is the basis of building DL models and directly affects the behavior of DL systems. In this work, we propose a novel approach, LEMON, to testing DL libraries. In particular, we (1) design a series of mutation rules for DL models, with the purpose of exploring different invoking sequences of library code and hard-to-trigger behaviors; and (2) propose a heuristic strategy to guide the model generation process towards the direction of amplifying the inconsistent degrees of the inconsistencies between different DL libraries caused by bugs, so as to mitigate the impact of potential noise introduced by uncertain factors in DL libraries. We conducted an empirical study to evaluate the effectiveness of LEMON with 20 release versions of 4 widely-used DL libraries, i.e., TensorFlow, Theano, CNTK, MXNet. The results demonstrate that LEMON detected 24 new bugs in the latest release versions of these libraries, where 7 bugs have been confirmed and one bug has been fixed by developers. Besides, the results confirm that the heuristic strategy for model generation indeed effectively guides LEMON in amplifying the inconsistent degrees for bugs.
Tue 10 Nov Times are displayed in time zone: (UTC) Coordinated Universal Time change
| 01:30 - 01:32 Talk | Correlations between Deep Neural Network Model Coverage Criteria and Model Quality Research Papers Shenao YanRutgers University, USA, Guanhong TaoPurdue University, USA, Xuwei LiuPurdue University, USA, Juan ZhaiRutgers University, USA, Shiqing MaRutgers University, USA, Lei XuNanjing University, China, Xiangyu ZhangPurdue University DOI | ||
| 01:33 - 01:34 Talk | Deep Learning Library Testing via Effective Model GenerationACM SIGSOFT Distinguished Paper Award Research Papers Zan WangTianjin University, China, Ming YanTianjin University, China, Junjie ChenTianjin University, China, Shuang LiuTianjin University, China, Dongdi ZhangTianjin University, China DOI | ||
| 01:35 - 01:36 Talk | Detecting Numerical Bugs in Neural Network ArchitecturesACM SIGSOFT Distinguished Paper Award Research Papers Yuhao ZhangPeking University, Luyao RenPeking University, China, Liqian ChenNational University of Defense Technology, China, Yingfei XiongPeking University, Shing-Chi CheungHong Kong University of Science and Technology, China, Tao XiePeking University DOI | ||
| 01:37 - 01:38 Talk | Dynamic Slicing for Deep Neural Networks Research Papers Ziqi ZhangPeking University, China, Yuanchun LiMicrosoft Research, China, Yao GuoPeking University, Xiangqun ChenPeking University, Yunxin LiuMicrosoft Research, China DOI | ||
| 01:39 - 01:40 Talk | Grammar Based Directed Testing of Machine Learning Systems Journal First Sakshi UdeshiSingapore University of Technology and Design, Sudipta ChattopadhyaySingapore University of Technology and Design | ||
| 01:41 - 01:42 Talk | Is Neuron Coverage a Meaningful Measure for Testing Deep Neural Networks? Research Papers Fabrice Harel-CanadaUniversity of California at Los Angeles, USA, Lingxiao WangUniversity of California at Los Angeles, USA, Muhammad Ali GulzarUniversity of California at Los Angeles, USA, Quanquan GuUniversity of California at Los Angeles, USA, Miryung KimUniversity of California at Los Angeles, USA DOI | ||
| 01:43 - 01:44 Talk | Operational Calibration: Debugging Confidence Errors for DNNs in the Field Research Papers Zenan LiNanjing University, China, Xiaoxing MaNanjing University, China, Chang XuNanjing University, China, Jingwei XuNanjing University, China, Chun CaoNanjing University, China, Jian LuNanjing University, China DOI | ||
| 01:45 - 02:00 Talk | Conversations on ML Testing 1 Research Papers Fabrice Harel-CanadaUniversity of California at Los Angeles, USA, Ming YanTianjin University, China, Sakshi UdeshiSingapore University of Technology and Design, Shenao YanRutgers University, USA, Yuhao ZhangPeking University, Zenan LiNanjing University, China, Ziqi ZhangPeking University, China, M: Hamid BagheriUniversity of Nebraska-Lincoln, USA | ||