Detecting Numerical Bugs in Neural Network ArchitecturesACM SIGSOFT Distinguished Paper Award
Detecting bugs in deep learning software at the architecture level provides additional benefits that detecting bugs at the model level does not provide. This paper makes the first attempt to conduct static analysis for detecting numerical bugs at the architecture level. We propose a static analysis approach for detecting numerical bugs in neural architectures based on abstract interpretation. Our approach mainly comprises two kinds of abstraction techniques, i.e., one for tensors and one for numerical values. Moreover, to scale up while maintaining adequate detection precision, we propose two abstraction techniques: tensor partitioning and (elementwise) affine relation analysis to abstract tensors and numerical values, respectively. We realize the combination scheme of tensor partitioning and affine relation analysis (together with interval analysis) as DEBAR, and evaluate it on two datasets: neural architectures with known bugs (collected from existing studies) and real-world neural architectures. The evaluation results show that DEBAR outperforms other tensor and numerical abstraction techniques on accuracy without losing scalability. DEBAR successfully detects all known numerical bugs with no false positives within 1.7–2.3 seconds per architecture. On the real-world architectures, DEBAR reports 529 warnings within 2.6–135.4 seconds per architecture, where 299 warnings are true positives.
Tue 10 NovDisplayed time zone: (UTC) Coordinated Universal Time change
01:30 - 02:00 | |||
01:30 2mTalk | Correlations between Deep Neural Network Model Coverage Criteria and Model Quality Research Papers Shenao Yan Rutgers University, USA, Guanhong Tao Purdue University, USA, Xuwei Liu Purdue University, USA, Juan Zhai Rutgers University, USA, Shiqing Ma Rutgers University, USA, Lei Xu Nanjing University, China, Xiangyu Zhang Purdue University DOI | ||
01:33 1mTalk | Deep Learning Library Testing via Effective Model GenerationACM SIGSOFT Distinguished Paper Award Research Papers Zan Wang Tianjin University, China, Ming Yan Tianjin University, China, Junjie Chen Tianjin University, China, Shuang Liu Tianjin University, China, Dongdi Zhang Tianjin University, China DOI | ||
01:35 1mTalk | Detecting Numerical Bugs in Neural Network ArchitecturesACM SIGSOFT Distinguished Paper Award Research Papers Yuhao Zhang Peking University, Luyao Ren Peking University, China, Liqian Chen National University of Defense Technology, China, Yingfei Xiong Peking University, Shing-Chi Cheung Hong Kong University of Science and Technology, China, Tao Xie Peking University DOI | ||
01:37 1mTalk | Dynamic Slicing for Deep Neural Networks Research Papers Ziqi Zhang Peking University, China, Yuanchun Li Microsoft Research, China, Yao Guo Peking University, Xiangqun Chen Peking University, Yunxin Liu Microsoft Research, China DOI | ||
01:39 1mTalk | Grammar Based Directed Testing of Machine Learning Systems Journal First Sakshi Udeshi Singapore University of Technology and Design, Sudipta Chattopadhyay Singapore University of Technology and Design | ||
01:41 1mTalk | Is Neuron Coverage a Meaningful Measure for Testing Deep Neural Networks? Research Papers Fabrice Harel-Canada University of California at Los Angeles, USA, Lingxiao Wang University of California at Los Angeles, USA, Muhammad Ali Gulzar University of California at Los Angeles, USA, Quanquan Gu University of California at Los Angeles, USA, Miryung Kim University of California at Los Angeles, USA DOI | ||
01:43 1mTalk | Operational Calibration: Debugging Confidence Errors for DNNs in the Field Research Papers Zenan Li Nanjing University, China, Xiaoxing Ma Nanjing University, China, Chang Xu Nanjing University, China, Jingwei Xu Nanjing University, China, Chun Cao Nanjing University, China, Jian Lu Nanjing University, China DOI | ||
01:45 15mTalk | Conversations on ML Testing 1 Research Papers Fabrice Harel-Canada University of California at Los Angeles, USA, Ming Yan Tianjin University, China, Sakshi Udeshi Singapore University of Technology and Design, Shenao Yan Rutgers University, USA, Yuhao Zhang Peking University, Zenan Li Nanjing University, China, Ziqi Zhang Peking University, China, M: Hamid Bagheri University of Nebraska-Lincoln, USA | ||