Detecting Numerical Bugs in Neural Network ArchitecturesACM SIGSOFT Distinguished Paper Award
Detecting bugs in deep learning software at the architecture level provides additional benefits that detecting bugs at the model level does not provide. This paper makes the first attempt to conduct static analysis for detecting numerical bugs at the architecture level. We propose a static analysis approach for detecting numerical bugs in neural architectures based on abstract interpretation. Our approach mainly comprises two kinds of abstraction techniques, i.e., one for tensors and one for numerical values. Moreover, to scale up while maintaining adequate detection precision, we propose two abstraction techniques: tensor partitioning and (elementwise) affine relation analysis to abstract tensors and numerical values, respectively. We realize the combination scheme of tensor partitioning and affine relation analysis (together with interval analysis) as DEBAR, and evaluate it on two datasets: neural architectures with known bugs (collected from existing studies) and real-world neural architectures. The evaluation results show that DEBAR outperforms other tensor and numerical abstraction techniques on accuracy without losing scalability. DEBAR successfully detects all known numerical bugs with no false positives within 1.7–2.3 seconds per architecture. On the real-world architectures, DEBAR reports 529 warnings within 2.6–135.4 seconds per architecture, where 299 warnings are true positives.
Tue 10 Nov Times are displayed in time zone: (UTC) Coordinated Universal Time change
| 01:30 - 01:32 Talk | Correlations between Deep Neural Network Model Coverage Criteria and Model Quality Research Papers Shenao YanRutgers University, USA, Guanhong TaoPurdue University, USA, Xuwei LiuPurdue University, USA, Juan ZhaiRutgers University, USA, Shiqing MaRutgers University, USA, Lei XuNanjing University, China, Xiangyu ZhangPurdue University DOI | ||
| 01:33 - 01:34 Talk | Deep Learning Library Testing via Effective Model GenerationACM SIGSOFT Distinguished Paper Award Research Papers Zan WangTianjin University, China, Ming YanTianjin University, China, Junjie ChenTianjin University, China, Shuang LiuTianjin University, China, Dongdi ZhangTianjin University, China DOI | ||
| 01:35 - 01:36 Talk | Detecting Numerical Bugs in Neural Network ArchitecturesACM SIGSOFT Distinguished Paper Award Research Papers Yuhao ZhangPeking University, Luyao RenPeking University, China, Liqian ChenNational University of Defense Technology, China, Yingfei XiongPeking University, Shing-Chi CheungHong Kong University of Science and Technology, China, Tao XiePeking University DOI | ||
| 01:37 - 01:38 Talk | Dynamic Slicing for Deep Neural Networks Research Papers Ziqi ZhangPeking University, China, Yuanchun LiMicrosoft Research, China, Yao GuoPeking University, Xiangqun ChenPeking University, Yunxin LiuMicrosoft Research, China DOI | ||
| 01:39 - 01:40 Talk | Grammar Based Directed Testing of Machine Learning Systems Journal First Sakshi UdeshiSingapore University of Technology and Design, Sudipta ChattopadhyaySingapore University of Technology and Design | ||
| 01:41 - 01:42 Talk | Is Neuron Coverage a Meaningful Measure for Testing Deep Neural Networks? Research Papers Fabrice Harel-CanadaUniversity of California at Los Angeles, USA, Lingxiao WangUniversity of California at Los Angeles, USA, Muhammad Ali GulzarUniversity of California at Los Angeles, USA, Quanquan GuUniversity of California at Los Angeles, USA, Miryung KimUniversity of California at Los Angeles, USA DOI | ||
| 01:43 - 01:44 Talk | Operational Calibration: Debugging Confidence Errors for DNNs in the Field Research Papers Zenan LiNanjing University, China, Xiaoxing MaNanjing University, China, Chang XuNanjing University, China, Jingwei XuNanjing University, China, Chun CaoNanjing University, China, Jian LuNanjing University, China DOI | ||
| 01:45 - 02:00 Talk | Conversations on ML Testing 1 Research Papers Fabrice Harel-CanadaUniversity of California at Los Angeles, USA, Ming YanTianjin University, China, Sakshi UdeshiSingapore University of Technology and Design, Shenao YanRutgers University, USA, Yuhao ZhangPeking University, Zenan LiNanjing University, China, Ziqi ZhangPeking University, China, M: Hamid BagheriUniversity of Nebraska-Lincoln, USA | ||