Write a Blog >>
Tue 10 Nov 2020 01:43 - 01:44 at Virtual room 1 - ML Testing 1

Trained DNN models are increasingly adopted as integral parts of software systems, but they often perform deficiently in the field. A particularly damaging problem is that DNN models often give false predictions with high confidence, due to the unavoidable slight divergences between operation data and training data. To minimize the loss caused by inaccurate confidence, operational calibration, i.e., calibrating the confidence function of a DNN classifier against its operation domain, becomes a necessary debugging step in the engineering of the whole system.

Operational calibration is difficult considering the limited budget of labeling operation data and the weak interpretability of DNN models. We propose a Bayesian approach to operational calibration that gradually corrects the confidence given by the model under calibration with a small number of labeled operation data deliberately selected from a larger set of unlabeled operation data. The approach is made effective and efficient by leveraging the locality of the learned representation of the DNN model and modeling the calibration as Gaussian Process Regression. Comprehensive experiments with various practical datasets and DNN models show that it significantly outperformed alternative methods, and in some difficult tasks it eliminated about 71% to 97% high-confidence (>0.9) errors with only about 10% of the minimal amount of labeled operation data needed for practical learning techniques to barely work

Tue 10 Nov
Times are displayed in time zone: (UTC) Coordinated Universal Time change

01:30 - 01:32
Talk
Correlations between Deep Neural Network Model Coverage Criteria and Model Quality
Research Papers
Shenao YanRutgers University, USA, Guanhong TaoPurdue University, USA, Xuwei LiuPurdue University, USA, Juan ZhaiRutgers University, USA, Shiqing MaRutgers University, USA, Lei XuNanjing University, China, Xiangyu ZhangPurdue University
DOI
01:33 - 01:34
Talk
Deep Learning Library Testing via Effective Model GenerationACM SIGSOFT Distinguished Paper Award
Research Papers
Zan WangTianjin University, China, Ming YanTianjin University, China, Junjie ChenTianjin University, China, Shuang LiuTianjin University, China, Dongdi ZhangTianjin University, China
DOI
01:35 - 01:36
Talk
Detecting Numerical Bugs in Neural Network ArchitecturesACM SIGSOFT Distinguished Paper Award
Research Papers
Yuhao ZhangPeking University, Luyao RenPeking University, China, Liqian ChenNational University of Defense Technology, China, Yingfei XiongPeking University, Shing-Chi CheungHong Kong University of Science and Technology, China, Tao XiePeking University
DOI
01:37 - 01:38
Talk
Dynamic Slicing for Deep Neural Networks
Research Papers
Ziqi ZhangPeking University, China, Yuanchun LiMicrosoft Research, China, Yao GuoPeking University, Xiangqun ChenPeking University, Yunxin LiuMicrosoft Research, China
DOI
01:39 - 01:40
Talk
Grammar Based Directed Testing of Machine Learning Systems
Journal First
Sakshi UdeshiSingapore University of Technology and Design, Sudipta ChattopadhyaySingapore University of Technology and Design
01:41 - 01:42
Talk
Is Neuron Coverage a Meaningful Measure for Testing Deep Neural Networks?
Research Papers
Fabrice Harel-CanadaUniversity of California at Los Angeles, USA, Lingxiao WangUniversity of California at Los Angeles, USA, Muhammad Ali GulzarUniversity of California at Los Angeles, USA, Quanquan GuUniversity of California at Los Angeles, USA, Miryung KimUniversity of California at Los Angeles, USA
DOI
01:43 - 01:44
Talk
Operational Calibration: Debugging Confidence Errors for DNNs in the Field
Research Papers
Zenan LiNanjing University, China, Xiaoxing MaNanjing University, China, Chang XuNanjing University, China, Jingwei XuNanjing University, China, Chun CaoNanjing University, China, Jian LuNanjing University, China
DOI
01:45 - 02:00
Talk
Conversations on ML Testing 1
Research Papers
Fabrice Harel-CanadaUniversity of California at Los Angeles, USA, Ming YanTianjin University, China, Sakshi UdeshiSingapore University of Technology and Design, Shenao YanRutgers University, USA, Yuhao ZhangPeking University, Zenan LiNanjing University, China, Ziqi ZhangPeking University, China, M: Hamid BagheriUniversity of Nebraska-Lincoln, USA