Clustering Test Steps in Natural Language toward Automating Test Automation
For large industrial applications, system test cases are still often described in natural language (NL), and their number can reach thousands. Test automation is to automatically execute the test cases. Achieving test automation typically requires substantial manual effort for creating executable test scripts from these NL test cases. In particular, given that each NL test case consists of a sequence of NL test steps, testers first implement a test API method for each test step and then write a test script for invoking these test API methods sequentially for test automation. Across different test cases, multiple test steps can share semantic similarities, supposedly mapped to the same API method. However, due to numerous test steps in various NL forms under manual inspection, testers may not realize those semantically similar test steps and thus waste effort to implement duplicate test API methods for them. To address this issue, in this paper, we propose a new approach based on natural language processing to cluster similar NL test steps together such that the test steps in each cluster can be mapped to the same test API method. Our approach includes domain-specific word embedding training along with measurement based on Relaxed Word Mover’sDistance to analyze the similarity of test steps. Our approach also includes a technique to combine hierarchical agglomerative clustering and K-means clustering post-refinement to derive high-quality and manually-adjustable clustering results. The evaluation results of our approach on a large industrial mobile app, WeChat, show that our approach can cluster the test steps with high accuracy, substantially reducing the number of clusters and thus reducing the downstream manual effort. In particular, compared with the baseline approach, our approach achieves 79.8% improvement on cluster quality, reducing 65.9% number of clusters, i.e., the number of test API methods to be implemented.
Thu 12 NovDisplayed time zone: (UTC) Coordinated Universal Time change
01:30 - 02:00 | Testing 2Tool Demos / Paper Presentations / Journal First / Industry Papers / Research Papers at Virtual room 2 | ||
01:30 2mTalk | Clustering Test Steps in Natural Language toward Automating Test Automation Industry Papers Linyi Li University of Illinois at Urbana-Champaign, Zhenwen Li Peking University, China, Weijie Zhang Tencent, China, Jun Zhou Tencent, China, Pengcheng Wang Tencent, China, Jing Wu Tencent, China, Guanghua He Tencent, China, Xia Zeng Tencent, China, Yuetang Deng Tencent, Inc., Tao Xie Peking University DOI | ||
01:33 1mTalk | PRF: A Framework for Building Automatic Program Repair Prototypes for JVM-Based Languages Tool Demos DOI Pre-print | ||
01:35 1mTalk | SOSRepair: Expressive Semantic Search for Real-World Program Repair Journal First Afsoon Afzal Carnegie Mellon University, Manish Motwani University of Massachusetts, Amherst, Kathryn Stolee North Carolina State University, Yuriy Brun University of Massachusetts Amherst, Claire Le Goues Carnegie Mellon University Link to publication DOI Pre-print Media Attached | ||
01:37 1mTalk | tsDetect: An Open Source Test Smells Detection Tool Tool Demos Anthony Peruma Rochester Institute of Technology, Khalid Almalki Rochester Institute of Technology, USA, Christian D. Newman Rochester Institute of Technology, Mohamed Wiem Mkaouer Rochester Institute of Technology, Ali Ouni ETS Montreal, University of Quebec, Fabio Palomba University of Salerno DOI Pre-print Media Attached | ||
01:39 1mTalk | Understanding and Automatically Detecting Conflicting Interactions between Smart Home IoT Applications Research Papers Rahmadi Trimananda University of California at Irvine, USA, Seyed Amir Hossein Aqajari University of California at Irvine, USA, Jason Chuang University of California at Irvine, USA, Brian Demsky University of California at Irvine, Guoqing Harry Xu University of California at Los Angeles, Shan Lu University of Chicago, USA DOI Pre-print Media Attached File Attached | ||
01:41 19mTalk | Conversations on Testing 2 Paper Presentations Afsoon Afzal Carnegie Mellon University, Anthony Peruma Rochester Institute of Technology, Linyi Li University of Illinois at Urbana-Champaign, Rahmadi Trimananda University of California at Irvine, USA, M: Corina S. Păsăreanu Carnegie Mellon University Silicon Valley, NASA Ames Research Center |