Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?
Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation. Existing studies for GUI element detection directly borrow the mature methods from computer vision (CV) domain, including old fashioned ones that rely on traditional image processing features (e.g., canny edge, contours), and deep learning models that learn to detect from large-scale GUI data. Unfortunately, these CV methods are not originally designed with the awareness of the unique characteristics of GUIs and GUI elements and the high localization accuracy of the GUI element detection task. We conduct the first large-scale empirical study of seven representative GUI element detection methods on over 50k GUI images to understand the capabilities, limitations and effective designs of these methods. This study not only sheds the light on the technical challenges to be addressed but also informs the design of new GUI element detection methods. We accordingly design a new GUI-specific old-fashioned method for non-text GUI element detection which adopts a novel top-down coarse-to-fine strategy, and incorporate it with the mature deep learning model for GUI text detection.Our evaluation on 25,000 GUI images shows that our method significantly advances the start-of-the-art performance in GUI element detection.
Thu 12 NovDisplayed time zone: (UTC) Coordinated Universal Time change
01:00 - 01:30 | |||
01:00 2mLong-paper | FrUITeR: A Framework for Evaluating UI Test Reuse Research Papers Yixue Zhao University of Massachusetts at Amherst, USA, Justin Chen Columbia University, USA, Adriana Sejfia University of Southern California, USA, Marcelo Schmitt Laser University of Southern California, USA, Jie M. Zhang University College London, UK, Federica Sarro University College London, UK, Mark Harman University College London, UK, Nenad Medvidović University of Southern California, USA DOI Pre-print Media Attached | ||
01:03 1mTalk | ModCon: A Model-Based Testing Platform for Smart Contracts Tool Demos Ye Liu Nanyang Technological University, Singapore, Yi Li Nanyang Technological University, Shang-Wei Lin Nanyang Technological University, Singapore, Qiang Yan WeBank, n.n. DOI Pre-print Media Attached | ||
01:05 1mTalk | Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? Research Papers Jieshan Chen Australian National University, Australia, Mulong Xie Australian National University, Australia, Zhenchang Xing Australian National University, Australia, Chunyang Chen Monash University, Australia, Xiwei (Sherry) Xu Data61 at CSIRO, Australia, Liming Zhu Data61 at CSIRO, Australia / UNSW, Australia, Guoqiang Li Shanghai Jiao Tong University, China DOI | ||
01:07 1mTalk | UIED: A Hybrid Tool for GUI Element Detection Tool Demos Mulong Xie Australian National University, Australia, Sidong Feng Australian National University, Australia, Zhenchang Xing Australian National University, Australia, Jieshan Chen Australian National University, Australia, Chunyang Chen Monash University, Australia DOI | ||
01:09 1mTalk | WebRR: Self-Replay Enhanced Robust Record/Replay for Web Application Testing Industry Papers Zhenyue Long China Southern Power Grid, China, Guoquan Wu Institute of Software at Chinese Academy of Sciences, China, Xiaojiang Chen China Southern Power Grid, China, Wei Chen Institute of Software at Chinese Academy of Sciences, China, Jun Wei State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences & University of Chinese Academy of Sciences DOI | ||
01:11 19mTalk | Conversations on Testing 1 Paper Presentations Guoquan Wu Institute of Software at Chinese Academy of Sciences, China, Jieshan Chen Australian National University, Australia, Sidong Feng Australian National University, Australia, Ye Liu Nanyang Technological University, Singapore, Yixue Zhao University of Massachusetts at Amherst, USA, Mulong Xie Australian National University, Australia, M: Corina S. Păsăreanu Carnegie Mellon University Silicon Valley, NASA Ames Research Center | ||