Write a Blog >>

This program is tentative and subject to change.

Tue 10 Nov 2020 17:03 - 17:04 at Virtual room 2 - Empirical

Background. Artifact evaluation has been introduced into the software engineering and programming languages research community with a pilot at ESEC/FSE 2011 and has since then enjoyed a healthy adoption throughout the conference landscape.
Objective. In this qualitative study, we examine the expectations of the community toward research artifacts and their evaluation processes.
Method. We conducted a survey including all members of artifact evaluation committees of major conferences in the software engineering and programming language field since the first pilot and compared the answers to expectations set by calls for artifacts and reviewing guidelines.
Results. While we find that some expectations exceed the ones expressed in calls and reviewing guidelines, there is no consensus on quality thresholds for artifacts in general.
We observe very specific quality expectations for specific artifact types for review and later usage, but also a lack of their communication in calls.
We also find problematic inconsistencies in the terminology used to express artifact evaluation’s most important purpose – replicability.
Conclusion. We derive several actionable suggestions which can help to mature artifact evaluation in the inspected community and also to aid its introduction into other communities in computer science.

This program is tentative and subject to change.

Tue 10 Nov
Times are displayed in time zone: (UTC) Coordinated Universal Time change

17:00 - 17:02
Talk
Research Papers
Jacob KrügerUniversity of Magdeburg, Germany, Thorsten BergerChalmers University of Technology, Sweden / University of Gothenburg, Sweden
17:03 - 17:04
Talk
Research Papers
Ben HermannTechnical University Dortmund, Stefan WinterTU Darmstadt, Germany, Janet SiegmundTU Chemnitz, Germany
DOI Pre-print Media Attached
17:05 - 17:06
Talk
Research Papers
Umme Ayda MannanOregon State University, USA, Iftekhar AhmedUniversity of California at Irvine, USA, Carlos JensenOregon State University, USA, Anita SarmaOregon State University, USA
17:07 - 17:08
Talk
Research Papers
Massimiliano Di PentaUniversity of Sannio, Italy, Gabriele BavotaUSI Lugano, Switzerland, Fiorella ZampettiUniversity of Sannio, Italy
17:09 - 17:10
Talk
Journal First
Akond RahmanTennessee Tech University, Effat FarhanaNC State University, Laurie WilliamsNorth Carolina State University
Link to publication DOI Pre-print Media Attached
17:11 - 17:12
Talk
Research Papers
Dan Gopstein New York University, USA, Anne-Laure FayardNew York University, USA, Sven ApelSaarland University, Germany, Justin CapposNew York University, USA
Pre-print
17:13 - 17:30
Talk
Paper Presentations