Write a Blog >>
Tue 10 Nov 2020 17:03 - 17:04 at Virtual room 2 - Empirical

\emph{Background.}
Artifact evaluation has been introduced into the software engineering and programming languages research community with a pilot at ESEC/FSE 2011 and has since then enjoyed a healthy adoption throughout the conference landscape.
%
\emph{Objective.}
In this qualitative study, we examine the expectations of the community toward research artifacts and their evaluation processes.
%
\emph{Method.}
We conducted a survey including all members of artifact evaluation committees of major conferences in the software engineering and programming language field since the first pilot and compared the answers to expectations set by calls for artifacts and reviewing guidelines.
%
\emph{Results.}
While we find that some expectations exceed the ones expressed in calls and reviewing guidelines, there is no consensus on quality thresholds for artifacts in general.
We observe very specific quality expectations for specific artifact types for review and later usage, but also a lack of their communication in calls.
We also find problematic inconsistencies in the terminology used to express artifact evaluation's most important purpose – \emph{replicability}.
%
\emph{Conclusion.}
We derive several actionable suggestions which can help to mature artifact evaluation in the inspected community and also to aid its introduction into other communities in computer science.

Tue 10 Nov

Displayed time zone: (UTC) Coordinated Universal Time change

17:00 - 17:30
17:00
2m
Talk
An Empirical Analysis of the Costs of Clone- and Platform-Oriented Software Reuse
Research Papers
Jacob Krüger University of Magdeburg, Germany, Thorsten Berger Chalmers University of Technology, Sweden / University of Gothenburg, Sweden
DOI
17:03
1m
Talk
Community Expectations for Research Artifacts and Evaluation ProcessesACM SIGSOFT Distinguished Paper Award
Research Papers
Ben Hermann Technical University Dortmund, Stefan Winter TU Darmstadt, Janet Siegmund TU Chemnitz, Germany
DOI Pre-print Media Attached
17:05
1m
Talk
On the Relationship between Design Discussions and Design Quality: A Case Study of Apache Projects
Research Papers
Umme Ayda Mannan Oregon State University, USA, Iftekhar Ahmed University of California at Irvine, USA, Carlos Jensen Oregon State University, USA, Anita Sarma Oregon State University, USA
DOI
17:07
1m
Talk
On the Relationship between Refactoring Actions and Bugs: A Differentiated Replication
Research Papers
Massimiliano Di Penta University of Sannio, Italy, Gabriele Bavota USI Lugano, Switzerland, Fiorella Zampetti University of Sannio, Italy
DOI
17:09
1m
Talk
The 'as Code' Activities: Development Anti-patterns for Infrastructure as Code
Journal First
Akond Rahman Tennessee Tech University, Effat Farhana NC State University, Laurie Williams North Carolina State University
Link to publication DOI Pre-print Media Attached
17:11
1m
Talk
Thinking Aloud about Confusing Code: A Qualitative Investigation of Program Comprehension and Atoms of Confusion
Research Papers
Dan Gopstein New York University, USA, Anne-Laure Fayard New York University, USA, Sven Apel Saarland University, Germany, Justin Cappos New York University, USA
DOI Pre-print
17:13
17m
Talk
Conversations on Empirical 1
Paper Presentations
Akond Rahman Tennessee Tech University, Ben Hermann Technical University Dortmund, Iftekhar Ahmed University of California at Irvine, USA, Jacob Krüger Otto von Guericke University Magdeburg, Massimiliano Di Penta University of Sannio, Italy, M: Brittany Johnson George Mason University