Abstract
Context: In recent years there has been growing concern about conflicting experimental results in empirical software engineering. This has been paralleled by awareness of how bias can impact research results. Objective: To explore the practicalities of blind analysis of experimental results to reduce bias. Method: We apply blind analysis to a real software engineering experiment that compares three feature weighting approaches with a naïve benchmark (sample mean) to the Finnish software effort data set. We use this experiment as an example to explore blind analysis as a method to reduce researcher bias. Results: Our experience shows that blinding can be a relatively straightforward procedure. We also highlight various statistical analysis decisions which ought not be guided by the hunt for statistical significance and show that results can be inverted merely through a seemingly inconsequential statistical nicety (i.e., the degree of trimming). Conclusion: Whilst there are minor challenges and some limits to the degree of blinding possible, blind analysis is a very practical and easy to implement method that supports more objective analysis of experimental results. Therefore we argue that blind analysis should be the norm for analysing software engineering experiments.
Original language | English |
---|---|
Title of host publication | Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015 |
Publisher | Association for Computing Machinery |
Volume | 27-29-April-2015 |
ISBN (Electronic) | 9781450333504 |
DOIs | |
Publication status | Published - Apr 27 2015 |
Event | 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015 - Nanjing, China Duration: Apr 27 2015 → Apr 29 2015 |
Other
Other | 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015 |
---|---|
Country | China |
City | Nanjing |
Period | 4/27/15 → 4/29/15 |
All Science Journal Classification (ASJC) codes
- Software
- Human-Computer Interaction
- Computer Vision and Pattern Recognition
- Computer Networks and Communications