Abstract
Context: In recent years there has been growing concern about conflicting experimental results in empirical software engineering. This has been paralleled by awareness of how bias can impact research results. Objective: To explore the practicalities of blind analysis of experimental results to reduce bias. Method: We apply blind analysis to a real software engineering experiment that compares three feature weighting approaches with a naïve benchmark (sample mean) to the Finnish software effort data set. We use this experiment as an example to explore blind analysis as a method to reduce researcher bias. Results: Our experience shows that blinding can be a relatively straightforward procedure. We also highlight various statistical analysis decisions which ought not be guided by the hunt for statistical significance and show that results can be inverted merely through a seemingly inconsequential statistical nicety (i.e., the degree of trimming). Conclusion: Whilst there are minor challenges and some limits to the degree of blinding possible, blind analysis is a very practical and easy to implement method that supports more objective analysis of experimental results. Therefore we argue that blind analysis should be the norm for analysing software engineering experiments.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015 |
| Publisher | Association for Computing Machinery |
| Volume | 27-29-April-2015 |
| ISBN (Electronic) | 9781450333504 |
| DOIs | |
| Publication status | Published - Apr 27 2015 |
| Event | 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015 - Nanjing, China Duration: Apr 27 2015 → Apr 29 2015 |
Other
| Other | 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015 |
|---|---|
| Country | China |
| City | Nanjing |
| Period | 4/27/15 → 4/29/15 |
Fingerprint
All Science Journal Classification (ASJC) codes
- Software
- Human-Computer Interaction
- Computer Vision and Pattern Recognition
- Computer Networks and Communications
Cite this
}
Using blind analysis for software engineering experiments. / Sigweni, Boyce; Shepperd, Martin.
Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015. Vol. 27-29-April-2015 Association for Computing Machinery, 2015. a32.Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
TY - GEN
T1 - Using blind analysis for software engineering experiments
AU - Sigweni, Boyce
AU - Shepperd, Martin
PY - 2015/4/27
Y1 - 2015/4/27
N2 - Context: In recent years there has been growing concern about conflicting experimental results in empirical software engineering. This has been paralleled by awareness of how bias can impact research results. Objective: To explore the practicalities of blind analysis of experimental results to reduce bias. Method: We apply blind analysis to a real software engineering experiment that compares three feature weighting approaches with a naïve benchmark (sample mean) to the Finnish software effort data set. We use this experiment as an example to explore blind analysis as a method to reduce researcher bias. Results: Our experience shows that blinding can be a relatively straightforward procedure. We also highlight various statistical analysis decisions which ought not be guided by the hunt for statistical significance and show that results can be inverted merely through a seemingly inconsequential statistical nicety (i.e., the degree of trimming). Conclusion: Whilst there are minor challenges and some limits to the degree of blinding possible, blind analysis is a very practical and easy to implement method that supports more objective analysis of experimental results. Therefore we argue that blind analysis should be the norm for analysing software engineering experiments.
AB - Context: In recent years there has been growing concern about conflicting experimental results in empirical software engineering. This has been paralleled by awareness of how bias can impact research results. Objective: To explore the practicalities of blind analysis of experimental results to reduce bias. Method: We apply blind analysis to a real software engineering experiment that compares three feature weighting approaches with a naïve benchmark (sample mean) to the Finnish software effort data set. We use this experiment as an example to explore blind analysis as a method to reduce researcher bias. Results: Our experience shows that blinding can be a relatively straightforward procedure. We also highlight various statistical analysis decisions which ought not be guided by the hunt for statistical significance and show that results can be inverted merely through a seemingly inconsequential statistical nicety (i.e., the degree of trimming). Conclusion: Whilst there are minor challenges and some limits to the degree of blinding possible, blind analysis is a very practical and easy to implement method that supports more objective analysis of experimental results. Therefore we argue that blind analysis should be the norm for analysing software engineering experiments.
UR - http://www.scopus.com/inward/record.url?scp=84961153212&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84961153212&partnerID=8YFLogxK
U2 - 10.1145/2745802.2745832
DO - 10.1145/2745802.2745832
M3 - Conference contribution
VL - 27-29-April-2015
BT - Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015
PB - Association for Computing Machinery
ER -