Using blind analysis for software engineering experiments

Boyce Sigweni, Martin Shepperd

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

Context: In recent years there has been growing concern about conflicting experimental results in empirical software engineering. This has been paralleled by awareness of how bias can impact research results. Objective: To explore the practicalities of blind analysis of experimental results to reduce bias. Method: We apply blind analysis to a real software engineering experiment that compares three feature weighting approaches with a naïve benchmark (sample mean) to the Finnish software effort data set. We use this experiment as an example to explore blind analysis as a method to reduce researcher bias. Results: Our experience shows that blinding can be a relatively straightforward procedure. We also highlight various statistical analysis decisions which ought not be guided by the hunt for statistical significance and show that results can be inverted merely through a seemingly inconsequential statistical nicety (i.e., the degree of trimming). Conclusion: Whilst there are minor challenges and some limits to the degree of blinding possible, blind analysis is a very practical and easy to implement method that supports more objective analysis of experimental results. Therefore we argue that blind analysis should be the norm for analysing software engineering experiments.

Original languageEnglish
Title of host publicationProceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015
PublisherAssociation for Computing Machinery
Volume27-29-April-2015
ISBN (Electronic)9781450333504
DOIs
Publication statusPublished - Apr 27 2015
Event19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015 - Nanjing, China
Duration: Apr 27 2015Apr 29 2015

Other

Other19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015
CountryChina
CityNanjing
Period4/27/154/29/15

All Science Journal Classification (ASJC) codes

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Using blind analysis for software engineering experiments'. Together they form a unique fingerprint.

  • Cite this

    Sigweni, B., & Shepperd, M. (2015). Using blind analysis for software engineering experiments. In Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015 (Vol. 27-29-April-2015). [a32] Association for Computing Machinery. https://doi.org/10.1145/2745802.2745832