Using blind analysis for software engineering experiments

Boyce Sigweni, Martin Shepperd

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

Context: In recent years there has been growing concern about conflicting experimental results in empirical software engineering. This has been paralleled by awareness of how bias can impact research results. Objective: To explore the practicalities of blind analysis of experimental results to reduce bias. Method: We apply blind analysis to a real software engineering experiment that compares three feature weighting approaches with a naïve benchmark (sample mean) to the Finnish software effort data set. We use this experiment as an example to explore blind analysis as a method to reduce researcher bias. Results: Our experience shows that blinding can be a relatively straightforward procedure. We also highlight various statistical analysis decisions which ought not be guided by the hunt for statistical significance and show that results can be inverted merely through a seemingly inconsequential statistical nicety (i.e., the degree of trimming). Conclusion: Whilst there are minor challenges and some limits to the degree of blinding possible, blind analysis is a very practical and easy to implement method that supports more objective analysis of experimental results. Therefore we argue that blind analysis should be the norm for analysing software engineering experiments.

Original languageEnglish
Title of host publicationProceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015
PublisherAssociation for Computing Machinery
Volume27-29-April-2015
ISBN (Electronic)9781450333504
DOIs
Publication statusPublished - Apr 27 2015
Event19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015 - Nanjing, China
Duration: Apr 27 2015Apr 29 2015

Other

Other19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015
CountryChina
CityNanjing
Period4/27/154/29/15

Fingerprint

Software engineering
Trimming
Experiments
Statistical methods

All Science Journal Classification (ASJC) codes

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications

Cite this

Sigweni, B., & Shepperd, M. (2015). Using blind analysis for software engineering experiments. In Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015 (Vol. 27-29-April-2015). [a32] Association for Computing Machinery. https://doi.org/10.1145/2745802.2745832
Sigweni, Boyce ; Shepperd, Martin. / Using blind analysis for software engineering experiments. Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015. Vol. 27-29-April-2015 Association for Computing Machinery, 2015.
@inproceedings{81e90c5138174e6db2a9a7f477cd0b04,
title = "Using blind analysis for software engineering experiments",
abstract = "Context: In recent years there has been growing concern about conflicting experimental results in empirical software engineering. This has been paralleled by awareness of how bias can impact research results. Objective: To explore the practicalities of blind analysis of experimental results to reduce bias. Method: We apply blind analysis to a real software engineering experiment that compares three feature weighting approaches with a na{\"i}ve benchmark (sample mean) to the Finnish software effort data set. We use this experiment as an example to explore blind analysis as a method to reduce researcher bias. Results: Our experience shows that blinding can be a relatively straightforward procedure. We also highlight various statistical analysis decisions which ought not be guided by the hunt for statistical significance and show that results can be inverted merely through a seemingly inconsequential statistical nicety (i.e., the degree of trimming). Conclusion: Whilst there are minor challenges and some limits to the degree of blinding possible, blind analysis is a very practical and easy to implement method that supports more objective analysis of experimental results. Therefore we argue that blind analysis should be the norm for analysing software engineering experiments.",
author = "Boyce Sigweni and Martin Shepperd",
year = "2015",
month = "4",
day = "27",
doi = "10.1145/2745802.2745832",
language = "English",
volume = "27-29-April-2015",
booktitle = "Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015",
publisher = "Association for Computing Machinery",

}

Sigweni, B & Shepperd, M 2015, Using blind analysis for software engineering experiments. in Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015. vol. 27-29-April-2015, a32, Association for Computing Machinery, 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015, Nanjing, China, 4/27/15. https://doi.org/10.1145/2745802.2745832

Using blind analysis for software engineering experiments. / Sigweni, Boyce; Shepperd, Martin.

Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015. Vol. 27-29-April-2015 Association for Computing Machinery, 2015. a32.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Using blind analysis for software engineering experiments

AU - Sigweni, Boyce

AU - Shepperd, Martin

PY - 2015/4/27

Y1 - 2015/4/27

N2 - Context: In recent years there has been growing concern about conflicting experimental results in empirical software engineering. This has been paralleled by awareness of how bias can impact research results. Objective: To explore the practicalities of blind analysis of experimental results to reduce bias. Method: We apply blind analysis to a real software engineering experiment that compares three feature weighting approaches with a naïve benchmark (sample mean) to the Finnish software effort data set. We use this experiment as an example to explore blind analysis as a method to reduce researcher bias. Results: Our experience shows that blinding can be a relatively straightforward procedure. We also highlight various statistical analysis decisions which ought not be guided by the hunt for statistical significance and show that results can be inverted merely through a seemingly inconsequential statistical nicety (i.e., the degree of trimming). Conclusion: Whilst there are minor challenges and some limits to the degree of blinding possible, blind analysis is a very practical and easy to implement method that supports more objective analysis of experimental results. Therefore we argue that blind analysis should be the norm for analysing software engineering experiments.

AB - Context: In recent years there has been growing concern about conflicting experimental results in empirical software engineering. This has been paralleled by awareness of how bias can impact research results. Objective: To explore the practicalities of blind analysis of experimental results to reduce bias. Method: We apply blind analysis to a real software engineering experiment that compares three feature weighting approaches with a naïve benchmark (sample mean) to the Finnish software effort data set. We use this experiment as an example to explore blind analysis as a method to reduce researcher bias. Results: Our experience shows that blinding can be a relatively straightforward procedure. We also highlight various statistical analysis decisions which ought not be guided by the hunt for statistical significance and show that results can be inverted merely through a seemingly inconsequential statistical nicety (i.e., the degree of trimming). Conclusion: Whilst there are minor challenges and some limits to the degree of blinding possible, blind analysis is a very practical and easy to implement method that supports more objective analysis of experimental results. Therefore we argue that blind analysis should be the norm for analysing software engineering experiments.

UR - http://www.scopus.com/inward/record.url?scp=84961153212&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84961153212&partnerID=8YFLogxK

U2 - 10.1145/2745802.2745832

DO - 10.1145/2745802.2745832

M3 - Conference contribution

VL - 27-29-April-2015

BT - Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015

PB - Association for Computing Machinery

ER -

Sigweni B, Shepperd M. Using blind analysis for software engineering experiments. In Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, EASE 2015. Vol. 27-29-April-2015. Association for Computing Machinery. 2015. a32 https://doi.org/10.1145/2745802.2745832