2024-01-23-bosma24a.md

title	abstract	layout	series	publisher	issn	id	month	tex_title	firstpage	lastpage	page	order	cycles	bibtex_author	author	date	address	container-title	volume	genre	issued	pdf	extras
Reproducibility of Training Deep Learning Models for Medical Image Analysis	Performance of deep learning algorithms varies due to their development data and training method, but also due to several stochastic processes during training. Due to these random factors, a single training run may not accurately reflect the performance of a given training method. Statistical comparisons in literature between different deep learning training methods typically ignore this performance variation between training runs and incorrectly claim significance of changes in training method. We hypothesize that the impact of such performance variation is substantial, such that it may invalidate biomedical competition leaderboards and some scientific papers. To test this, we investigate the reproducibility of training deep learning algorithms for medical image analysis. We repeated training runs from prior scientific studies: three diagnostic tasks (pancreatic cancer detection in CT, clinically significant prostate cancer detection in MRI, and lung nodule malignancy risk estimation in low-dose CT) and two organ segmentation tasks (pancreas segmentation in CT and prostate segmentation in MRI). A previously published top-performing algorithm for each task was trained multiple times to determine the variance in model performance. For all three diagnostic algorithms, performance variation from retraining was significant compared to data variance. Statistically comparing independently trained algorithms from the same training method using the same dataset should follow the null hypothesis, but we observed claimed significance with a p-value below 0.05 in 15% of comparisons with conventional testing (paired bootstrapping). We conclude that variance in model performance due to retraining is substantial and should be accounted for.	inproceedings	Proceedings of Machine Learning Research	PMLR	2640-3498	bosma24a	0	Reproducibility of Training Deep Learning Models for Medical Image Analysis	1269	1287	1269-1287	1269	false	Bosma, Joeran Sander and Peeters, Dr\'e and Alves, Nat\'alia and Saha, Anindo and Saghir, Zaigham and Jacobs, Colin and Huisman, Henkjan	given family Joeran Sander Bosma given family Dré Peeters given family Natália Alves given family Anindo Saha given family Zaigham Saghir given family Colin Jacobs given family Henkjan Huisman	2024-01-23		Medical Imaging with Deep Learning	227	inproceedings	date-parts 2024 1 23	https://proceedings.mlr.press/v227/bosma24a/bosma24a.pdf