2024-01-23-windsor24a.md

title	abstract	layout	series	publisher	issn	id	month	tex_title	firstpage	lastpage	page	order	cycles	bibtex_author	author	date	address	container-title	volume	genre	issued	pdf	extras
Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime	This paper explores training medical vision-language models (VLMs) – where the visual and language inputs are embedded into a common space – with a particular focus on scenarios where training data is limited, as is often the case in clinical datasets. We explore several candidate methods to improve low-data performance, including: (i) adapting generic pre-trained models to novel image and text domains (i.e. medical imaging and reports) via unimodal self-supervision; (ii) using local (e.g. GLoRIA) & global (e.g. InfoNCE) contrastive loss functions as well as a combination of the two; (iii) extra supervision during VLM training, via: (a) image- and text-only self-supervision, and (b) creating additional positive image-text pairs for training through augmentation and nearest-neighbour search. Using text-to-image retrieval as a benchmark, we evaluate the performance of these methods with variable sized training datasets of paired chest X-rays and radiological reports. Combined, they significantly improve retrieval compared to fine-tuning CLIP, roughly equivalent to training with $10\times$ the data. A similar pattern is found in the downstream task classification of CXR-related conditions with our method outperforming CLIP and also BioVIL, a strong CXR VLM benchmark, in the zero-shot and linear probing settings. We conclude with a set of recommendations for researchers aiming to train vision-language models on other medical imaging modalities when training data is scarce. To facilitate further research, we will make our code and models publicly available.	inproceedings	Proceedings of Machine Learning Research	PMLR	2640-3498	windsor24a	0	Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime	53	73	53-73	53	false	Windsor, Rhydian and Jamaludin, Amir and Kadir, Timor and Zisserman, Andrew	given family Rhydian Windsor given family Amir Jamaludin given family Timor Kadir given family Andrew Zisserman	2024-01-23		Medical Imaging with Deep Learning	227	inproceedings	date-parts 2024 1 23	https://proceedings.mlr.press/v227/windsor24a/windsor24a.pdf