Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration

This repo contains the code for our NeurIPS 2021 paper, Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration [arxiv].

The ImageNet-16H dataset is available on the OSF. Please also see our work Bayesian Modeling of Human-AI Complementarity [Steyvers et al., 2022] describing this dataset in more detail.

Project Structure

• Data for the CIFAR-10h experiments is contained in the /cifar10h directory.
• /experiments contains various scripts for reproducing the experiments in our paper.
• calibrators.py implements various calibration methods.
• combination_methods.py implements various combination methods.
• data_utils.py contains useful data processing methods.

References

Please consider citing our paper as:

@inproceedings{kerrigan2021combining,
  title={Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration},
  author={Kerrigan, Gavin and Smyth, Padhraic and Steyvers, Mark},
  booktitle={Advances in Neural Information Processing Systems},
  year={2021}
}