ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real World

Website   |   📄 Paper   |   Apply for data access

ClinicalLab, a comprehensive clinical diagnosis agent alignment suite. ClinicalLab includes ClinicalBench, an end-to-end multi-departmental clinical diagnostic evaluation benchmark for evaluating medical agents and LLMs. ClinicalBench is based on real cases that cover 24 departments and 150 diseases. ClinicalLab also includes four novel metrics (ClinicalMetrics) for evaluating the effectiveness of LLMs in clinical diagnostic tasks. ClinicalAgent, an end-to-end clinical agent that aligns with real-world clinical diagnostic practices. Our findings demonstrate the importance of aligning with modern medical practices in designing medical agents.

🌈 Update

• [2024.07.15] ClinicalBench has completed 30 distributions. Please ⭐ our project before requesting data! Thanks.
• [2024.06.19] 🎉🎉🎉 ClinicalLab is published！🎉🎉🎉

License

• ClinicalBench is released under the ClinicalBench Usage and Data Distribution License Agreement. Please be sure to comply with the terms of use.
• The remaining code is released under the Apache-2.0 license. Please be sure to comply with the terms of use.

Datasets

In the appendix of our paper and the data_examples folder, we present examples from the ClinicalBench dataset, including both Chinese and English versions. Please note that accessing the ClinicalBench dataset requires an application. If you wish to access the full dataset, please read the licensing documentation and submit an access request. We will send the data to your specified email address within 48 hours.

For more information about the dataset, please refer to the DATA CARD.

Dependencies

You can install everything all at once

conda create -n clinicallab python=3.9 -y
pip install -r requirements.txt

We also provide an environment.yaml file for your use.

conda env create -f environment.yaml

Code

Inference

python code/inference/eval.py --model_name your_model_name --model_path your_model_path --api_key your_api_key --data_load_name data_examples/data_example_en.json

Using GPT-4 as an example

python code/inference/eval.py --model_name gpt4 --api_key your_api_key --data_load_name data_examples/data_example_en.json

Using internlm2chat as an example

python code/inference/eval.py --model_name internlm2chat --model_path your_model_path --data_load_name data_examples/data_example_en.json

Evaluation

Coming soon...

Citation

Please cite the paper if you use the data or code from ClinicalLab.

@article{yan2024clinicallab,
  title={ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real World},
  author={Yan, Weixiang and Liu, Haitian and Wu, Tengxiao and Chen, Qian and Wang, Wen and Chai, Haoyuan and Wang, Jiayi and Zhao, Weishan and Zhang, Yixin and Zhang, Renjun and others},
  journal={arXiv preprint arXiv:2406.13890},
  year={2024}
}

Contact

For questions, please feel free to reach out via email at [email protected].