DynRefer: Delving into Region-level Multi-modality Tasks via Dynamic Resolution

This is the official implementaion of paper 'DynRefer: Delving into Region-level Multi-modality Tasks via Dynamic Resolution'. This repository contains Pytorch training code, evaluation code.

1. Contents

• DynRefer: Delving into Region-level Multi-modality Tasks via Dynamic Resolution
  • 1. Contents
  • 2. Todo
  • 3. Introduction
  • 4. Results
  • 5. Code Usage
  • 6. Contacts
  • 7. Acknowledgment
  • 8. Citation

2. Todo

• Release training and evaluation code
• Release demo code

3. Introduction

Region-level multi-modality methods can translate referred image regions to human preferred language descriptions. Unfortunately, most of existing methods using fixed visual inputs remain lacking the resolution adaptability to find out precise language descriptions. In this study, we propose a dynamic resolution approach, referred to as DynRefer, to pursue high-accuracy region-level referring through mimicking the resolution adaptability of human visual cognition. DynRefer first implements stochastic vision-language alignment. It aligns desired language descriptions of multi-modality tasks with images of stochastic resolution, which are constructed by nesting a set of views around the referred region. DynRefer then implements dynamic multi-modality referring, which is realized by selecting views based on image and language priors. This allows the visual information used for referring to better match human preferences, thereby improving the representational adaptability of region-level multi-modality models. Extensive experiments show that DynRefer brings mutual improvement upon tasks including region-level captioning, open-vocabulary region recognition and attribute detection. Last but not least, DynRefer achieves new state-of-the-art on multiple region-level multi-modality tasks using a single model.

4. Results

5. Code Usage

• Environment

• Dataset and Checkpoints

• Training and Evaluation

6. Contacts

If you have any question about our work or this repository, please don't hesitate to contact us by emails or open an issue under this project.

• [email protected]
• [email protected]
• [email protected]

7. Acknowledgment

• Part of the code is borrowed from LAVIS, GlaMM, Osprey, RAM, and OVAD, we sincerely thank them for their contributions to the community.

8. Citation

@misc{zhao2024dynrefer,
      title={DynRefer: Delving into Region-level Multi-modality Tasks via Dynamic Resolution}, 
      author={Yuzhong Zhao and Feng Liu and Yue Liu and Mingxiang Liao and Chen Gong and Qixiang Ye and Fang Wan},
      year={2024},
      eprint={2405.16071},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

@misc{zhao2024controlcap,
      title={ControlCap: Controllable Region-level Captioning}, 
      author={Yuzhong Zhao and Yue Liu and Zonghao Guo and Weijia Wu and Chen Gong and Fang Wan and Qixiang Ye},
      year={2024},
      eprint={2401.17910},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}