SnowFormer: Context Interaction Transformer with Scale-awareness for Single Image Desnowing

Authors: Sixiang Chen, Tian Ye, Yun Liu, Erkang Chen

SnowFormer: Context Interaction Transformer with Scale-awareness for Single Image Desnowing

Abstract: Due to various and complicated snow degradations, single image desnowing is a challenging image restoration task. As prior arts can not handle it ideally, we propose a novel transformer, SnowFormer, which explores efficient cross-attentions to build local-global context interaction across patches and surpasses existing works that employ local operators or vanilla transformers. Compared to prior desnowing methods and universal image restoration methods, SnowFormer has several benefits. Firstly, unlike the multi-head self-attention in recent image restoration Vision Transformers, SnowFormer incorporates the multi-head cross-attention mechanism to perform local-global context interaction between scale-aware snow queries and local-patch embeddings. Second, the snow queries in SnowFormer are generated by the query generator from aggregated scale-aware features, which are rich in potential clean cues, leading to superior restoration results. Third, SnowFormer outshines advanced state-of-the-art desnowing networks and the prevalent universal image restoration transformers on six synthetic and real-world datasets.

News

• November 22, 2022: Checkpoint of CSD is updated. 🔥

Network Architecture

Installation

Our SnowFormer is built in Pytorch1.12.0, we train and test it ion Ubuntu20.04 environment (Python3.8, Cuda11.6).

For installing, please follow these intructions.

conda create -n py38 python=3.8
conda activate py38
conda install pytorch=1.12 
pip install opencv-python tqdm tensorboardX ....

Dataset

We sample the 2000 images from every desnowing dataset for the test stage, including CSD, SRRS, Snow100K, SnowCityScapes and SnowKITTI. We provide the download link of datasets, and the password is ephe.

Dataset	CSD	SRRS	Snow100K	SnowCityScapes	SnowKITTI
Link	Download	Download	Download	Download	Download

Testing Stage

You can download pre-trained model of CSD from Pre-trained model   Password:ephe and save it in model_path.

Run the following commands:

python3  test.py --dataset_type CSD --dataset_CSD dataset_CSD --model_path model_path

The rusults are saved in ./out/dataset_type/

TODO List

• Checkpoints of SRRS, Snow100K, SnowCityScapes and SnowKITTI
• Train.py

Citation

@article{chen2022snowformer,
  title={SnowFormer: Scale-aware Transformer via Context Interaction for Single Image Desnowing},
  author={Chen, Sixiang and Ye, Tian and Liu, Yun and Chen, Erkang and Shi, Jun and Zhou, Jingchun},
  journal={arXiv preprint arXiv:2208.09703},
  year={2022}
}

Contact

If you have any questions, please contact the email [email protected], [email protected] or [email protected].