Authors: Sixiang Chen, Tian Ye, Yun Liu, Erkang Chen
SnowFormer: Context Interaction Transformer with Scale-awareness for Single Image Desnowing
Abstract: Due to various and complicated snow degradations, single image desnowing is a challenging image restoration task. As prior arts can not handle it ideally, we propose a novel transformer, SnowFormer, which explores efficient cross-attentions to build local-global context interaction across patches and surpasses existing works that employ local operators or vanilla transformers. Compared to prior desnowing methods and universal image restoration methods, SnowFormer has several benefits. Firstly, unlike the multi-head self-attention in recent image restoration Vision Transformers, SnowFormer incorporates the multi-head cross-attention mechanism to perform local-global context interaction between scale-aware snow queries and local-patch embeddings. Second, the snow queries in SnowFormer are generated by the query generator from aggregated scale-aware features, which are rich in potential clean cues, leading to superior restoration results. Third, SnowFormer outshines advanced state-of-the-art desnowing networks and the prevalent universal image restoration transformers on six synthetic and real-world datasets.
- November 22, 2022: Checkpoint of CSD is updated. 🔥
|
|
|
Our SnowFormer is built in Pytorch1.12.0, we train and test it ion Ubuntu20.04 environment (Python3.8, Cuda11.6).
For installing, please follow these intructions.
conda create -n py38 python=3.8
conda activate py38
conda install pytorch=1.12
pip install opencv-python tqdm tensorboardX ....
We sample the 2000 images from every desnowing dataset for the test stage, including CSD, SRRS, Snow100K, SnowCityScapes and SnowKITTI. We provide the download link of datasets, and the password is ephe.
| Dataset | CSD | SRRS | Snow100K | SnowCityScapes | SnowKITTI |
|---|---|---|---|---|---|
| Link | Download | Download | Download | Download | Download |
You can download pre-trained model of CSD from Pre-trained model Password:ephe and save it in model_path.
Run the following commands:
python3 test.py --dataset_type CSD --dataset_CSD dataset_CSD --model_path model_pathThe rusults are saved in ./out/dataset_type/
- Checkpoints of SRRS, Snow100K, SnowCityScapes and SnowKITTI
- Train.py
@article{chen2022snowformer,
title={SnowFormer: Scale-aware Transformer via Context Interaction for Single Image Desnowing},
author={Chen, Sixiang and Ye, Tian and Liu, Yun and Chen, Erkang and Shi, Jun and Zhou, Jingchun},
journal={arXiv preprint arXiv:2208.09703},
year={2022}
}If you have any questions, please contact the email [email protected], [email protected] or [email protected].