CT-Bound: Robust Boundary Detection From Noisy Images Via Hybrid Convolution and Transformer Neural Networks

Wei Xu, Junjie Luo, and Qi Guo

Elmore Family School of Electrical and Computer Engineering, Purdue University

Contact: [email protected]

arXiv | Dataset & Pretrained weights | Real-time video processing demo

Content

• 0 Introduction
• 1 Usage
  • 1.1 Quick start
  • 1.2 Training
  • 1.3 Testing and evaluation
    • 1.3.1 MS COCO
    • 1.3.2 BSDS500 and NYUDv2
• 2 Results
• 3 Citation

0 Introduction

CT-Bound is a robust and fast boundary estimation method for noisy images using a hybrid Convolution and Transformer neural network. The proposed architecture decomposes boundary estimation into two tasks: local detection and global regularization of image boundaries.

1 Usage

1.1 Quick start

To run the code with conda, please follow the prompts below.

git clone https://github.com/guo-research-group/CT-Bound.git
cd CT-Bound
conda create -n ct_bound python=3.9
conda activate ct_bound
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip3 install -r requirements.txt

After downloading and unzipping the dataset and pretrained weights, the full folder content is shown below.

CT-Bound
│   requirements.txt
│   ct_bound.py
│   init_training.py
│   ref_training.py
│   ...
│
└───eval
│   │   eval_bsds.m
│   │   eval_nyud.m
│   │   ...
│
└───dataset
    │   best_ran_pretrained_init.pth
    │   best_ran_pretrained_ref.pth
    │
    └───initialization
    │   │   ...
    │
    └───refinement
    │   │   ...
    │
    └───eval
        │
        └───BSDS500
        │   │   ...
        │
        └───NYUDv2
            │   ...

1.2 Training

To train the initialization stage, run

python init_training.py

To train the refinement stage, run

python ref_training.py

1.3 Testing and evaluation

1.3.1 MS COCO

To investigate the performance of whole pipeline with the MS COCO testing set, run

python ct_bound.py

1.3.2 BSDS500 and NYUDv2

To evaluate with BSDS500 or NYUDv2 testing set on photo level of $n$ ($\alpha_{\text{test}}=n$), run

python ct_bound.py --data_path './dataset/eval/' --eval 'BSDS500' --eval_alpha n

or

python ct_bound.py --data_path './dataset/eval/' --eval 'NYUDv2' --eval_alpha n

To calculate the ODS F1-score, run

cd eval
matlab -nodisplay -nodesktop -nosplash -r eval_bsds.m

or

cd eval
matlab -nodisplay -nodesktop -nosplash -r eval_nyud.m

Note that if you want to calculate the metrics for color maps, add --metrics True parameter with the calling command above.

2 Results

The ODS F1-score can be found in the table below.

Dataset	Photon level \alpha
	2	4	6	8
BSDS500	0.541	0.627	0.640	0.633
NYUDv2	0.552	0.633	0.646	0.647

3 Citation

@article{xu2024ctbound,
  title={CT-Bound: Robust Boundary Detection From Noisy Images Via Hybrid Convolution and Transformer Neural Networks}, 
  author={Wei Xu and Junjie Luo and Qi Guo},
  journal={arXiv preprint arXiv:2403.16494},
  year={2024}
}

Some of the code is borrowed from FoJ.