Skip to content

Commit

Permalink
Merge branch 'main' into main
Browse files Browse the repository at this point in the history
  • Loading branch information
sarlinpe authored Oct 23, 2023
2 parents ea56b73 + 43cf81a commit c4b7cbc
Show file tree
Hide file tree
Showing 38 changed files with 898 additions and 282 deletions.
30 changes: 30 additions & 0 deletions .github/workflows/python-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: Python Tests
on:
push:
branches:
- main
pull_request:
types: [ assigned, opened, synchronize, reopened ]
jobs:
build:
name: Run Python Tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip'
- name: Install dependencies
run: |
sudo apt-get remove libunwind-14-dev || true
sudo apt-get install -y libceres-dev libeigen3-dev
python -m pip install --upgrade pip
python -m pip install pytest pytest-cov
python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
python -m pip install -e .[dev]
python -m pip install -e .[extra]
- name: Test with pytest
run: |
set -o pipefail
pytest --junitxml=pytest.xml --cov=gluefactory tests/
31 changes: 17 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ python3 -m pip install -e .[extra]
All models and datasets in gluefactory have auto-downloaders, so you can get started right away!

## License
The code and trained models in Glue Factory are released with an Apache-2.0 license. This includes LightGlue trained with an [open version of SuperPoint](https://github.com/rpautrat/SuperPoint). Third-party models that are not compatible with this license, such as SuperPoint (original) and SuperGlue, are provided in `gluefactory_nonfree`, where each model might follow its own, restrictive license.
The code and trained models in Glue Factory are released with an Apache-2.0 license. This includes LightGlue and an [open version of SuperPoint](https://github.com/rpautrat/SuperPoint). Third-party models that are not compatible with this license, such as SuperPoint (original) and SuperGlue, are provided in `gluefactory_nonfree`, where each model might follow its own, restrictive license.

## Evaluation

Expand Down Expand Up @@ -66,8 +66,8 @@ Here are the results as Area Under the Curve (AUC) of the homography error at 1

| Methods | DLT | [OpenCV](../gluefactory/robust_estimators/homography/opencv.py) | [PoseLib](../gluefactory/robust_estimators/homography/poselib.py) |
| ------------------------------------------------------------ | ------------------ | ------------------ | ------------------ |
| [SuperPoint + SuperGlue](../gluefactory/configs/superpoint+superglue.yaml) | 32.1 / 65.0 / 75.7 | 32.9 / 55.7 / 68.0 | 37.0 / 68.2 / 78.7 |
| [SuperPoint + LightGlue](../gluefactory/configs/superpoint+lightglue.yaml) | 35.1 / 67.2 / 77.6 | 34.2 / 57.9 / 69.9 | 37.1 / 67.4 / 77.8 |
| [SuperPoint + SuperGlue](gluefactory/configs/superpoint+superglue-official.yaml) | 32.1 / 65.0 / 75.7 | 32.9 / 55.7 / 68.0 | 37.0 / 68.2 / 78.7 |
| [SuperPoint + LightGlue](gluefactory/configs/superpoint+lightglue-official.yaml) | 35.1 / 67.2 / 77.6 | 34.2 / 57.9 / 69.9 | 37.1 / 67.4 / 77.8 |


</details>
Expand Down Expand Up @@ -159,9 +159,12 @@ Here are the results as Area Under the Curve (AUC) of the pose error at 5/10/20

| Methods | [pycolmap](../gluefactory/robust_estimators/relative_pose/pycolmap.py) | [OpenCV](../gluefactory/robust_estimators/relative_pose/opencv.py) | [PoseLib](../gluefactory/robust_estimators/relative_pose/poselib.py) |
| ------------------------------------------------------------ | ------------------ | ------------------ | ------------------ |
| [SuperPoint + SuperGlue](../gluefactory/configs/superpoint+superglue.yaml) | 54.4 / 70.4 / 82.4 | 48.7 / 65.6 / 79.0 | 64.8 / 77.9 / 87.0 |
| [SuperPoint + LightGlue](../gluefactory/configs/superpoint+lightglue.yaml) | 56.7 / 72.4 / 83.7 | 51.0 / 68.1 / 80.7 | 66.8 / 79.3 / 87.9 |
| [SuperPoint + GlueStick](../gluefactory/configs/superpoint+lsd+gluestick.yaml) | 53.2 / 69.8 / 81.9 | 46.3 / 64.2 / 78.1 | 64.4 / 77.5 / 86.5 |
| [SuperPoint + SuperGlue](gluefactory/configs/superpoint+superglue-official.yaml) | 54.4 / 70.4 / 82.4 | 48.7 / 65.6 / 79.0 | 64.8 / 77.9 / 87.0 |
| [SuperPoint + LightGlue](gluefactory/configs/superpoint+lightglue-official.yaml) | 56.7 / 72.4 / 83.7 | 51.0 / 68.1 / 80.7 | 66.8 / 79.3 / 87.9 |
| [SIFT (2K) + LightGlue](gluefactory/configs/sift+lightglue-official.yaml) | ? / ? / ? | 43.5 / 61.5 / 75.9 | 60.4 / 74.3 / 84.5 |
| [SIFT (4K) + LightGlue](gluefactory/configs/sift+lightglue-official.yaml) | ? / ? / ? | 49.9 / 67.3 / 80.3 | 65.9 / 78.6 / 87.4 |
| [ALIKED + LightGlue](gluefactory/configs/aliked+lightglue-official.yaml) | ? / ? / ? | 51.5 / 68.1 / 80.4 | 66.3 / 78.7 / 87.5 |
| [SuperPoint + GlueStick](gluefactory/configs/superpoint+lsd+gluestick.yaml) | 53.2 / 69.8 / 81.9 | 46.3 / 64.2 / 78.1 | 64.4 / 77.5 / 86.5 |

</details>

Expand Down Expand Up @@ -223,36 +226,36 @@ All training commands automatically download the datasets.
<details>
<summary>[Training LightGlue]</summary>

We show how to train LightGlue with [SuperPoint open](https://github.com/rpautrat/SuperPoint).
We show how to train LightGlue with [SuperPoint](https://github.com/magicleap/SuperPointPretrainedNetwork).
We first pre-train LightGlue on the homography dataset:
```bash
python -m gluefactory.train sp+lg_homography \ # experiment name
--conf gluefactory/configs/superpoint-open+lightglue_homography.yaml
--conf gluefactory/configs/superpoint+lightglue_homography.yaml
```
Feel free to use any other experiment name. By default the checkpoints are written to `outputs/training/`. The default batch size of 128 corresponds to the results reported in the paper and requires 2x 3090 GPUs with 24GB of VRAM each as well as PyTorch >= 2.0 (FlashAttention).
Configurations are managed by [OmegaConf](https://omegaconf.readthedocs.io/) so any entry can be overridden from the command line.
If you have PyTorch < 2.0 or weaker GPUs, you may thus need to reduce the batch size via:
```bash
python -m gluefactory.train sp+lg_homography \
--conf gluefactory/configs/superpoint-open+lightglue_homography.yaml \
--conf gluefactory/configs/superpoint+lightglue_homography.yaml \
data.batch_size=32 # for 1x 1080 GPU
```
Be aware that this can impact the overall performance. You might need to adjust the learning rate accordingly.

We then fine-tune the model on the MegaDepth dataset:
```bash
python -m gluefactory.train sp+lg_megadepth \
--conf gluefactory/configs/superpoint-open+lightglue_megadepth.yaml \
--conf gluefactory/configs/superpoint+lightglue_megadepth.yaml \
train.load_experiment=sp+lg_homography
```

Here the default batch size is 32. To speed up training on MegaDepth, we suggest to cache the local features before training (requires around 150 GB of disk space):
```bash
# extract features
python -m gluefactory.scripts.export_megadepth --method sp_open --num_workers 8
python -m gluefactory.scripts.export_megadepth --method sp --num_workers 8
# run training with cached features
python -m gluefactory.train sp+lg_megadepth \
--conf gluefactory/configs/superpoint-open+lightglue_megadepth.yaml \
--conf gluefactory/configs/superpoint+lightglue_megadepth.yaml \
train.load_experiment=sp+lg_homography \
data.load_features.do=True
```
Expand Down Expand Up @@ -297,10 +300,10 @@ Using the following local feature extractors:
| Model | LightGlue config |
| --------- | --------- |
| [SuperPoint (open)](https://github.com/rpautrat/SuperPoint) | `superpoint-open+lightglue_{homography,megadepth}.yaml` |
| [SuperPoint (official)](https://github.com/magicleap/SuperPointPretrainedNetwork) | ❌ TODO |
| [SuperPoint (official)](https://github.com/magicleap/SuperPointPretrainedNetwork) | `superpoint+lightglue_{homography,megadepth}.yaml` |
| SIFT (via [pycolmap](https://github.com/colmap/pycolmap)) | `sift+lightglue_{homography,megadepth}.yaml` |
| [ALIKED](https://github.com/Shiaoming/ALIKED) | `aliked+lightglue_{homography,megadepth}.yaml` |
| [DISK](https://github.com/cvlab-epfl/disk) | ❌ TODO |
| [DISK](https://github.com/cvlab-epfl/disk) | `disk+lightglue_{homography,megadepth}.yaml` |
| Key.Net + HardNet | ❌ TODO |

## Coming soon
Expand Down
Binary file added assets/boat1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/boat2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
28 changes: 28 additions & 0 deletions gluefactory/configs/aliked+lightglue-official.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
model:
name: two_view_pipeline
extractor:
name: extractors.aliked
max_num_keypoints: 2048
detection_threshold: 0.0
matcher:
name: matchers.lightglue_pretrained
features: aliked
depth_confidence: -1
width_confidence: -1
filter_threshold: 0.1
benchmarks:
megadepth1500:
data:
preprocessing:
side: long
resize: 1600
eval:
estimator: opencv
ransac_th: 0.5
hpatches:
eval:
estimator: opencv
ransac_th: 0.5
model:
extractor:
max_num_keypoints: 1024 # overwrite config above
47 changes: 47 additions & 0 deletions gluefactory/configs/disk+lightglue_homography.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
data:
name: homographies
data_dir: revisitop1m
train_size: 150000
val_size: 2000
batch_size: 128
num_workers: 14
homography:
difficulty: 0.7
max_angle: 45
photometric:
name: lg
model:
name: two_view_pipeline
extractor:
name: extractors.disk_kornia
max_num_keypoints: 512
force_num_keypoints: True
detection_threshold: 0.0
trainable: False
ground_truth:
name: matchers.homography_matcher
th_positive: 3
th_negative: 3
matcher:
name: matchers.lightglue
filter_threshold: 0.1
input_dim: 128
flash: false
checkpointed: true
train:
seed: 0
epochs: 40
log_every_iter: 100
eval_every_iter: 500
lr: 1e-4
lr_schedule:
start: 20
type: exp
on_epoch: true
exp_div_10: 10
plot: [5, 'gluefactory.visualization.visualize_batch.make_match_figures']
benchmarks:
hpatches:
eval:
estimator: opencv
ransac_th: 0.5
70 changes: 70 additions & 0 deletions gluefactory/configs/disk+lightglue_megadepth.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
data:
name: megadepth
preprocessing:
resize: 1024
side: long
square_pad: True
train_split: train_scenes_clean.txt
train_num_per_scene: 300
val_split: valid_scenes_clean.txt
val_pairs: valid_pairs.txt
min_overlap: 0.1
max_overlap: 0.7
num_overlap_bins: 3
read_depth: true
read_image: true
batch_size: 32
num_workers: 14
load_features:
do: false # enable this if you have cached predictions
path: exports/megadepth-undist-depth-r1024_DISK-k2048-nms5/{scene}.h5
padding_length: 2048
padding_fn: pad_local_features
model:
name: two_view_pipeline
extractor:
name: extractors.disk_kornia
max_num_keypoints: 512
force_num_keypoints: True
detection_threshold: 0.0
trainable: False
ground_truth:
name: matchers.homography_matcher
th_positive: 3
th_negative: 3
matcher:
name: matchers.lightglue
filter_threshold: 0.1
input_dim: 128
flash: false
checkpointed: true
allow_no_extract: True
train:
seed: 0
epochs: 50
log_every_iter: 100
eval_every_iter: 1000
lr: 1e-4
lr_schedule:
start: 30
type: exp
on_epoch: true
exp_div_10: 10
dataset_callback_fn: sample_new_items
plot: [5, 'gluefactory.visualization.visualize_batch.make_match_figures']
benchmarks:
megadepth1500:
data:
preprocessing:
side: long
resize: 1024
eval:
estimator: opencv
ransac_th: 0.5
hpatches:
eval:
estimator: opencv
ransac_th: 0.5
model:
extractor:
max_num_keypoints: 1024
28 changes: 28 additions & 0 deletions gluefactory/configs/sift+lightglue-official.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
model:
name: two_view_pipeline
extractor:
name: extractors.sift
backend: pycolmap_cuda
max_num_keypoints: 4096
matcher:
name: matchers.lightglue_pretrained
features: sift
depth_confidence: -1
width_confidence: -1
filter_threshold: 0.1
benchmarks:
megadepth1500:
data:
preprocessing:
side: long
resize: 1600
eval:
estimator: opencv
ransac_th: 0.5
hpatches:
eval:
estimator: opencv
ransac_th: 0.5
model:
extractor:
max_num_keypoints: 1024 # overwrite config above
7 changes: 5 additions & 2 deletions gluefactory/configs/sift+lightglue_homography.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ model:
name: two_view_pipeline
extractor:
name: extractors.sift
detector: pycolmap_cuda
backend: pycolmap_cuda
max_num_keypoints: 1024
force_num_keypoints: True
detection_threshold: 0.0001
nms_radius: 3
trainable: False
ground_truth:
name: matchers.homography_matcher
Expand Down Expand Up @@ -46,3 +46,6 @@ benchmarks:
eval:
estimator: opencv
ransac_th: 0.5
model:
extractor:
nms_radius: 0
8 changes: 6 additions & 2 deletions gluefactory/configs/sift+lightglue_megadepth.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,10 @@ model:
name: two_view_pipeline
extractor:
name: extractors.sift
detector: pycolmap_cuda
backend: pycolmap_cuda
max_num_keypoints: 2048
force_num_keypoints: True
detection_threshold: 0.0001
nms_radius: 3
trainable: False
matcher:
name: matchers.lightglue
Expand Down Expand Up @@ -62,6 +62,9 @@ benchmarks:
preprocessing:
side: long
resize: 1600
model:
extractor:
nms_radius: 0
eval:
estimator: opencv
ransac_th: 0.5
Expand All @@ -72,3 +75,4 @@ benchmarks:
model:
extractor:
max_num_keypoints: 1024
nms_radius: 0
47 changes: 47 additions & 0 deletions gluefactory/configs/superpoint+lightglue_homography.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
data:
name: homographies
data_dir: revisitop1m
train_size: 150000
val_size: 2000
batch_size: 128
num_workers: 14
homography:
difficulty: 0.7
max_angle: 45
photometric:
name: lg
model:
name: two_view_pipeline
extractor:
name: gluefactory_nonfree.superpoint
max_num_keypoints: 512
force_num_keypoints: True
detection_threshold: 0.0
nms_radius: 3
trainable: False
ground_truth:
name: matchers.homography_matcher
th_positive: 3
th_negative: 3
matcher:
name: matchers.lightglue
filter_threshold: 0.1
flash: false
checkpointed: true
train:
seed: 0
epochs: 40
log_every_iter: 100
eval_every_iter: 500
lr: 1e-4
lr_schedule:
start: 20
type: exp
on_epoch: true
exp_div_10: 10
plot: [5, 'gluefactory.visualization.visualize_batch.make_match_figures']
benchmarks:
hpatches:
eval:
estimator: opencv
ransac_th: 0.5
Loading

0 comments on commit c4b7cbc

Please sign in to comment.