Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement defocus deblurring. #469

Draft
wants to merge 54 commits into
base: main
Choose a base branch
from
Draft

Implement defocus deblurring. #469

wants to merge 54 commits into from

Conversation

jefequien
Copy link
Contributor

@jefequien jefequien commented Oct 28, 2024 •
edited
Loading

Loosely based on Deblurring 3D Gaussian Splatting, https://arxiv.org/pdf/2401.00834 and BAGS, https://arxiv.org/pdf/2403.04926.

Each step requires rendering both clear and blurry images so this method is >2x slower than normal 3DGS.

defocuscake_15k.mp4

Evaluation

Run bash examples/benchmarks/mcmc_deblur.sh

psnr ssim lpips num_GS
defocuscake 26.8007 0.891839 0.137933 250000
defocuscaps 24.3061 0.838382 0.201384 250000
defocuscisco 20.4781 0.861181 0.0854522 250000
defocuscoral 19.3734 0.719444 0.285026 250000
defocuscupcake 22.2543 0.82716 0.127639 250000
defocuscups 25.2897 0.875219 0.147801 250000
defocusdaisy 23.6329 0.837565 0.171997 250000
defocussausage 18.4746 0.817682 0.216327 250000
defocusseal 25.757 0.883999 0.121537 250000
defocustools 27.2291 0.933414 0.0903888 250000
AVERAGE 23.3596 0.848589 0.158548 250000

@jefequien jefequien marked this pull request as draft October 28, 2024 18:58
@ichsan2895
Copy link

How to use this method? What is the command?

@jefequien
Copy link
Contributor Author

Run bash examples/benchmarks/mcmc_deblur.sh. You may need to install https://github.com/NVlabs/tiny-cuda-nn to use the mlps. I'll post more detailed instructions soon.

@ichsan2895
Copy link

I tried with multi GPU support, but it did not work:

>> CUDA_VISIBLE_DEVICES=0,1 python3 examples/simple_trainer.py mcmc --disable_viewer --data_factor $DATA_FACTOR \
        --strategy.cap-max $CAP_MAX \
        --blur_opt \
        --blur_opt_lr 1e-3 \
        --blur_a 0.8 \
        --blur_mask_reg 0.002 \
        --render_traj_path $RENDER_TRAJ_PATH \
        --data_dir $SCENE_DIR/$SCENE/ \
        --result_dir $RESULT_DIR/$SCENE \
        --steps-scaler 0.5

  0%|          | 0/15000 [00:01<?, ?it/s]
  0%|          | 0/15000 [00:01<?, ?it/s]
Exception in thread Thread-2 (_pin_memory_loop):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/pin_memory.py", line 54, in _pin_memory_loop
    do_one_step()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/pin_memory.py", line 31, in do_one_step
    r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 122, in get
    return _ForkingPickler.loads(res)
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/reductions.py", line 355, in rebuild_storage_fd
    fd = df.detach()
  File "/usr/lib/python3.10/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/usr/lib/python3.10/multiprocessing/resource_sharer.py", line 86, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 508, in Client
    answer_challenge(c, authkey)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 752, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/workspace/GSPLAT_DEFOCUSJEF/gsplat/examples/simple_trainer.py", line 1193, in <module>
    cli(main, cfg, verbose=True)
  File "/workspace/GSPLAT_DEFOCUSJEF/gsplat/gsplat/distributed.py", line 344, in cli
    process_context.join()
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 163, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 74, in _wrap
    fn(i, *args)
  File "/workspace/GSPLAT_DEFOCUSJEF/gsplat/gsplat/distributed.py", line 295, in _distributed_worker
    fn(local_rank, world_rank, world_size, args)
  File "/workspace/GSPLAT_DEFOCUSJEF/gsplat/examples/simple_trainer.py", line 1138, in main
    runner.train()
  File "/workspace/GSPLAT_DEFOCUSJEF/gsplat/examples/simple_trainer.py", line 659, in train
    blur_mask = self.blur_module.predict_mask(image_ids, depths)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'DistributedDataParallel' object has no attribute 'predict_mask'

It worked well with single GPU setup. I will share the result later. Its training now.

@ichsan2895
Copy link

ichsan2895 commented Nov 2, 2024 •
edited
Loading

Some of result is coming :)

Top video = Gsplat MCMC with 500k splats & Data-Factor 4.
Bottom video = Gsplat MCMC with 500k splats & Data-Factor 4 with defocusing.

defocuscupcake_comparison_29999.mp4
defocusdaisy_comparison_29999.mp4
defocusseal_comparison_29999.mp4

It worked well with single GPU setup. Hopefully the multiGPU setup will be fixed.

@ichsan2895
Copy link

Gsplat MCMC with 500k splats & Data-Factor 4.

NUM SCENE PSNR SSIM LPIPS VRAM (GB) NUM_GS
0 defocuscake 24.716 0.832 0.207 0.714 500000
1 defocuscaps 22.442 0.744 0.285 0.751 500000
2 defocuscisco 20.571 0.848 0.119 0.805 500000
3 defocuscoral 19.649 0.750 0.292 0.741 500000
4 defocuscupcake 21.388 0.762 0.191 0.812 500000
5 defocuscups 24.274 0.827 0.218 0.744 500000
6 defocusdaisy 22.175 0.720 0.285 0.695 500000
7 defocussausage 17.324 0.729 0.302 0.896 500000
8 defocusseal 22.105 0.681 0.286 0.780 500000
9 defocustools 24.886 0.884 0.148 0.808 500000
10 AVERAGE 21.953 0.778 0.233 0.775 500000

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing.

NUM SCENE PSNR SSIM LPIPS VRAM (GB) NUM_GS
0 defocuscake 26.439 0.886 0.130 1.814 500000
1 defocuscaps 23.874 0.837 0.183 1.815 500000
2 defocuscisco 20.488 0.853 0.101 1.828 500000
3 defocuscoral 19.419 0.834 0.156 1.838 500000
4 defocuscupcake 22.529 0.850 0.090 1.847 500000
5 defocuscups 24.510 0.851 0.175 1.844 500000
6 defocusdaisy 23.659 0.856 0.123 1.806 500000
7 defocussausage 18.159 0.830 0.181 1.856 500000
8 defocusseal 25.041 0.865 0.129 1.797 500000
9 defocustools 27.869 0.947 0.057 1.841 500000
10 AVERAGE 23.199 0.861 0.133 1.829 500000

Tested with :
Ubuntu 22.04.3 LTS
Python 3.10.12
Torch 2.1.2+cu118
gsplat commit 8f7e92c

@jefequien
Copy link
Contributor Author

jefequien commented Nov 2, 2024 •
edited
Loading

Thanks for running this. I can get slightly higher metrics with different hyperparameters but I haven't been able to beat the 23.61 avg reported in the original paper. Also, if anyone has an out of focus real world dataset, I'd be curious to see how this performs.

@ichsan2895
Copy link

Thanks for running this. I can get slightly higher metrics with different hyperparameters but I haven't been able to beat the 23.61 avg reported in the original paper. Also, if anyone has an out of focus real world dataset, I'd be curious to see how this performs.

How about using higher num of splats such as 1 million?

@jefequien
Copy link
Contributor Author

jefequien commented Nov 2, 2024 •
edited
Loading

I found 250k and 500k get similar metrics. The original paper uses ~300k so it would be good to keep things fair.

@ichsan2895
Copy link

ichsan2895 commented Nov 2, 2024 •
edited
Loading

BTW SSIM is the way higher than the original paper. (0.861 vs 0.7471) 👍

@jefequien
Copy link
Contributor Author

jefequien commented Nov 2, 2024 •
edited
Loading

I looked into it briefly. The SSIM implementation is different so don't pay much attention to that.

@ichsan2895
Copy link

Somehow, it do not work good in real_motion_blur dataset.

Gsplat MCMC with 500k splats & Data-Factor 4

NUM SCENE PSNR SSIM LPIPS VRAM (GB) NUM_GS
0 blurball 23.112 0.643 0.322 0.767 500000
1 blurbasket 23.342 0.767 0.260 0.732 500000
2 blurbuick 20.915 0.700 0.263 0.734 500000
3 blurcoffee 25.657 0.857 0.217 0.786 500000
4 blurdecoration 21.951 0.716 0.274 0.768 500000
5 blurgirl 19.622 0.774 0.258 0.744 500000
6 blurheron 20.090 0.558 0.317 0.711 500000
7 blurparterre 12.799 0.299 0.661 0.787 500000
8 blurpuppet 21.687 0.703 0.273 0.751 500000
9 blurstair 22.600 0.613 0.346 0.844 500000
10 AVERAGE 21.178 0.663 0.319 0.762 500000

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing.

NUM SCENE PSNR SSIM LPIPS VRAM (GB) NUM_GS
0 blurball 23.211 0.646 0.316 1.836 500000
1 blurbasket 24.216 0.794 0.236 1.828 500000
2 blurbuick 22.053 0.726 0.231 1.830 500000
3 blurcoffee 26.553 0.866 0.198 1.880 500000
4 blurdecoration 23.267 0.760 0.242 1.854 500000
5 blurgirl 19.707 0.788 0.232 1.840 500000
6 blurheron 20.664 0.598 0.294 1.816 500000
7 blurparterre 14.115 0.302 0.631 1.846 500000
8 blurpuppet 22.475 0.733 0.242 1.835 500000
9 blurstair 22.766 0.647 0.311 1.881 500000
10 AVERAGE 21.903 0.686 0.293 1.845 500000

How about other real world dataset?

I tested a dataset called dozer from nerfstudio-download-data. The metrics is low since it has more sharper image than original image!

0001_dozer_val_step29999_defocussing

@ichsan2895
Copy link

Seems an interesting update. Hopefully I can benchmark it with previous dataset again.

@ichsan2895
Copy link

Here is it my benchmark. Hopefully it will helpful for you @jefequien

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing commit 8f7e92c.

CUDA_VISIBLE_DEVICES=0 python3 examples/simple_trainer.py mcmc \
        --disable_viewer --data_factor $DATA_FACTOR \
        --strategy.cap-max $CAP_MAX \
        --blur_opt \
        --blur_opt_lr 1e-3 \
        --blur_a 0.8 \
        --blur_mask_reg 0.002 \
        --render_traj_path $RENDER_TRAJ_PATH \
        --data_dir $SCENE_DIR/$SCENE/ \
        --result_dir $RESULT_DIR/$SCENE

real deblur dataset

NUM SCENE PSNR SSIM LPIPS VRAM (GB) NUM_GS
0 defocuscake 26.439 0.886 0.130 1.814 500000
1 defocuscaps 23.874 0.837 0.183 1.815 500000
2 defocuscisco 20.488 0.853 0.101 1.828 500000
3 defocuscoral 19.419 0.834 0.156 1.838 500000
4 defocuscupcake 22.529 0.850 0.090 1.847 500000
5 defocuscups 24.510 0.851 0.175 1.844 500000
6 defocusdaisy 23.659 0.856 0.123 1.806 500000
7 defocussausage 18.159 0.830 0.181 1.856 500000
8 defocusseal 25.041 0.865 0.129 1.797 500000
9 defocustools 27.869 0.947 0.057 1.841 500000
10 AVERAGE 23.199 0.861 0.133 1.829 500000

real motion blur dataset

NUM SCENE PSNR SSIM LPIPS VRAM (GB) NUM_GS
0 blurball 23.211 0.646 0.316 1.836 500000
1 blurbasket 24.216 0.794 0.236 1.828 500000
2 blurbuick 22.053 0.726 0.231 1.830 500000
3 blurcoffee 26.553 0.866 0.198 1.880 500000
4 blurdecoration 23.267 0.760 0.242 1.854 500000
5 blurgirl 19.707 0.788 0.232 1.840 500000
6 blurheron 20.664 0.598 0.294 1.816 500000
7 blurparterre 14.115 0.302 0.631 1.846 500000
8 blurpuppet 22.475 0.733 0.242 1.835 500000
9 blurstair 22.766 0.647 0.311 1.881 500000
10 AVERAGE 21.903 0.686 0.293 1.845 500000

Gsplat MCMC with 500k splats & Data-Factor 4 with Defocusing commit d9db9e0

CUDA_VISIBLE_DEVICES=0 python3 examples/simple_trainer.py mcmc \
        --disable_viewer --data_factor $DATA_FACTOR \
        --strategy.cap-max $CAP_MAX \
        --blur_opt \
        --render_traj_path $RENDER_TRAJ_PATH \
        --data_dir $SCENE_DIR/$SCENE/ \
        --result_dir $RESULT_DIR/$SCENE

real deblur dataset

NUM SCENE PSNR SSIM LPIPS VRAM (GB) NUM_GS
0 defocuscake 26.138 0.869 0.157 2.491 500000
1 defocuscaps 23.579 0.829 0.194 2.506 500000
2 defocuscisco 20.496 0.860 0.084 2.537 500000
3 defocuscoral 19.670 0.764 0.248 2.509 500000
4 defocuscupcake 22.490 0.832 0.117 2.562 500000
5 defocuscups 25.439 0.876 0.145 2.560 500000
6 defocusdaisy 23.366 0.846 0.138 2.454 500000
7 defocussausage 18.232 0.823 0.192 2.570 500000
8 defocusseal 25.527 0.879 0.122 2.467 500000
9 defocustools 27.717 0.944 0.063 2.548 500000
10 AVERAGE 23.265 0.852 0.146 2.520 500000

real motion blur dataset

NUM SCENE PSNR SSIM LPIPS VRAM (GB) NUM_GS
0 blurball 23.219 0.649 0.316 2.531 500000
1 blurBasket 23.731 0.783 0.236 2.518 500000
2 blurBuick 22.016 0.725 0.238 2.537 500000
3 blurCoffee 26.288 0.862 0.194 2.549 500000
4 blurDecoration 22.716 0.735 0.265 2.577 500000
5 blurGirl 18.615 0.772 0.250 2.540 500000
6 blurHeron 20.751 0.602 0.284 2.500 500000
7 blurParterre 22.894 0.648 0.303 2.483 500000
8 blurPuppet 21.597 0.711 0.265 2.553 500000
9 blurStair 22.791 0.619 0.336 2.646 500000
10 AVERAGE 22.462 0.711 0.269 2.543 500000

@ichsan2895
Copy link

Top : commit 8f7e92c
Bottom : commit d9db9e0

defocuscake_defocus1-vs-defocus2_traj_val_29999.mp4
defocuscoral_comparison_deblur1-vs-2_29999.mp4
defocusseal_comparison_deblur1-vs-2_29999.mp4

@jefequien jefequien mentioned this pull request Dec 6, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jefequien @ichsan2895