From 09d9c73b41d0744bb75b62e4535679edac4203d5 Mon Sep 17 00:00:00 2001 From: Valery-Dewil Date: Fri, 30 Aug 2024 19:03:21 +0200 Subject: [PATCH] first commit --- index.html | 509 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 508 insertions(+), 1 deletion(-) diff --git a/index.html b/index.html index 557db03..4fbd00a 100644 --- a/index.html +++ b/index.html @@ -1 +1,508 @@ -Hello World + + + + + + + + Adapting MIMO video restoration networks to low latency constraints + + + + + + + + + + + + +
+
+ Adapting MIMO video restoration networks to low latency constraints (TODO, first commit right now)
+ +
+ + + + + + + + + + +
+
+ Valéry Dewil +
+ 		+
+ Zhe Zheng +
+ 		+
+ Arnaud Barral +
+ 		+
+ Lara Raad +
+ 		+
+ Nao Nicolas +
+ 		+
+ Ioannis Cassagne +
+ 		+
+ Jean-Michel Morel +
+ 		+
+ Gabriele Facciolo +
+ 		+
+ Pablo
Arias +
+
+ + + + + + +
+
+ [Paper] +
+ 		+
+ [GitHub]
+
+ 		+
+ [Dataset]
+
+
+ +
+
+
+ + + + +
+
+ +
+
+ + + + +
+ L1BSR produces a 5m high-resolution (HR) output with all bands correctly registered from a single 10m low-resolution (LR) Sentinel-2 L1B image with misaligned bands. Note that our method is trained on real data with self-supervision, i.e. without any ground truth HR targets. +
+ + + + +
+ Project developed at the ENS Paris-Saclay, Centre Borelli and accepted at EarthVision 2023.
+ +
+
+ +
+ + +

Abstract

+ + + +
+MIMO (multiple input, multiple output) approaches are a recent trend in neural network architectures for video restoration problems, where each network evaluation produces multiple output frames. The video is split into non-overlapping stacks of frames that are processed independently, resulting in a very appealing trade-off between output quality and computational cost. In this work we focus on the low-latency setting by limiting the number of available future frames. We find that MIMO architectures suffer from problems that have received little attention so far, namely (1) the performance drops significantly due to the reduced temporal receptive field, particularly for frames at the boundaries of the stack, (2) there are strong temporal discontinuities at stack transitions which induce a step-wise motion artifact. We propose two simple solutions to alleviate these problems: recurrence across MIMO stacks to boost the output quality by implicitly increasing the temporal receptive field, and overlapping of the output stacks to smooth the temporal discontinuity at stack transitions. These modifications can be applied to any MIMO architecture. We test them on three state-of-the-art video denoising networks with different computational cost. The proposed contributions result in a new state-of-the-art for low-latency networks, both in terms of reconstruction error and temporal consistency. As an additional contribution, we introduce a new benchmark consisting of drone footage that highlights temporal consistency issues that are not apparent in the standard benchmarks. +
+
+ + +
+ +

Proposed framework

+ + +
+
+ + + +
+
+ + + + + + +
+
+
+ +
+
+ + + +
+ Super-Resolution framework. Overview of our proposed self-supervised L1BSR framework for Sentinel-2 L1B at training time. Note that at inference time, only one input and the reconstruction module are required. +
+ + + + + + +
+
+
+ +
+
+ + + +
+ Cross-Spectral Registration framework. Training setup of our proposed cross-spectral registration (CSR) module. +
+ +
+
+  [Try our code] [Try our demo] + +
+
+
+
+ +

Sentinel-2 sensor layout and L1BSR dataset

+ + + + + + + +
+
+
+ +
+
+ + + +
+ Sensor layout of the Sentinel-2 MSI. The Sentinel-2 MSI carries 12 CMOS detectors for the VNIR bands, with adjacent detectors having overlapping fields of view that result in overlapping regions in level-1B (L1B) images. The push-broom acquisition is done in the vertical direction. +
+ +
+
+ + + +
+ The L1BSR dataset includes 3740 pairs of overlapping image crops extracted from two L1B products. Each crop has a height of around 400 pixels and a variable width that depends on the overlap width between detectors for RGBN bands, typically around 120-200 pixels. In addition to detector parallax, there is also cross-band parallax for each detector, resulting in shifts between bands. Pre-registration is performed for both cross-band and cross-detector parallax, with a precision of up to a few pixels (typically less than 10 pixels). +
+
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+ + + +
+
+ + + +
+ Examples of overlapping L1B crops from the L1BSR dataset. +
+ + +
+
+  [Download link. L1BSR dataset] + +
+
+
+ + +
+ + +

Qualitative comparison with a supervised SR method

+
+ + + + +
+
+ +
+
+
+ +
+
+ + + +
+ We compare our self-supervised method L1BSR with a L1C-based supervised method, which uses PlanetScope images as ground-truth HR targets to train the SR model. The L1B and L1C images are from the same acquisition. +
+
+
+ +

Paper

+ + + + +
N. L. Nguyen, J. Anger, A. Davy, P. Arias, and G. Facciolo
+ L1BSR: Exploiting Detector Overlap for Self-Supervised Single-Image Super-Resolution of Sentinel-2 L1B Imagery.
+ In EarthVision 2023.
+ (hosted on ArXiv)
+ +
+ +
+
+ + + + + +
+ [Bibtex] +
+ +
+
+ + + + + +
+ +

Acknowledgements

+ This template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project; the code can be found here. + +
+ +
+ + + +