Deep Learning for Localization and Mapping

This repository is a collection of deep learning based localization and mapping approaches. A survey is offered in the following paper:

A Survey on Deep Learning for Localization and Mapping: Towards the Age of Spatial Machine Intelligence

Changhao Chen, Bing Wang, Chris Xiaoxuan Lu, Niki Trigoni and Andrew Markham

arXiv:2006.12567 [PDF]

This survey is also briefly introduced in Chinese [Zhihu]

News

Update: Jun-22-2020

• We released our survey paper.

TO DO

Category

• Odometry Estimation
  • Visual Odometry
  • Visual-Inertial Odometry
  • Inertial Odometry
  • LIDAR Odometry
• Mapping
  • Geometric Mapping
  • Semantic Mapping
  • General Mapping
• Global localization
  • 2D-to-2D Localization
  • 2D-to-3D Localization
  • 3D-to-3D Localization
• Simultaneous Localization and Mapping (SLAM)
  • Local Optimization
  • Global Optimization
  • Keyframe and Loop-closure Detection
  • Uncertainty Estimation

If you find this repository useful, please cite our paper:

@misc{chen2020survey,
title={A Survey on Deep Learning for Localization and Mapping: Towards the Age of Spatial Machine Intelligence},
author={Changhao Chen and Bing Wang and Chris Xiaoxuan Lu and Niki Trigoni and Andrew Markham},
year={2020},
eprint={2006.12567},
archivePrefix={arXiv},
primaryClass={cs.CV}
}

Categorized by Topic

*The Date in the table denotes the publication date (e.g. date of conference).

Odometry Estimation

Visual Odometry

Models	Date	Publication	Paper	Code
Konda et al.	2015	VISAPP	Learning visual odometry with a convolutional network
Costante et al.	2016	RA-L	Exploring Representation Learning With CNNs for Frame-to-Frame Ego-Motion Estimation
Backprop KF	2016	NeurIPS	Backprop KF: Learning Discriminative Deterministic State Estimators
DeepVO	2017	ICRA	DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks
SfmLearner	2017	CVPR	unsupervised learning of depth and ego-motion from video	TF PT
Yin et al.	2017	ICCV	Scale Recovery for Monocular Visual Odometry Using Depth Estimated With Deep Convolutional Neural Fields
UnDeepVO	2018	ICRA	UnDeepVO: Monocular Visual Odometry through Unsupervised Deep Learning
Barnes et al.	2018	ICRA	Driven to Distraction: Self-Supervised Distractor Learning for Robust Monocular Visual Odometry in Urban Environments
GeoNet	2018	CVPR	GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose	TF
Zhan et al.	2018	CVPR	Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction	Caffe
DPF	2018	RSS	Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors	TF
Yang et al.	2018	ECCV	Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry
Zhao et al.	2018	IROS	Learning monocular visual odometry with dense 3d mapping from dense 3d flow
Turan et al.	2018	IROS	Unsupervised Odometry and Depth Learning for Endoscopic Capsule Robots
Struct2Depth	2019	AAAI	Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos	TF
Saputra et al.	2019	ICRA	Learning monocular visual odometry through geometry-aware curriculum learning
GANVO	2019	ICRA	GANVO: Unsupervised deep monocular visual odometry and depth estimation with generative adversarial networks
CNN-SVO	2019	ICRA	CNN-SVO: Improving the Mapping in Semi-Direct Visual Odometry Using Single-Image Depth Prediction	ROS
Li et al.	2019	ICRA	Pose graph optimization for unsupervised monocular visual odometry
Xue et al.	2019	CVPR	Beyond tracking: Selecting memory and refining poses for deep visual odometry
Wang et al.	2019	CVPR	Recurrent neural network for (un-) supervised learning of monocular video visual odometry and depth
Li et al.	2019	ICCV	Sequential adversarial learning for self-supervised deep visual odometry
Saputra et al.	2019	ICCV	Distilling knowledge from a deep pose regressor network
Gordon et al.	2019	ICCV	Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras	TF
Koumis et al.	2019	IROS	Estimating Metric Scale Visual Odometry from Videos using 3D Convolutional Networks
Bian et al.	2019	NeurIPS	Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video	PT
D3VO	2020	CVPR	D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry
Jiang et al.	2020	CVPR	Joint Unsupervised Learning of Optical Flow and Egomotion with Bi-Level Optimization

Visual-Inertial Odometry

Models	Date	Publication	Paper	Code
VINet	2017	AAAI	VINet: Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem
VIOLearner	2019	TPAMI	Unsupervised deep visual-inertial odometry with online error correction for rgb-d imagery
SelectFusion	2019	CVPR	Selective Sensor Fusion for Neural Visual-Inertial Odometry
DeepVIO	2019	IROS	DeepVIO: Self-supervised deep learning of monocular visual inertial odometry using 3d geometric constraints

Inertial Odometry

Models	Date	Publication	Paper	Code
IONet	2018	AAAI	IONet: Learning to Cure the Curse of Drift in Inertial Odometry
RIDI	2018	ECCV	RIDI: Robust IMU Double Integration	Py
Wagstaff et al.	2018	IPIN	LSTM-Based Zero-Velocity Detection for Robust Inertial Navigation	PT
Cortes et al.	2019	MLSP	Deep Learning Based Speed Estimation for Constraining Strapdown Inertial Navigation on Smartphones
MotionTransformer	2019	AAAI	MotionTransformer: Transferring Neural Inertial Tracking between Domains
AbolDeepIO	2019	TITS	AbolDeepIO: A Novel Deep Inertial Odometry Network for Autonomous Vehicles
Brossard et al.	2019	ICRA	Learning wheel odometry and imu errors for localization
OriNet	2019	RA-L	OriNet: Robust 3-D Orientation Estimation With a Single Particular IMU	PT
L-IONet	2020	IoT-J	Deep Learning based Pedestrian Inertial Navigation: Methods, Dataset and On-Device Inference

LIDAR Odometry

Models	Date	Publication	Paper	Code
Velas et al.	2018	ICARSC	CNN for IMU Assisted Odometry Estimation using Velodyne LiDAR
LO-Net	2019	CVPR	LO-Net: Deep Real-time Lidar Odometry
DeepPCO	2019	IROS	DeepPCO: End-to-End Point Cloud Odometry through Deep Parallel Neural Network
Valente et al.	2019	IROS	Deep sensor fusion for real-time odometry estimation

Mapping

Geometric Mapping

Depth Representation

• Joint learning of depth and ego-motion has been discussed in Visual Odometry. We do not include these works here, although they can produce depth representation.

Models	Date	Publication	Paper	Code
Eigen et al.	2014	NeurIPS	Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Liu et al.	2015	TPAMI	Learning depth from single monocular images using deep convolutional neural fields
Garg et al.	2016	ECCV	Unsupervised cnn for single view depth estimation: Geometry to the rescue
Demon	2017	CVPR	Demon: Depth and motion network for learning monocular stereo
Godard et al.	2017	CVPR	Unsupervised monocular depth estimation with left-right consistency
Wang et al.	2018	CVPR	Learning depth from monocular videos using direct methods

Voxel Representation

Models	Date	Publication	Paper	Code
SurfaceNet	2017	CVPR	SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis
Dai et al.	2017	CVPR	Shape completion using 3d-encoder-predictor cnns and shape synthesis
Hane et al.	2017	3DV	Hierarchical surface prediction for 3d object reconstruction
OctNetFusion	2017	3DV	Octnetfusion: Learning depth fusion from data
OGN	2017	ICCV	Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs
Kar et al.	2017	NeurIPS	Learning a multi-view stereo machine
RayNet	2018	CVPR	RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials

Point Representation

Models	Date	Publication	Paper	Code
Fan et al.	2017	CVPR	A point set generation network for 3d object reconstruction from a single image

Mesh Representation

Models	Date	Publication	Paper	Code
Ladicky et al.	2017	ICCV	From point clouds to mesh using regression
Mukasa et al.	2017	ICCVW	3d scene mesh from cnn depth predictions and sparse monocular slam
Wang et al.	2018	ECCV	Pixel2mesh: Generating 3d mesh models from single rgb images
Groueix et al.	2018	CVPR	AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation
Scan2Mesh	2019	CVPR	Scan2mesh: From unstructured range scans to 3d meshes
Bloesch et al.	2019	ICCV	Learning meshes for dense visual SLAM

Semantic Mapping

Models	Date	Publication	Paper	Code
SemanticFusion	2017	ICRA	Semanticfusion: Dense 3d semantic mapping with convolutional neural networks
DA-RNN	2017	RSS	DA-RNN: Semantic mapping with data associated recurrent neural networks
Ma et al.	2017	IROS	Multi-view deep learning for consistent semantic mapping with rgb-d cameras
Sunderhauf et al.	2017	IROS	Meaningful maps with object-oriented semantic mapping
Fusion++	2018	3DV	Fusion++: Volumetric object-level SLAM
Grinvald et al.	2019	RA-L	Volumetric instance-aware semantic mapping and 3d object discovery
PanopticFusion	2019	IROS	Panopticfusion: Online volumetric semantic mapping at the level of stuff and things

General Mapping

• neural scene representation, task-driven representation

Models	Date	Publication	Paper	Code
Mirowski et al.	2017	ICLR	Learning to navigate in complex environments
Zhu et al.	2017	ICRA	Target-driven visual navigation in indoor scenes using deep reinforcement learning
Eslami et al.	2018	Science	Neural scene representation and rendering
CodeSLAM	2018	CVPR	CodeSLAM — Learning a Compact, Optimisable Representation for Dense Visual SLAM
Mirowski et al.	2018	NeurIPS	Learning to navigate in cities without a map
SRN	2019	NeurIPS	Scene representation networks: Continuous 3d-structure-aware neural scene representations
Tobin et al.	2019	NeurIPS	Geometry-aware neural rendering
Lim et al.	2019	NeurIPS	Neural multisensory scene inference

Global Localization

2D-to-2D Localization

Implicit Map Based Localization

Models	Date	Publication	Paper	Code
PoseNet	2015	ICCV	PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization
Bayesian PoseNet	2016	ICRA	Modelling uncertainty in deep learning for camera relocalization
BranchNet	2017	ICRA	Delving deeper into convolutional neural networks for camera relocalization
VidLoc	2017	CVPR	VidLoc: A Deep Spatio-Temporal Model for 6-DoF Video-Clip Relocalization
Geometric PoseNet	2017	CVPR	Geometric loss functions for camera pose regression with deep learning
Naseer et al.	2017	IROS	Deep Regression for Monocular Camera-based 6-DoF Global Localization in Outdoor Environments
LSTM-PoseNet	2017	ICCV	Image-based localization using lstms for structured feature correlation
Hourglass PoseNet	2017	ICCV Workshops	Image-based localization using hourglass networks
VLocNet	2018	ICRA	Deep auxiliary learning for visual localization and odometry
MapNet	2018	CVPR	Geometry-Aware Learning of Maps for Camera Localization
SPP-Net	2018	BMVC	Synthetic view generation for absolute pose regression and image synthesis
GPoseNet	2018	BMVC	A hybrid probabilistic model for camera relocalization
VLocNet++	2018	RA-L	Vlocnet++: Deep multitask learning for semantic visual localization and odometry
Xue et al.	2019	ICCV	Local supports global: Deep camera relocalization with sequence enhancement
Huang et al.	2019	ICCV	Prior guided dropout for robust visual localization in dynamic environments
Bui et al.	2019	ICCVW	Adversarial networks for camera pose regression and refinement
GN-Net	2020	RA-L	GN-Net: The Gauss-Newton Loss for Multi-Weather Relocalization
AtLoc	2020	AAAI	AtLoc: Attention Guided Camera Localization

Explicit Map Based Localization

Models	Date	Publication	Paper	Code
Laskar et al.	2017	ICCV Workshops	Camera Relocalization by Computing Pairwise Relative Poses Using Convolutional Neural Network
DELS-3D	2018	CVPR	Dels-3d: Deep localization and segmentation with a 3d semantic map
AnchorNet	2018	BMVC	Improved visual relocalization by discovering anchor points
RelocNet	2018	ECCV	RelocNet: Continuous Metric Learning Relocalisation using Neural Nets
CamNet	2019	ICCV	Camnet: Coarse-to-fine retrieval for camera re-localization

2D-to-3D Localization

Descriptor Matching

Models	Date	Publication	Paper	Code
NetVLAD	2016	CVPR	Netvlad: Cnn architecture for weakly supervised place recognition
DELF	2017	CVPR	Large-scale image retrieval with attentive deep local features
Schonberger et al.	2018/06	CVPR	Semantic Visual Localization
SuperPoint	2018	CVPRW	Superpoint: Selfsupervised interest point detection and description
NC-Net	2018	NeurIPS	Neighbourhood consensus networks
Sarlin et al.	2019/06	CVPR	From Coarse to Fine: Robust Hierarchical Localization at Large Scale
2D3D-MatchNet	2019	ICRA	2d3d-matchnet: learning to match keypoints across 2d image and 3d point cloud
D2-Net	2019	CVPR	D2-net: A trainable cnn for joint description and detection of local features
Speciale et al.	2019	CVPR	Privacy preserving image-based localization
OOI-Net	2019	CVPR	Visual localization by learning objects-of-interest dense match regression
Camposeco et al.	2019	CVPR	scene compression for visual localization
Cheng et al.	2019	CVPR	Cascaded parallel filtering for memory-efficient image-based localization
Taira et al.	2019	CVPR	Is this the right place? geometric-semantic pose verification for indoor visual localization
R2D2	2019	NeurIPS	R2d2: Repeatable and reliable detector and descriptor
ASLFeat	2020	CVPR	Aslfeat: Learning local features of accurate shape and localization

Scene Coordinate Regression

Models	Date	Publication	Paper	Code
DSAC	2017/07	CVPR	DSAC - Differentiable RANSAC for Camera Localization
DSAC++	2018/06	CVPR	Learning less is more-6d camera localization via 3d surface regression
Dense SCR	2018/07	RSS	Full-Frame Scene Coordinate Regression for Image-Based Localization
DSAC++ angle	2018/09	ECCV	Scene coordinate regression with angle-based reprojection loss for camera relocalization
Confidence SCR	2018/09	BMVC	Scene Coordinate and Correspondence Learning for Image-Based Localization
ESAC	2019/10	ICCV	Expert Sample Consensus Applied to Camera Re-Localization
NG-RANSAC	2019/06	CVPR	Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses
SANet	2019/10	ICCV	SANet: scene agnostic network for camera localization
HSC-Net	2020	CVPR	Hierarchical scene coordinate classification and regression for visual localization
KF-Net	2020	CVPR	Kfnet: Learning temporal camera relocalization using kalman filtering

3D-to-3D Localization

Models	Date	Publication	Paper	Code
LocNet	2018	IV	Locnet: Global localization in 3d point clouds for mobile vehicles
PointNetVLAD	2018	CVPR	Pointnetvlad: Deep point cloud based retrieval for large-scale place recognition
Barsan et al.	2018	CoRL	Learning to localize using a lidar intensity map
L3-Net	2019	CVPR	L3-net: Towards learning based lidar localization for autonomous driving
PCAN	2019	CVPR	PCAN: 3D Attention Map Learning Using Contextual Information for Point Cloud Based Retrieval
DeepICP	2019	CVPR	Deepicp: An end-to-end deep neural network for 3d point cloud registration
DCP	2019	CVPR	Deep closest point: Learning representations for point cloud registration
D3Feat	2020	CVPR	D3feat: Joint learning of dense detection and description of 3d local features

SLAM

Local Optimization

Models	Date	Publication	Paper	Code
LS-Net	2018	ECCV	Learning to solve nonlinear least squares for monocular stereo
BA-Net	2019	ICLR	BA-Net: Dense bundle adjustment network

Global Optimization

Models	Date	Publication	Paper	Code
CNN-SLAM	2017	CVPR	CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction
Li et al.	2019	ICRA	Pose graph optimization for unsupervised monocular visual odometry
DeepTAM	2020	IJCV	DeepTAM: Deep Tracking and Mapping with Convolutional Neural Networks
DeepFactors	2020	RA-L	DeepFactors: Real-Time Probabilistic Dense Monocular SLAM

Keyframe and Loop-closure Detection

Models	Date	Publication	Paper	Code
Sunderhauf et al.	2015	RSS	Place recognition with convnet landmarks: Viewpoint-robust, condition-robust, training-free
Gao et al.	2017	AR	Unsupervised learning to detect loops using deep neural networks for visual slam system
Huang et al.	2018	RSS	Lightweight unsupervised deep loop closure
Sheng et al.	2019	ICCV	Unsupervised Collaborative Learning of Keyframe Detection and Visual Odometry Towards Monocular Deep SLAM
Memon et al.	2020	RAS	Loop closure detection using supervised and unsupervised deep neural networks for monocular slam systems

Uncertainty Estimation

Models	Date	Publication	Paper	Code
Kendall et al.	2016	ICRA	Modelling uncertainty in deep learning for camera relocalization
Kendall et al.	2017	NeurIPS	What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
VidLoc	2017	CVPR	VidLoc: A Deep Spatio-Temporal Model for 6-DoF Video-Clip Relocalization
Wang et al.	2018	IJRR	End-to-end, sequenceto-sequence probabilistic visual odometry through deep neural networks
Chen et al.	2019	TMC	Deep neural network based inertial odometry using low-cost inertial measurement units

This list is maintained by Changhao Chen and Bing Wang, Department of Computer Science, University of Oxford.

Please contact them (email: [email protected]; [email protected]), if you have any question or would like to add your work on this list.