FusionSLAM-Unifying-Instant-NGP-for-Monocular-SLAM

Dive into cutting-edge FusionSLAM, where SuperPoint, SuperGlue, Neural Depth Estimation, and Instant-NGP converge, elevating Monocular SLAM to unparalleled precision and performance. Redefining mapping, localization, and reconstruction in a single camera setup.

🏁 Dependencies

1. NVIDIA Driver (Official Download Link)
2. CUDA Toolkit (Official Link)
3. ZED SDK (Official Guide)
4. OpenCV CUDA (Github Guide)
5. ROS 2 Humble (Official Link)
6. Miniconda (Official Link)
7. ZED ROS 2 Wrapper (Official Github Link)
8. RTAB-Map (Official Github Link)
9. RTAB-Map ROS 2 (Official Github Link)
10. PyTorch (Official Link)
11. Instant-ngp (Official Github Link)
12. SuperPoint (Official Github Link)
13. SuperGlue (Official Github Link)
14. Nlohmann-JSON (Official Github Link)

⚙️ Install

1. Install all non ROS 2 libraries
2. Clone all ROS 2 packages into workspace
3. Clone reporsitory into ROS 2 workspace
4. colcon build --symlink-install --cmake-args -DRTABMAP_SYNC_MULTI_RGBD=ON -DRTABMAP_SYNC_USER_DATA=ON -DPYTHON_EXECUTABLE=/usr/bin/python3 -DCMAKE_BUILD_TYPE=Release --parallel-workers $(nproc) --executor sequential
5. source ~/.bashrc or source ROS 2 workspace
6. Run python trace.py and change path of SuperPoint weights, this will generate a model compatible with your version of PyTorch
7. Add libtorch path export LD_LIBRARY_PATH=LD_LIBRARY_PATH:../miniconda3/envs/rtabmap/lib/python3.10/site-packages/torch/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}} ensure the path is correct else RTAB-Map will not work

⌛️ SLAM

1. Run SLAM to generate dataset
2. ros2 launch ngp_ros2 slam.launch.py rgb_topic:=/zed2i/zed_node/rgb/image_rect_color depth_topic:=/zed2i/zed_node/depth/depth_registered camera_info_topic:=/zed2i/zed_node/rgb/camera_info odom_topic:=/zed2i/zed_node/odom imu_topic:=/zed2i/zed_node/imu/data scan_cloud_topic:=/zed2i/zed_node/point_cloud/cloud_registered superpoint_model_path:=../SuperPointPretrainedNetwork/superpoint_v1.pt pydetector_path:=../rtabmap_superpoint.py pymatcher_path:=../rtabmap_superglue.py detection_rate:=1 image_path:=../images/ transform_path:=../transforms.json

SLAM

Your dataset should get created in the image_path along with transforms.json in the transform_path

🖼️ Instant-NGP

1. cd /instant-ngp/build
2. ./instant-ngp ../PATH give the path to your dataset where image_path and transforms.json are located

NERF

⚠️ Note

1. Ensure ZED ROS 2 Wrapper is set to run using Neural Depth Mode and Image quality is set to HD1080 for best renders
2. This Render uses Depth Supervision, feel free to change RTAB-Map and instant-ngp parameters to generate better renders

🔮 Future Updates

1. Use Pose-Graph from RTAB-Map to include loop closures for better renders
2. Add Segmentation masks using Semantic Segmentation Network
3. Generate render using Multi-camera SLAM