Skip to content

Commit

Permalink
Merge pull request #21 from meyerls/dev
Browse files Browse the repository at this point in the history 
Dev
  • Loading branch information
@meyerls
meyerls authored Oct 14, 2022
2 parents 3c9c4e4 + 9cee011 commit 8c082f7
Show file tree
Hide file tree
Showing 10 changed files with 75 additions and 158 deletions.
146 changes: 68 additions & 78 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
![](https://media.githubusercontent.com/media/meyerls/aruco-estimator/main/img/header.png)
<p align="center" width="100%">
<img width="100%" src="https://github.com/meyerls/aruco-estimator/blob/dev/img/wood.png?raw=true">
</p>

# Aruco Scale Factor Estimation for COLMAP (Work in Progress!)
# Automatic Aruco marker-based scale factor estimation (Work in Progress!)

<a href="https://pypi.org/project/aruco-estimator/"><img alt="PyPI - Python Version" src="https://img.shields.io/pypi/pyversions/aruco-estimator"></a>
<a href="https://pypi.org/project/aruco-estimator/"><img alt="PyPI" src="https://img.shields.io/pypi/v/aruco-estimator"></a>
Expand All @@ -17,46 +19,7 @@
## About

This project aims to automatically compute the correct scale of a point cloud generated
with [COLMAP](https://colmap.github.io/). By adding an aruco marker with into the scene it is possible to overcome the
scale ambiguity from monocular SfM and SLAM. In the literature it is commonly suggested to manually measure a known
distance of a calibration object in the 3D reconstruction and then scale the reconstruction accordingly or make photos
from two known positions[1]. Since there is unfortunately no automated approach to calculate an accurate scaling factor,
this project aims to be attached as a post processing step to the reconstruction process.

## Setup

Before taking the images for the reconstruction, an aruco marker must be printed out. Care should be taken to ensure
that the size of the marker fits the scene to be recognizable in the images. Additionally the marker is placed on a
planar surface and does not have curvature (otherwise, the marker will be recognized incorrectly or not at all). The
aruco marker can either be generated
with [ ArUco-markers-with-OpenCV-and-Python](https://github.com/KhairulIzwan/ArUco-markers-with-OpenCV-and-Python) or
generated easily on [this website](https://chev.me/arucogen/). When taking the images, make sure that the aruco marker
is visible in at least two images; otherwise, no solution exists. It is advisable to have at least five images with the
aruco marker for an accurate solution (two pictures should already be enough for a sufficient approximation. However,
blurry images or aruco markers that are too small could falsify the result).
After the images have been acquired, the reconstructions process can be carried out using COLMAP. For the algorithm to
work it only needs the result of the Structure from Motion part (extrinsic parameters). The MVS part for the dense
reconstruction is used to visually verify the result.

<!-- ## Theory
At first the extrinsic <img src="https://render.githubusercontent.com/render/math?math=\mathbf{M}_i"> and intrinsic
paramters <img src="https://render.githubusercontent.com/render/math?math=\mathbf{K}_i"> for every
image <img src="https://render.githubusercontent.com/render/math?math=\mathbf{I}_i"> of the reconstruction are readout
from the parsed COLMAP project and their underlying [binary output format](https://colmap.github.io/format.html). Then,
in each image <img src="https://render.githubusercontent.com/render/math?math=\mathbf{I}_i">, it is checked whether an
Arco marker is present. If so, all four corners of the square Aruco markers are extracted as image
coordinates <img src="https://render.githubusercontent.com/render/math?math=\mathbf{x} = (x_i, y_i, 1)^\top">. Thus it
is possible to cast a ray from the origin of the camera center
<img src="https://render.githubusercontent.com/render/math?math=\mathbf{C}_i">
through all four Aruco corners trough two points according to
<img src="https://render.githubusercontent.com/render/math?math=r_{vec} = \mathbf{C}_i + ||\mathbf{x} \mathbf{K}^{-1} ||_2^2 \mathbf{R}_i \lambda">
with <img src="https://render.githubusercontent.com/render/math?math=\lambda \in [-\infty, +\infty]"> . Here x is the
image coordinate in homogeneous coordinates, K^-1 is the inverse matrix of the intrinsic matrix of the camera, R_i the
rotation of the extrinsic camera parameters and C_i the translation t_i of the camera pose. Four rays are cast for each
image in which an aruco marker is detected. Thus, with a minimum of two rays per aruco corner, the position in 3D space
can be determined trough the intersection of lines. The intersection of several 3D lines can then be calculated using a
least-squares method [2].-->
with [COLMAP](https://colmap.github.io/) by placing an aruco marker into the scene.

## Installation

Expand All @@ -72,14 +35,26 @@ pip install aruco-estimator
## Usage

### Dataset
A simple dataset of an aruco marker on a door is provided. It is automatically downloaded by using

An exemplary data set is provided. The dataset shows a simple scene of a door with an aruco marker. Other dataset might
follow in future work. It can be downloaded by using

````python
from aruco_estimator import download

dataset = download.Dataset()
dataset.download_door_dataset()
dataset.download_door_dataset(output_path='.')
````

### API

A use of the code on the provided dataset can be seen in the following block. The most important function is
``aruco_scale_factor.run()``. Here, an aruco marker is searched for in each image. If a marker is found in at
least 2 images, the position of the aruco corner in 3D is calculated based on the camera poses and the corners of
the aruco maker.Based on the positions of the corners of the square aruco marker, the size of the marker in the unscaled
reconstruction can be determined. With the correct metric size of the marker, the scene can be scaled true to scale
using ``aruco_scale_factor.apply(true_scale)``.

````python
from aruco_estimator.aruco_scale_factor import ArucoScaleFactor
from aruco_estimator import download
Expand All @@ -101,22 +76,29 @@ print('Point cloud and poses are scaled by: ', scale_factor)
print('Size of the scaled (true to scale) aruco markers in meters: ', aruco_distance * scale_factor)

# Visualization of the scene and rays BEFORE scaling. This might be necessary for debugging
aruco_scale_factor.visualize_estimation(frustum_scale=0.2)
aruco_scale_factor.visualize_estimation(frustum_scale=0.4)
o3d.io.write_point_cloud(os.path.join(dataset.colmap_project, 'scaled.ply'), dense)
aruco_scale_factor.write_data()
````

## Source

### Get Repo
If you want to install the repo from source make sure that conda is installed. Afterwards clone this repository, give
the bash file executable rights and install the conda env:

````angular2html
git clone https://github.com/meyerls/aruco-estimator.git
cd aruco-estimator
conda env create -f environment.yml
conda activate aruco_estimator
pip install -r requirements.txt
chmod u+x init_env.sh
./init_env.sh
````

Finally install all python dependencies in the activated conda environment via

````angular2html
pip install -r requirements.txt
````

### Usage of Command Line

````angular2html
Expand All @@ -125,58 +107,66 @@ usage: scale_estimator.py [-h] [--colmap_project COLMAP_PROJECT] [--dense_model
Estimate scale factor for COLMAP projects with aruco markers.
optional arguments:
-h, --help show this help message and exit
--colmap_project COLMAP_PROJECT Path to COLMAP project
--dense_model DENSE_MODEL name to the dense model
--aruco_size ARUCO_SIZE Size of the aruco marker in cm.
--visualize Flag to enable visualization
--point_size POINT_SIZE Point size of the visualized dense point cloud. Depending on the number of points in the model. Between 0.001 and 2
--frustum_size FRUSTUM_SIZE Size of the visualized camera frustums. Between 0 (small) and 1 (large)
--test_data Download and try out test data
-h, --help show this help message and exit
--colmap_project COLMAP_PROJECT Path to COLMAP project
--dense_model DENSE_MODEL name to the dense model
--aruco_size ARUCO_SIZE Size of the aruco marker in cm.
--visualize Flag to enable visualization
--point_size POINT_SIZE Point size of the visualized dense point cloud. Depending on the number of points in the model. Between 0.001 and 2
--frustum_size FRUSTUM_SIZE Size of the visualized camera frustums. Between 0 (small) and 1 (large)
--test_data Download and try out test data
````

To test the code on your local machine try the example project by using:

````angular2html
python scale_estimator.py --test_data
````
<p align="center" width="100%">
<img width="100%" src="https://github.com/meyerls/aruco-estimator/blob/dev/img/door.png?raw=true">
</p>

### Visualization

The visualization is suitable for displaying the scene, the camera frustums, and the casted rays. It is usefull to check
whether the corners of the aruco marker are detected and positioned correctly. If the aruco markers are not detected
correctly the aruco settings must be changed according to the scene to be more robust.
<p align="center" width="100%">
<img width="100%" src="https://github.com/meyerls/aruco-estimator/blob/dev/img/output.gif?raw=true">
</p>

![](https://media.githubusercontent.com/media/meyerls/aruco-estimator/main/img/visualization.png)
## Limitation / Improvements

## Limitation/Improvements

- [x] Make package for PyPi
- [x] Upload to PyPi during CI
- [x] Make dataset available for download
- [x] Put aruco marker detection in threads
- [x] Scale poses of camera/extrinsics.
- [ ] Up to know only SIMPLE_RADIAL and PINHOLE camera models are supported. Extend all models
- [ ] Up to now only SIMPLE_RADIAL and PINHOLE camera models are supported. Extend all models
- [ ] Install CLI Tool vi PyPi
- [ ] Up to now only one aruco marker per scene can be detected. Multiple aruco marker could improve the scale
estimation
- [ ] Different aruco marker settings should be investigated for different scenarios to make it either more robust to
- [ ] Different aruco marker settings and marker types should be investigated for different scenarios to make it either more robust to
false detections
- [ ] Geo referencing of aruco markers with earth coordinate system using GPS or RTK
- [ ] Alternatives to aruco marker should be investigated.
- [ ] Are the corners from the aruco marker returned identical regarding the orientation in the image?
- [ ] Only COLMAP is supported. Add additional reconstruction software.

## Acknowledgement

The Code to read out the binary COLMAP data is partly borrowed from the
* The Code to read out the binary COLMAP data is partly borrowed from the
repo [COLMAP Utility Scripts](https://github.com/uzh-rpg/colmap_utils) by [uzh-rpg](https://github.com/uzh-rpg).
* The visualization of the wooden block is created from the dataset found in [[1](https://robocip-aist.github.io/sii_nerf_scans/)]

## Trouble Shooting

- In some cases cv2 does not detect the aruco marker module. Reinstalling opencv-python and opencv-python-python might
help [Source](https://stackoverflow.com/questions/45972357/python-opencv-aruco-no-module-named-cv2-aruco)

## Resources
## References

<div class="csl-entry">[1] Erich, F., Bourreau, B., <i>Neural Scanning: Rendering and determining geometry of household objects using Neural Radiance Fields</i> <a href="https://robocip-aist.github.io/sii_nerf_scans/">Link</a>. 2022</div>

## Citation

Please cite this paper, if this work helps you with your research:

<div class="csl-entry">[1] Lourakis, M., &#38; Zabulis, X. (n.d.). <i>LNCS 8047 - Accurate Scale Factor Estimation in 3D Reconstruction</i>. 2013</div>
<div class="csl-entry">[2] Traa, J., <i>Least-Squares Intersection of Lines</i>. 2013</div>
```
@InProceedings{ ,
author="H",
title="",
booktitle="",
year="",
pages="",
isbn=""
}
```
6 changes: 0 additions & 6 deletions aruco_estimator/aruco.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,6 @@ def ray_cast_aruco_corners_visualization(p_i: np.ndarray, n_i: np.ndarray, corne
t_2 = np.linalg.norm((p1_2 - p_i))
t_3 = np.linalg.norm((p1_3 - p_i))


#t_0 = np.mean((p1_0 - p_i) / n_i[0])
#t_1 = np.mean((p1_1 - p_i) / n_i[1])
#t_2 = np.mean((p1_2 - p_i) / n_i[2])
#t_3 = np.mean((p1_3 - p_i) / n_i[3])

points_camera_plane = [
p_i,
p_i + n_i[0] * t_0, # p1_0,
Expand Down
74 changes: 4 additions & 70 deletions aruco_estimator/aruco_scale_factor.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ def __init__(self, project_path: str, dense_path: str = 'fused.ply'):

# Multi Processing
self.progress_bar = True
self.num_processes = 1 # 8 if os.cpu_count() > 8 else os.cpu_count()
self.num_processes = 8 if os.cpu_count() > 8 else os.cpu_count()
print('Num process: ', self.num_processes)
self.image_names = []
# Prepare parsed data for multi processing
Expand Down Expand Up @@ -232,73 +232,6 @@ def __evaluate(aruco_corners_3d: np.ndarray) -> np.ndarray:
# Average
return np.mean([dist1, dist2, dist3, dist4])

def visualize_scaled_scene(self, frustum_scale: float = 0.15, point_size: float = 1., sphere_size: float = 0.01):
"""
This visualization function show the scaled dense and scaled extrinsic parameters.
@param sphere_size:
@param frustum_scale:
@param point_size:
"""

geometries = [self.dense_scaled]

for image_idx in self.images_scaled.keys():
line_set, sphere, mesh = draw_camera_viewport(extrinsics=self.images_scaled[image_idx].extrinsics,
intrinsics=self.images_scaled[image_idx].intrinsics.K,
image=self.images_scaled[image_idx].image,
scale=frustum_scale)

# aruco_line_set = ray_cast_aruco_corners_visualization(extrinsics=M,
# intrinsics=camera_intrinsics.K,
# corners=self.images[image_idx].aruco_corners)

# geometries.append(aruco_line_set)
geometries.append(mesh)
geometries.append(line_set)
geometries.extend(sphere)

aruco_sphere = create_sphere_mesh(t=self.aruco_corners_3d * self.scale_factor,
color=[[0, 0, 0],
[1, 0, 0],
[0, 0, 1],
[1, 1, 1]],
radius=sphere_size)
aruco_rect = generate_line_set(points=[self.aruco_corners_3d[0] * self.scale_factor,
self.aruco_corners_3d[1] * self.scale_factor,
self.aruco_corners_3d[2] * self.scale_factor,
self.aruco_corners_3d[3] * self.scale_factor],
lines=[[0, 1], [1, 2], [2, 3], [3, 0]], color=[1, 0, 0])

'''
# Plot/show text of distance in 3 Reco. Currently not working as text is rotated wrongly. tbd!
pos_text = (self.aruco_corners_3d[0] + (
self.aruco_corners_3d[1] - self.aruco_corners_3d[0]) / 2) * self.scale_factor
pcd_tree = o3d.geometry.KDTreeFlann(self.dense_scaled)
[k, idx, _] = pcd_tree.search_knn_vector_3d(pos_text, 100)
dir_vec = []
for i in idx:
dir_vec.append(self.dense_scaled.normals[i])
dir_vec = np.asarray(dir_vec).mean(axis=0)
dist = np.linalg.norm(
self.aruco_corners_3d[0] * self.scale_factor - self.aruco_corners_3d[1] * self.scale_factor)
pcd_text = text_3d(text='{:.4f} cm'.format(dist*100),
pos=pos_text,
direction=dir_vec)
geometries.append(pcd_text)
'''

geometries.extend(aruco_sphere)
geometries.append(aruco_rect)

self.start_visualizer_scaled(geometries=geometries, point_size=point_size,
title='Aruco Scale Factor Estimation Scaled')

def visualize_estimation(self, frustum_scale: float = 1, point_size: float = 1., sphere_size: float = 0.02):
"""
Expand All @@ -308,8 +241,9 @@ def visualize_estimation(self, frustum_scale: float = 1, point_size: float = 1.,
"""

# Add Dense & sparse Model to scene
self.add_colmap_dense2geometrie()
self.add_colmap_sparse2geometrie()
dense_exists = self.add_colmap_dense2geometrie()
if not dense_exists:
self.add_colmap_sparse2geometrie()
# Add camera frustums to scene
self.add_colmap_frustums2geometrie(frustum_scale=frustum_scale)

Expand Down
Binary file added img/door.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added img/output.gif
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added img/wood.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added img/wood_orig.jpg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added img/wood_reco.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 2 additions & 3 deletions scale_estimator.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@
parser.add_argument('--visualize', action='store_true', help='Flag to enable visualization')
parser.add_argument('--point_size', type=float, help='Point size of the visualized dense point cloud. '
'Depending on the number of points in the model. '
'Between 0.001 and 2', default=0.2)
'Between 0.001 and 2', default=0.1)
parser.add_argument('--frustum_size', type=float, help='Size of the visualized camera frustums. '
'Between 0 (small) and 1 (large)', default=0.75)
'Between 0 (small) and 1 (large)', default=0.5)
parser.add_argument('--test_data', action='store_true', help='Download and try out test data')
args = parser.parse_args()

Expand Down Expand Up @@ -60,6 +60,5 @@
if args.visualize:
aruco_scale_factor.visualize_estimation(frustum_scale=args.frustum_size, point_size=args.point_size)

# aruco_scale_factor._ArucoScaleFactor__visualization_scaled_scene(frustum_scale=0.2)
# Todo: Save output, PCD and poses. Visualize!
aruco_scale_factor.write_data()
2 changes: 1 addition & 1 deletion setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

setuptools.setup(
name='aruco-estimator',
version='1.1.4',
version='1.1.5',
description='Aruco Scale Factor Estimation',
license="MIT",
long_description=long_description,
Expand Down

0 comments on commit 8c082f7

Please sign in to comment.