Noise in depth gt #4

BlueNine9 · 2023-10-10T01:03:02Z

Thank you for your excellent work!

I've downloaded the test set from the website and converted the depth image into point cloud using the following code:

import os
import numpy as np
from numpy.matlib import repmat

os.environ['OPENCV_IO_ENABLE_OPENEXR'] = '1'

import cv2

image = cv2.imread('0000.exr', cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)[..., 0]

pixel_size = image.shape[0] * image.shape[1]

depth_image = image.reshape(pixel_size)
u_coord = repmat(np.r_[0:image.shape[1]:1], image.shape[0], 1).reshape(pixel_size)
v_coord = repmat(np.c_[0:image.shape[0]:1], 1, image.shape[1]).reshape(pixel_size)

fx = 499.9999781443055
fy = 499.9999781443055
cx = image.shape[1] / 2
cy = image.shape[0] / 2

k = np.identity(3)
k[0, 2] = cx
k[1, 2] = cy
k[0, 0] = k[1, 1] = fx

p2d = np.array([u_coord, v_coord, np.ones_like(u_coord)])

p3d = np.transpose(np.dot(np.linalg.inv(k), p2d) * depth_image).reshape(-1, 3)

point_cloud = []

for x, y, z in p3d:
	x, y, z = z, x, -y
	point_cloud.append([x, y, z])

point_cloud = np.array(point_cloud)

from plyfile import PlyData, PlyElement

vertices = np.array([(x, y, z) for x, y, z in point_cloud.reshape(-1, 3)], dtype=[('x', 'f4'), ('y', 'f4'), ('z', 'f4')])
plydata = PlyData([PlyElement.describe(vertices, 'vertex')], text=False)
plydata.write('point_cloud.ply')

The shape of the point cloud seemed nice, however there seemed to be a noticeable amount of noise in the point cloud.

Is this normal or am I using a wrong way to convert the exr file?
I'm opening 0000.exr from small_city_road_horizon_test_depth.tar.

The text was updated successfully, but these errors were encountered:

yixuanli98 · 2023-10-18T06:27:57Z

Sorry for the late reply. Have you tried depth from other view? Did you observe similar phenomenon?

yixuanli98 · 2023-10-22T10:19:28Z

Thank you very much for you question! I think the way you read depth is right. We have checked our depth map and find the reason. For the best rendering quality, we use the anti-alias setting to reduce noise, which will have a smoothing effect on the image. The depths change dramatically at the edge of the object and anti-alias reduces this change, which will cause the noisy point cloud. The depth map without and with anti-alias are shown below, which is a zoom-in pixel-level comparison.

We think the depth maps rendered with anti-lias can better align the rgb images rendered with anti-alias and can support most of the requirements, like depth guidance for reconstruction. But for generating point cloud from depth map, anti-alias will bring many noises like you find. We have tried generating point cloud from the depth rendered without anti-alias and solved the problem. We will update a new depth map version without anti-alias soon.

ChaoyiZh · 2023-11-15T02:13:33Z

Hi Guys
Can I ask why the depth value from the .exr may has thousands? Is there any scaling for it?

yixuanli98 · 2023-11-15T02:21:35Z

The unit is cm.

yixuanli98 · 2023-11-15T03:04:30Z

The unit in our pose file (street/pose and aerial/pose) is 10000cm. If you use the pose file to train nerf, you can scale the depth file with 10000.

ChaoyiZh · 2023-11-15T04:44:34Z

I am trying to fuse serval RGB and depth pairs to generate the pointclouds. Here is my core code and please feel free to point out if there is any problems. According to the instruction from YiXuan and previous post, I divided the depth with 10000 and did the same coordinate conversion. But the results seems wired.

os.environ['OPENCV_IO_ENABLE_OPENEXR'] = '1'
def depth_to_pc(depth_path, intrinsic, transform_matrix):
    depth_map = cv2.imread(depth_path, cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)[..., 0]
    pixel_size = depth_map.shape[0] * depth_map.shape[1]
    depth_image = depth_map.reshape(pixel_size)/10000
    u_coord = np.tile(np.arange(depth_map.shape[1]), (depth_map.shape[0], 1)).reshape(pixel_size)
    v_coord = np.tile(np.arange(depth_map.shape[0]), (depth_map.shape[1], 1)).T.reshape(pixel_size)
    k = np.identity(3)
    k[0, 2] = intrinsic["cx"]
    k[1, 2] = intrinsic["cy"]
    k[0, 0] = intrinsic["fl_x"]
    k[1, 1] = intrinsic["fl_x"]
    p2d = np.array([u_coord, v_coord, np.ones_like(u_coord)])
    p3d = np.transpose(np.dot(np.linalg.inv(k), p2d) * depth_image).reshape(-1, 3)

    # Left coord to right
    point_cloud = p3d
    point_cloud[:, 1] = -point_cloud[:, 1]
    ones = np.ones((point_cloud.shape[0], 1))
    homogenous_coordinate = np.hstack((point_cloud, ones))
    transformed_points = (transform_matrix @ homogenous_coordinate.T).T

    return transformed_points[:, :3]
frames = data["frames"]

for frame in frames[:10]:
    rel_img_path = frame["file_path"]
    transform_matrix = frame["transform_matrix"]
    transform_matrix = np.array(transform_matrix)



    abs_img_path = os.path.abspath(os.path.join(pose_path, rel_img_path)).replace("/pose", "")
    abs_depth_path = abs_img_path.replace("small_city", "small_city_depth",1).replace("small_city_road_down", "small_city_road_down_depth").replace("png", "exr")
    if not os.path.exists(abs_img_path) or not os.path.exists(abs_depth_path):
        continue
    depth_image = cv2.imread(abs_depth_path, cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)[..., 0]
    color_image = cv2.imread(abs_img_path)
    color_image = cv2.cvtColor(color_image, cv2.COLOR_BGR2RGB)
    point_cloud = depth_to_pc(abs_depth_path, camera_intrinsics, transform_matrix)

    all_points.append(point_cloud)

    colors = color_image.reshape(-1, 3)/255.0
    all_colors.append(colors)


merged_points = np.vstack(all_points)
merged_colors = np.vstack(all_colors)


final_pcd = o3d.geometry.PointCloud()
final_pcd.points = o3d.utility.Vector3dVector(merged_points)
final_pcd.colors = o3d.utility.Vector3dVector(merged_colors)
o3d.io.write_point_cloud("final_merged_point_cloud_2.pcd", final_pcd)

Goochaozheng · 2023-11-20T06:52:23Z

Same problem as @ChaoyiZh. Is there any solution?

yixuanli98 · 2023-11-20T08:13:54Z

I am trying to fuse serval RGB and depth pairs to generate the pointclouds. Here is my core code and please feel free to point out if there is any problems. According to the instruction from YiXuan and previous post, I divided the depth with 10000 and did the same coordinate conversion. But the results seems wired.

os.environ['OPENCV_IO_ENABLE_OPENEXR'] = '1'
def depth_to_pc(depth_path, intrinsic, transform_matrix):
    depth_map = cv2.imread(depth_path, cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)[..., 0]
    pixel_size = depth_map.shape[0] * depth_map.shape[1]
    depth_image = depth_map.reshape(pixel_size)/10000
    u_coord = np.tile(np.arange(depth_map.shape[1]), (depth_map.shape[0], 1)).reshape(pixel_size)
    v_coord = np.tile(np.arange(depth_map.shape[0]), (depth_map.shape[1], 1)).T.reshape(pixel_size)
    k = np.identity(3)
    k[0, 2] = intrinsic["cx"]
    k[1, 2] = intrinsic["cy"]
    k[0, 0] = intrinsic["fl_x"]
    k[1, 1] = intrinsic["fl_x"]
    p2d = np.array([u_coord, v_coord, np.ones_like(u_coord)])
    p3d = np.transpose(np.dot(np.linalg.inv(k), p2d) * depth_image).reshape(-1, 3)

    # Left coord to right
    point_cloud = p3d
    point_cloud[:, 1] = -point_cloud[:, 1]
    ones = np.ones((point_cloud.shape[0], 1))
    homogenous_coordinate = np.hstack((point_cloud, ones))
    transformed_points = (transform_matrix @ homogenous_coordinate.T).T

    return transformed_points[:, :3]
frames = data["frames"]

for frame in frames[:10]:
    rel_img_path = frame["file_path"]
    transform_matrix = frame["transform_matrix"]
    transform_matrix = np.array(transform_matrix)



    abs_img_path = os.path.abspath(os.path.join(pose_path, rel_img_path)).replace("/pose", "")
    abs_depth_path = abs_img_path.replace("small_city", "small_city_depth",1).replace("small_city_road_down", "small_city_road_down_depth").replace("png", "exr")
    if not os.path.exists(abs_img_path) or not os.path.exists(abs_depth_path):
        continue
    depth_image = cv2.imread(abs_depth_path, cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)[..., 0]
    color_image = cv2.imread(abs_img_path)
    color_image = cv2.cvtColor(color_image, cv2.COLOR_BGR2RGB)
    point_cloud = depth_to_pc(abs_depth_path, camera_intrinsics, transform_matrix)

    all_points.append(point_cloud)

    colors = color_image.reshape(-1, 3)/255.0
    all_colors.append(colors)


merged_points = np.vstack(all_points)
merged_colors = np.vstack(all_colors)


final_pcd = o3d.geometry.PointCloud()
final_pcd.points = o3d.utility.Vector3dVector(merged_points)
final_pcd.colors = o3d.utility.Vector3dVector(merged_colors)
o3d.io.write_point_cloud("final_merged_point_cloud_2.pcd", final_pcd)

Have you successfully generated the reasonable point cloud from a single RGB-depth pairs?

yixuanli98 · 2023-11-20T08:18:29Z

Besides, could you tell me the which json file you use?

ChaoyiZh · 2023-11-20T21:05:37Z

Besides, could you tell me the which json file you use?

Yes. I totally understand we are using Nerf(OPENGL), which is a right-hand coordinate system. And I can generate one pointcloud with some image-depth pair and the corresponding pose. Here are the files I am using.
small_city/street/pose/block_A/transforms_train.json.
The result I posted before was from the fusion of first ten images.

yixuanli98 · 2023-11-21T02:07:53Z

Thx! I will try to fix this bug.

Snosixtyboo · 2023-12-09T00:21:24Z

I am trying to fuse serval RGB and depth pairs to generate the pointclouds. Here is my core code and please feel free to point out if there is any problems. According to the instruction from YiXuan and previous post, I divided the depth with 10000 and did the same coordinate conversion. But the results seems wired.

os.environ['OPENCV_IO_ENABLE_OPENEXR'] = '1'
def depth_to_pc(depth_path, intrinsic, transform_matrix):
    depth_map = cv2.imread(depth_path, cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)[..., 0]
    pixel_size = depth_map.shape[0] * depth_map.shape[1]
    depth_image = depth_map.reshape(pixel_size)/10000
    u_coord = np.tile(np.arange(depth_map.shape[1]), (depth_map.shape[0], 1)).reshape(pixel_size)
    v_coord = np.tile(np.arange(depth_map.shape[0]), (depth_map.shape[1], 1)).T.reshape(pixel_size)
    k = np.identity(3)
    k[0, 2] = intrinsic["cx"]
    k[1, 2] = intrinsic["cy"]
    k[0, 0] = intrinsic["fl_x"]
    k[1, 1] = intrinsic["fl_x"]
    p2d = np.array([u_coord, v_coord, np.ones_like(u_coord)])
    p3d = np.transpose(np.dot(np.linalg.inv(k), p2d) * depth_image).reshape(-1, 3)

    # Left coord to right
    point_cloud = p3d
    point_cloud[:, 1] = -point_cloud[:, 1]
    ones = np.ones((point_cloud.shape[0], 1))
    homogenous_coordinate = np.hstack((point_cloud, ones))
    transformed_points = (transform_matrix @ homogenous_coordinate.T).T

    return transformed_points[:, :3]
frames = data["frames"]

for frame in frames[:10]:
    rel_img_path = frame["file_path"]
    transform_matrix = frame["transform_matrix"]
    transform_matrix = np.array(transform_matrix)



    abs_img_path = os.path.abspath(os.path.join(pose_path, rel_img_path)).replace("/pose", "")
    abs_depth_path = abs_img_path.replace("small_city", "small_city_depth",1).replace("small_city_road_down", "small_city_road_down_depth").replace("png", "exr")
    if not os.path.exists(abs_img_path) or not os.path.exists(abs_depth_path):
        continue
    depth_image = cv2.imread(abs_depth_path, cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)[..., 0]
    color_image = cv2.imread(abs_img_path)
    color_image = cv2.cvtColor(color_image, cv2.COLOR_BGR2RGB)
    point_cloud = depth_to_pc(abs_depth_path, camera_intrinsics, transform_matrix)

    all_points.append(point_cloud)

    colors = color_image.reshape(-1, 3)/255.0
    all_colors.append(colors)


merged_points = np.vstack(all_points)
merged_colors = np.vstack(all_colors)


final_pcd = o3d.geometry.PointCloud()
final_pcd.points = o3d.utility.Vector3dVector(merged_points)
final_pcd.colors = o3d.utility.Vector3dVector(merged_colors)
o3d.io.write_point_cloud("final_merged_point_cloud_2.pcd", final_pcd)

I think you wanna do

point_cloud[:, 2] = -point_cloud[:, 2]

For me, that makes the points align as expected

yzslab · 2024-01-09T08:42:18Z

I am trying to fuse serval RGB and depth pairs to generate the pointclouds. Here is my core code and please feel free to point out if there is any problems. According to the instruction from YiXuan and previous post, I divided the depth with 10000 and did the same coordinate conversion. But the results seems wired.

Hi, have you solve this problem? I am trying to build point cloud too, but face the same problem as you.

yzslab · 2024-01-09T08:44:06Z

I think you wanna do
point_cloud[:, 2] = -point_cloud[:, 2]
For me, that makes the points align as expected

Hi, would you mind sharing more details about your implementation? I tried point_cloud[:, 2] = -point_cloud[:, 2] but it doesn't work.

yzslab · 2024-01-16T13:14:29Z

I think you wanna do
point_cloud[:, 2] = -point_cloud[:, 2]
For me, that makes the points align as expected

Thanks, with below operations, the points align successfully:

point_cloud[:, 1] = -point_cloud[:, 1]
point_cloud[:, 2] = -point_cloud[:, 2]

guangkaixu · 2024-02-07T01:58:32Z

Thank you very much for you question! I think the way you read depth is right. We have checked our depth map and find the reason. For the best rendering quality, we use the anti-alias setting to reduce noise, which will have a smoothing effect on the image. The depths change dramatically at the edge of the object and anti-alias reduces this change, which will cause the noisy point cloud. The depth map without and with anti-alias are shown below, which is a zoom-in pixel-level comparison.

We think the depth maps rendered with anti-lias can better align the rgb images rendered with anti-alias and can support most of the requirements, like depth guidance for reconstruction. But for generating point cloud from depth map, anti-alias will bring many noises like you find. We have tried generating point cloud from the depth rendered without anti-alias and solved the problem. We will update a new depth map version without anti-alias soon.

Hi, is there any update on the ground-truth depth map? It seems that the update date of "small_city_depth" is still 3 months ago. Besides, has the anti-alias problem been solved in the "big_city_depth"?

yixuanli98 · 2024-02-07T03:29:30Z

Thank you very much for you question! I think the way you read depth is right. We have checked our depth map and find the reason. For the best rendering quality, we use the anti-alias setting to reduce noise, which will have a smoothing effect on the image. The depths change dramatically at the edge of the object and anti-alias reduces this change, which will cause the noisy point cloud. The depth map without and with anti-alias are shown below, which is a zoom-in pixel-level comparison.
We think the depth maps rendered with anti-lias can better align the rgb images rendered with anti-alias and can support most of the requirements, like depth guidance for reconstruction. But for generating point cloud from depth map, anti-alias will bring many noises like you find. We have tried generating point cloud from the depth rendered without anti-alias and solved the problem. We will update a new depth map version without anti-alias soon.

Hi, is there any update on the ground-truth depth map? It seems that the update date of "small_city_depth" is still 3 months ago. Besides, has the anti-alias problem been solved in the "big_city_depth"?

Currently, we do not provide the depth map without anti-alias. If you want to use the depth map to guide reconstruction, I think the depth map with anti-alias is more suitable because it aligns with rgb images. If you want to use the depth map to obtain geometry, you can refer to https://github.com/city-super/MatrixCity/blob/main/MatrixCityPlugin/docs/Render-Data.md#depth to render the depth map without anti-alias, we have already provided the camera sequences, render config, plugin and detailed documentation.

yixuanli98 · 2024-04-30T07:54:50Z

we have updated the script extracting point clouds from multi rgb-depth pairs at https://github.com/city-super/MatrixCity/blob/main/scripts/rgbd2pc.py @ChaoyiZh @yzslab @Goochaozheng @Snosixtyboo

jianglh-WHU mentioned this issue Apr 2, 2024

Questions about SceneDepth to "metric depth(distance from camera center to scene)" #24

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Noise in depth gt #4

Noise in depth gt #4

BlueNine9 commented Oct 10, 2023 •

edited

Loading

yixuanli98 commented Oct 18, 2023

yixuanli98 commented Oct 22, 2023

ChaoyiZh commented Nov 15, 2023

yixuanli98 commented Nov 15, 2023

yixuanli98 commented Nov 15, 2023

ChaoyiZh commented Nov 15, 2023 •

edited

Loading

Goochaozheng commented Nov 20, 2023

yixuanli98 commented Nov 20, 2023

yixuanli98 commented Nov 20, 2023

ChaoyiZh commented Nov 20, 2023

yixuanli98 commented Nov 21, 2023

Snosixtyboo commented Dec 9, 2023

yzslab commented Jan 9, 2024

yzslab commented Jan 9, 2024

yzslab commented Jan 16, 2024

guangkaixu commented Feb 7, 2024

yixuanli98 commented Feb 7, 2024

yixuanli98 commented Apr 30, 2024

Noise in depth gt #4

Noise in depth gt #4

Comments

BlueNine9 commented Oct 10, 2023 • edited Loading

yixuanli98 commented Oct 18, 2023

yixuanli98 commented Oct 22, 2023

ChaoyiZh commented Nov 15, 2023

yixuanli98 commented Nov 15, 2023

yixuanli98 commented Nov 15, 2023

ChaoyiZh commented Nov 15, 2023 • edited Loading

Goochaozheng commented Nov 20, 2023

yixuanli98 commented Nov 20, 2023

yixuanli98 commented Nov 20, 2023

ChaoyiZh commented Nov 20, 2023

yixuanli98 commented Nov 21, 2023

Snosixtyboo commented Dec 9, 2023

yzslab commented Jan 9, 2024

yzslab commented Jan 9, 2024

yzslab commented Jan 16, 2024

guangkaixu commented Feb 7, 2024

yixuanli98 commented Feb 7, 2024

yixuanli98 commented Apr 30, 2024

BlueNine9 commented Oct 10, 2023 •

edited

Loading

ChaoyiZh commented Nov 15, 2023 •

edited

Loading