Can't Do Depth-Supervision in NeRFStudio

[aidris823]

Hello:

I have a dataset of RGB images and their corresponding depth images. I first process the data in COLMAP using:

ns-process-data images --data ./Tractor_RGB_Depth/TractorRGB/ --output-dir Processed_Tractor_RGB_Depth

After this, I manually moved the depth maps (which are a depths folder followed by versions I downsampled in Python). This is what my Processed_Tractor_RGB_Depth folder looks like:

(nerfstudio) bash-4.4$ tree -d

.
├── colmap
│   └── sparse
│       └── 0
├── depths
├── depths_2
├── depths_4
├── depths_8
├── images
├── images_2
├── images_4
└── images_8

I then try to run:

Why is depths-path giving an error and saying it doesn't recognize it, but then proceeds to suggest running depths-path? What is going on?

[mpEsri]

Did you have the "depth_file_path" for each frame in the transforms.json?, e.g.,

"frames": [
        {
            "file_path": "images/1_top.png",
            "depth_file_path": "images/1_top_depth.png",
            "transform_matrix": [
                [
                    0.9674752470564001,
                    -0.24428335846855864,
                    -0.06570606599454189,
                    0.5244950340632486
                ],
                [
                    -0.24400542080376744,
                    -0.9696952760161552,
                    0.01234610425716215,
                    0.11477608703373846
                ],
                [
                    -0.06673080961245557,
                    0.004088086015976274,
                    -0.9977626404116322,
                    -0.08361680439408765
                ],
                [
                    0.0,
                    0.0,
                    0.0,
                    1.0
                ]
            ]
        },

I had depth and rgb images in the same folder and it worked for me.
see https://docs.nerf.studio/quickstart/data_conventions.html#depth-images

[Irving87]

Did you have the "depth_file_path" for each frame in the transforms.json?, e.g.,

"frames": [
        {
            "file_path": "images/1_top.png",
            "depth_file_path": "images/1_top_depth.png",
            "transform_matrix": [
                [
                    0.9674752470564001,
                    -0.24428335846855864,
                    -0.06570606599454189,
                    0.5244950340632486
                ],
                [
                    -0.24400542080376744,
                    -0.9696952760161552,
                    0.01234610425716215,
                    0.11477608703373846
                ],
                [
                    -0.06673080961245557,
                    0.004088086015976274,
                    -0.9977626404116322,
                    -0.08361680439408765
                ],
                [
                    0.0,
                    0.0,
                    0.0,
                    1.0
                ]
            ]
        },

I had depth and rgb images in the same folder and it worked for me. see https://docs.nerf.studio/quickstart/data_conventions.html#depth-images

May I ask how to automatically add “depth_file_path” to transforms.json? Is there any scripts in nerfstudio? I can only find ns-process-data command in nerfstudio docs...

[mpEsri]

Take a look at this Python script: https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/process_data/polycam_utils.py
Function def polycam_to_json() should give you a good starting point to automate it.

[Irving87]

ould give you a good starting

Thanks