Rendering the Exact Same Image Using transforms.json (Converted to cameras.json by the Viewer)

[sonoda-git]

Hello!

I want to find the transformation between transforms.json (generated by ns-process-data) and cameras.json (generated by ns-viewer). Even though I manually select a pose in the viewer, add a keyframe, and ns-render camera-path, the resulting image appears different because the field of view (FOV) does not match exactly (I guess). Therefore, I want to convert transforms.json directly into the cameras.json format.

I prefer not to use ns-render dataset since I want to add additional poses to transforms.json and render them directly.

in specific, transforms.json

{
    "w": 1920,
    "h": 1080,
    "fl_x": 1673.37194824,
    "fl_y": 1673.37194824,
    "cx": 960.0,
    "cy": 540.0,
    "k1": 0.0,
    "k2": 0.0,
    "p1": 0.0,
    "p2": 0.0,
    "camera_model": "OPENCV",
    "frames": [
        {
            "file_path": "seq2/frame00231.png",
            "transform_matrix": [
                [
                    0.4779116355791738,
                    -0.03771821539522755,
                    -0.8775978698743648,
                    -12.459449675450125
                ],
                [
                    0.01748035686323424,
                    0.9992884202970155,
                    -0.03342889202950022,
                    2.0382813036301997
                ],
                [
                    0.8782340929983059,
                    0.0006352311126985094,
                    0.47823099921246043,
                    5.551989619681693
                ],
                [
                    0.0,
                    0.0,
                    0.0,
                    1.0
                ]
            ],
            "colmap_im_id": 1
        }]
}

should be converted to format of cameras.json (not exact value. i think rotation is correct, however translation does not match to original translation of transforms.json)

{
    "default_fov": 75.0, # should be changed
    "default_transition_sec": 2.0,
    "keyframes": [
        {
            "matrix": [
                0.47539108991622925,
                -0.04053268954157829,
                -0.878840446472168,
                -0.40117236971855164
            ,

                -0.8796073794364929,
                -0.0024145529605448246,
                -0.47569477558135986,
                -0.37228038907051086
            ,

                0.017159029841423035,
                0.9991755485534668,
                -0.036800578236579895,
                0.011988549493253231,
                0.0,
                0.0,
                0.0,
                1.0
            ],
            "fov": 75.0, # should be changed
            "aspect": 1.7777777777777777,
            "override_transition_enabled": false,
            "override_transition_sec": null
        }
    ],
    "camera_type": "perspective",
    "render_height": 1080.0,
    "render_width": 1920.0,
    "fps": 30.0,
    "seconds": 2.0,
    "is_cycle": false,
    "smoothness_value": 0.0,
    "camera_path": [
        {
            "camera_to_world": [
                0.47539108991622925,
                -0.04053268954157829,
                -0.878840446472168,
                -0.40117236971855164
            ,
            
                -0.8796073794364929,
                -0.0024145529605448246,
                -0.47569477558135986,
                -0.37228038907051086
            ,
            
                0.017159029841423035,
                0.9991755485534668,
                -0.036800578236579895,
                0.011988549493253231,

                0.0,
                0.0,
                0.0,
                1.0
            ],
            "fov": 75.0, # should be changed
            "aspect": 1.7777777777777777
        }
    ]
}

dataparser_transform.json

{
    "transform": [
        [
            0.9999960660934448,
            -2.6524455734033836e-06,
            -0.002804776420816779,
            4.475800037384033
        ],
        [
            -2.6524455734033836e-06,
            0.9999982118606567,
            -0.0018913720268756151,
            -1.9000475406646729
        ],
        [
            0.002804776420816779,
            0.0018913720268756151,
            0.9999942779541016,
            -1.7533822059631348
        ]
    ],
    "scale": 0.15030615222924348
}

i guess,

• default_fov and fov should be same, and calculated using 2atan(width/2fx)
• and render_width and render_height should be same to w and h
• then convert transforms.json to camera_path.json (also using dataparser_transform.json) B = TAs
  : I referred this page

however, rendered result does not match that I imagined. what's the problem?

[lyg1597]

Hope this answer helps with your question #3484 (comment)