[PR] Saving Inferred 3D Hand Keypoints

[Kaszanas]

As the name suggests.

There were some unused imports in the demo.pl.

As described in the issues below this should calculate the 3D keypoints and save them with np.save.

Related:

• How to output each coordinate information as .pkl #86
• Question: Obtaining Bones and Joints #88

If you can verify and see if this is the solution for the issues above @geopavlakos.

[dhingratul]

The concatenation doesn't work fyi.

[Kaszanas]

The concatenation doesn't work fyi.

I am just now testing the solution. Do you know how to make it work?

[Kaszanas]

I think this works now @dhingratul I am not aware of the precise mechanism of how these values should be mutated mathematically. but given the description

You can get the 3D coordinates of the hand keypoints in the camera frame by adding pred_cam_t_full (which is calculated here) to out['pred_keypoints_3d'] (which is calculated here). Then you can export the sum in a pkl file.

I think this is how it is supposed to work.

specifically this piece of code handles this addition for each of the keypoints:

                    # Iterating over all of the samples in the batch:
                    for i in range(len(pred_keypoints_3d)):
                        batch_element_keypoints = pred_keypoints_3d[i]
                        batch_element_cam_t = pred_cam_t_full[i]

                        for keypoint_idx in range(len(batch_element_keypoints)):
                            # Adding the camera translation to the keypoints:
                            batch_element_keypoints[keypoint_idx] += batch_element_cam_t

Some input from @geopavlakos would surely be appreciated. I am also pretty sure that this can be done with some simpler numpy calls, but then the understanding of the dimensionality is lost for me.

Apparently this could be:

pred_keypoints_3d += pred_cam_t_full[:, np.newaxis, :]

But it would be great to get a review from someone.

[dhingratul]

I am still getting around this, but did you visualize and check ?

[Kaszanas]

I am still getting around this, but did you visualize and check ?

Didn't have time (or the knowledge), I've stopped at running through debugger and seeing the results as numbers.

Shapes matched. I think (2,3) and (2,21,3). So adding an empty axis and elementwise adding the last dimension makes sense?

[Kaszanas]

I am still getting around this, but did you visualize and check ?

Based on the discussion in #88 (comment), I think we need some input from @geopavlakos @dhingratul

[dhingratul]

So, i visualized the kps, and something seems off with the translation of the points with this current solution. If/when I find a solution, i can post. Otherwise the keypoints generated seem coherent (except for translation to the original image plane)

This works for me (partially):
pred_keypoints_3d = out['pred_keypoints_3d'][n].detach().cpu().numpy() + pred_cam_t_full[n, np.newaxis, :]

[ManaLight]

@dhingratul, could you please share how you visualized the keypoints please?

@Kaszanas, I want to import the keypoints to blender for hand animation. Could you please advise?

Thanks!

[Kaszanas]

@dhingratul, could you please share how you visualized the keypoints please?

@Kaszanas, I want to import the keypoints to blender for hand animation. Could you please advise?

Thanks!

Sure thing, you need to load the file using a script with the Blender scripting capabilities somehow:

• https://numpy.org/doc/stable/reference/generated/numpy.load.html
• https://docs.blender.org/api/current/info_quickstart.html
• https://docs.blender.org/api/current/info_advanced_blender_as_bpy.html

then you need to load the model somehow as well :D
I am not familiar with the specifics as I have not worked that much with Blender and Python at the same time.

If you figure it out let me know @ManaLight

Some steps that you might take:

1. Install Blender either fully or as a module
2. Install numpy into the environment where you have the Blender Python module.
3. load the keyopints from the file that you get hamer
4. load the model from the file that you get from hamer

Note that keypoints are supposedly in the OpenPose order: https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/02_output.md#hand-output-format