In Animalpose top-down-inference, Single bbox is OK, but bboxes don't work.

[rukaeto]

Hello.
Thank you for releasing nice projects.

I want to solve this title question.

I'm using YOLOv4 as bbox-detect model and pretrained hrnet_w48_animalpose_256x256 as pose_model.
Afterwards, I use inference_top_down_pose_model(pose_model, img_or_path, person_results=[detected-bboxes of YOLOv4]...) in mmpose.apis.inference.py by CPU-only.

This img_or_path is an image_path of 4 cows image.
So, detected-bboxes of YOLOv4 contain 4 bboxes[x1,y1,x2,y2, prob].

Unfortunately, this inference doesn't work.

So, I changed _inference_single_pose_model as below.

def _inference_single_pose_model(
...
...
...

with torch.no_grad():
   result = model(img=batch_data['img'][0].unsqueeze(dim=0), img_metas=[batch_data['img_metas'][0]], return_loss=False, return_heatmap=False)

...

In short, I gave Tensor[1, 3, 256, 256] as img.
This inference results is OK.

Do you know cause of this problem?

---

[My enevironment MEMO]
M1 mac BigSur
python: 3.8.12
pytorch: 1.9.0
mmpose: 0.20.0
gcc: Apple clang version 12.0.5

[ly015]

Hi, please check the following demo code from demo/top_down_img_demo_with_mmdet.py and make sure that nothing is missing in your code. For example, you would need to call process_mmdet_results before feeding it to inference_top_down_pose_model.

mmpose/demo/top_down_img_demo_with_mmdet.py

Lines 93 to 115 in 4a3b12a

	mmdet_results = inference_detector(det_model, image_name)
	# keep the person class bounding boxes.
	person_results = process_mmdet_results(mmdet_results, args.det_cat_id)
	# test a single image, with a list of bboxes.
	# optional
	return_heatmap = False
	# e.g. use ('backbone', ) to return backbone feature
	output_layer_names = None
	pose_results, returned_outputs = inference_top_down_pose_model(
	pose_model,
	image_name,
	person_results,
	bbox_thr=args.bbox_thr,
	format='xyxy',
	dataset=dataset,
	dataset_info=dataset_info,
	return_heatmap=return_heatmap,
	outputs=output_layer_names)

[rukaeto]

Thank you for your answer.

I try your advice, but inference don't work.
I used mmpose/demo/top_down_img_demo_with_mmdet.py but doesn't work.

pose_results as below.

[{'bbox': array([8.2493518e+02, 4.1235556e+02, 1.2136844e+03, 8.0789642e+02,
         9.9961936e-01], dtype=float32),
  'keypoints': array([[764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan]], dtype=float32)},
 {'bbox': array([455.3737   , 363.28598  , 820.494    , 791.87994  ,   0.9995347],
        dtype=float32),
  'keypoints': array([[361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
...
...

However, In google colab, MMPose_Tutorial.ipynb is OK.

In short, My local environment problem...?

If the number of object is 1, pose-inference is OK. (the number of bboxes is 1)
The number of object is 2,3,4... , pose-inference doesn't work. (the number of bboxes is 2,3,4...)

Are there another methods to solve this problem?

[ly015]

Is there any error information when you use a detection result with multiple boxes?

[rukaeto]

YOLOv4 object-detection result is OK.
YOLOv4 properly detected multiple objects.

---

Good inference image as below.

---

Bad inference image as below.

This time,

person_result
[{'bbox': array([8.2493518e+02, 4.1235556e+02, 1.2136844e+03, 8.0789642e+02,
         9.9961936e-01], dtype=float32)},
 {'bbox': array([455.3737   , 363.28598  , 820.494    , 791.87994  ,   0.9995347],
        dtype=float32)},
 {'bbox': array([ 16.834986 , 330.94113  , 185.2066   , 634.6386   ,   0.9984503],
        dtype=float32)},
 {'bbox': array([214.1364    , 349.70105   , 408.8373    , 644.5446    ,
           0.99291915], dtype=float32)}]

pose_results
[{'bbox': array([8.2493518e+02, 4.1235556e+02, 1.2136844e+03, 8.0789642e+02,
         9.9961936e-01], dtype=float32),
  'keypoints': array([[764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan],
         [764.2268, 355.043 ,      nan]], dtype=float32)},
 {'bbox': array([455.3737   , 363.28598  , 820.494    , 791.87994  ,   0.9995347],
        dtype=float32),
  'keypoints': array([[361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan],
         [361.54712, 301.19623,       nan]], dtype=float32)},
 {'bbox': array([ 16.834986 , 330.94113  , 185.2066   , 634.6386   ,   0.9984503],
        dtype=float32),
  'keypoints': array([[-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan],
         [-94.866264, 286.90283 ,        nan]], dtype=float32)},
 {'bbox': array([214.1364    , 349.70105   , 408.8373    , 644.5446    ,
           0.99291915], dtype=float32),
  'keypoints': array([[121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan],
         [121.30641, 306.94238,       nan]], dtype=float32)}]

In short, object-detection step is OK. On the other hand, pose-estimate step result is bad.

[ly015]

Could you please check that an image path is assigned to image_name of inference_top_down_pose_model, instead of a np.array or a tensor?

mmpose/demo/top_down_img_demo_with_mmdet.py

Lines 106 to 115 in 4a3b12a

	pose_results, returned_outputs = inference_top_down_pose_model(
	pose_model,
	image_name,
	person_results,
	bbox_thr=args.bbox_thr,
	format='xyxy',
	dataset=dataset,
	dataset_info=dataset_info,
	return_heatmap=return_heatmap,
	outputs=output_layer_names)

[rukaeto]

The argument [image_name] is image_path like (./data/input.png).

I tried np.array img as [image_name].

np_img = cv2.imread(image_path)

However, the inference result doesn't change.

pose_result

[{'bbox': array([8.2686163e+02, 4.1394363e+02, 1.2164086e+03, 8.1262982e+02,
         9.9833554e-01], dtype=float32),
  'keypoints': array([[764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ],
         [764.52484, 356.17645,   2.     ]], dtype=float32)},
 {'bbox': array([451.35822  , 356.3029   , 825.75305  , 789.80786  ,   0.9961522],
        dtype=float32),
  'keypoints': array([[359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan],
         [359.0036, 293.5033,      nan]], dtype=float32)},
 {'bbox': array([212.66995   , 358.31082   , 417.30667   , 633.789     ,
           0.99613416], dtype=float32),
  'keypoints': array([[137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan],
         [137.28949, 318.3511 ,       nan]], dtype=float32)},
 {'bbox': array([  9.786472 , 333.62576  , 195.6419   , 632.14667  ,   0.9929786],
        dtype=float32),
  'keypoints': array([[-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan],
         [-89.836395, 290.33566 ,        nan]], dtype=float32)}]

---

The inference result of object only-one (result is good)

[ly015]

Sorry, we could reproduce this problem in our environments. We tried the exact image and pose model, and the result is alright. Could you please provide the entire command you used which may help us locate the problem?

[rukaeto]

I tried this code on Jupyter Lab of conda environment.

python: 3.8.12
torch: 1.9.0
mmpose: 0.20.0
gcc: Apple clang version 12.0.5 (clang-1205.0.22.11)

import cv2
from mmpose.apis import (inference_top_down_pose_model, init_pose_model,
                         vis_pose_result, process_mmdet_results)
from mmdet.apis import inference_detector, init_detector
from tqdm.notebook import tqdm as tqdm

pose_config = './mmpose/configs/animal/2d_kpt_sview_rgb_img/topdown_heatmap/animalpose/hrnet_w48_animalpose_256x256.py'
pose_checkpoint = './hrnet_w48_animalpose_256x256-34644726_20210426.pth'
det_config = './mmpose/demo/mmdetection_cfg/faster_rcnn_r50_fpn_coco.py'
det_checkpoint = 'faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth'

# initialize pose model
pose_model = init_pose_model(pose_config, pose_checkpoint, device='cpu')
# initialize detector
det_model = init_detector(det_config, det_checkpoint, device='cpu')

img = 'input_cow.png'

# inference detection
mmdet_results = inference_detector(det_model, img)

# cat_id:20[cow]
person_results = process_mmdet_results(mmdet_results, cat_id=20) 

# inference pose
pose_results, returned_outputs = inference_top_down_pose_model(pose_model,
                                                               img,
                                                               person_results,
                                                               bbox_thr=0.3,
                                                               format='xyxy',
                                                               dataset=pose_model.cfg.data.test.type)

# show pose estimation results
vis_result = vis_pose_result(pose_model,
                             np_img,
                             pose_results,
                             dataset=pose_model.cfg.data.test.type,
                             show=False)

# show result-img
cv2_imshow(vis_result)

input_cow.png

[eric-clutch]

Same happens to me; seems like it's an issue happening on Apple M1 chip (using MacBook Pro w/ M1 PRO). Running the demo on a fresh install of mmpose on the latest release:

python demo/top_down_img_demo.py \
    configs/body/2d_kpt_sview_rgb_img/topdown_heatmap/coco/hrnet_w48_coco_256x192.py \
    https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth \
    --img-root tests/data/coco/ --json-file tests/data/coco/test_coco.json \
    --out-img-root vis_results \
    --device=cpu

Provides different outputs from the Apple M1 than running on Ubuntu 18 (where it is correct). If there's only one bbox, result is alright in M1, otherwise only the detected boxes are returned.

[eric-clutch]

Quick fix: when there are more than 1 bbox in an image, do inference one by one and then append the results together.

[rukaeto]

This code is OK even M1 chip.
But inference time is longer.

def _inference_single_pose_model(model,
                                 img_or_path,
                                 bboxes,
                                 dataset='TopDownCocoDataset',
                                 dataset_info=None,
                                 return_heatmap=False):

...
...
result = []
with torch.no_grad():
    for i in range(len(batch_data['img'])):
         one_result = model(
               img=batch_data['img'][i].unsqueeze(dim=0),
                img_metas=[batch_data['img_metas'][i]],
                return_loss=False,
                return_heatmap=return_heatmap)
         result.append(one_result['preds'][0])
result = np.array(result)

[eric-clutch]

Yes, we are in the same situation.. My guess is that M1 can't handle batched inputs / a list of bboxes. Anyway, M1 inference speed is super slow everywhere, so probably not the best choice to use this as a development tool with PyTorch.