no sd model found?

[hexxter]

how can i set the sd model? It is null?

without i get this error:

find model: /home/alex/KI/ComfyUI/custom_nodes/ComfyUI-Hallo/pretrained_models/face_analysis/models/scrfd_10g_bnkps.onnx detection [1, 3, '?', '?'] 127.5 128.0
set det-size: (640, 640)
No faces detected in the image. Using the entire image as the face region.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723142528.486053   13763 gl_context_egl.cc:85] Successfully initialized EGL. Major : 1 Minor: 5
I0000 00:00:1723142528.543733   13830 gl_context.cc:357] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 555.42.06), renderer: NVIDIA GeForce RTX 4070/PCIe/SSE2
W0000 00:00:1723142528.544131   13763 face_landmarker_graph.cc:174] Sets FaceBlendshapesGraph acceleration to xnnpack by default.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
W0000 00:00:1723142528.547903   13838 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
W0000 00:00:1723142528.555137   13847 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
#face is invalid: 0
Traceback (most recent call last):
  File "/home/alex/KI/ComfyUI/custom_nodes/ComfyUI-Hallo/Hallo/scripts/inference.py", line 375, in <module>
    inference_process(command_line_args)
  File "/home/alex/KI/ComfyUI/custom_nodes/ComfyUI-Hallo/Hallo/scripts/inference.py", line 162, in inference_process
    source_image_lip_mask = image_processor.preprocess(
  File "/home/alex/KI/ComfyUI/custom_nodes/ComfyUI-Hallo/Hallo/hallo/datasets/image_processor.py", line 138, in preprocess
    get_mask(source_image_path, cache_dir, face_region_ratio)
  File "/home/alex/KI/ComfyUI/custom_nodes/ComfyUI-Hallo/Hallo/hallo/utils/util.py", line 439, in get_mask
    get_lip_mask(landmarks, height, width, os.path.join(
  File "/home/alex/KI/ComfyUI/custom_nodes/ComfyUI-Hallo/Hallo/hallo/utils/util.py", line 391, in get_lip_mask
    lip_landmarks = np.take(landmarks, lip_ids, 0)
  File "/home/alex/KI/ComfyUI/venv/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 192, in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
  File "/home/alex/KI/ComfyUI/venv/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 59, in _wrapfunc
    return bound(*args, **kwds)
IndexError: cannot do a non-empty take from an empty axes.

[yueboxinli]

model download:

https://huggingface.co/fudan-generative-ai/hallo

[emircanerkul]

@yueboxinli I already did, i can see all my omnx data within https://github.com/deepinsight/insightface/releases/download/v0.7/antelopev2.zip

But still having same issue.

The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
2024-10-30 08:18:15.3583885 [E:onnxruntime:Default, provider_bridge_ort.cc:1992 onnxruntime::TryGetProviderInfo_CUDA] D:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1637 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\Users\emirc\ComfyUI\venv\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: C:\Users\emirc\ComfyUI\custom_nodes\ComfyUI-Hallo\pretrained_models\face_analysis\models\1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
2024-10-30 08:18:15.8852737 [E:onnxruntime:Default, provider_bridge_ort.cc:1992 onnxruntime::TryGetProviderInfo_CUDA] D:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1637 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\Users\emirc\ComfyUI\venv\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: C:\Users\emirc\ComfyUI\custom_nodes\ComfyUI-Hallo\pretrained_models\face_analysis\models\2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
2024-10-30 08:18:15.9234412 [E:onnxruntime:Default, provider_bridge_ort.cc:1992 onnxruntime::TryGetProviderInfo_CUDA] D:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1637 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\Users\emirc\ComfyUI\venv\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: C:\Users\emirc\ComfyUI\custom_nodes\ComfyUI-Hallo\pretrained_models\face_analysis\models\genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
2024-10-30 08:18:16.0792130 [E:onnxruntime:Default, provider_bridge_ort.cc:1992 onnxruntime::TryGetProviderInfo_CUDA] D:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1637 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\Users\emirc\ComfyUI\venv\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: C:\Users\emirc\ComfyUI\custom_nodes\ComfyUI-Hallo\pretrained_models\face_analysis\models\glintr100.onnx recognition ['None', 3, 112, 112] 127.5 127.5
2024-10-30 08:18:16.6692921 [E:onnxruntime:Default, provider_bridge_ort.cc:1992 onnxruntime::TryGetProviderInfo_CUDA] D:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1637 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\Users\emirc\ComfyUI\venv\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: C:\Users\emirc\ComfyUI\custom_nodes\ComfyUI-Hallo\pretrained_models\face_analysis\models\scrfd_10g_bnkps.onnx detection [1, 3, '?', '?'] 127.5 128.0
set det-size: (640, 640)
No faces detected in the image. Using the entire image as the face region.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
W0000 00:00:1730265496.857607   16348 face_landmarker_graph.cc:174] Sets FaceBlendshapesGraph acceleration to xnnpack by default.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
W0000 00:00:1730265496.866764   15024 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
W0000 00:00:1730265496.878018   15024 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
#face is invalid: 0
Traceback (most recent call last):
  File "C:\Users\emirc\ComfyUI\custom_nodes\ComfyUI-Hallo\Hallo\scripts\inference.py", line 375, in <module>
    inference_process(command_line_args)
  File "C:\Users\emirc\ComfyUI\custom_nodes\ComfyUI-Hallo\Hallo\scripts\inference.py", line 162, in inference_process
    source_image_lip_mask = image_processor.preprocess(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\emirc\ComfyUI\custom_nodes\ComfyUI-Hallo\Hallo\hallo\datasets\image_processor.py", line 138, in preprocess
    get_mask(source_image_path, cache_dir, face_region_ratio)
  File "C:\Users\emirc\ComfyUI\custom_nodes\ComfyUI-Hallo\Hallo\hallo\utils\util.py", line 439, in get_mask
    get_lip_mask(landmarks, height, width, os.path.join(
  File "C:\Users\emirc\ComfyUI\custom_nodes\ComfyUI-Hallo\Hallo\hallo\utils\util.py", line 391, in get_lip_mask
    lip_landmarks = np.take(landmarks, lip_ids, 0)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\emirc\ComfyUI\venv\Lib\site-packages\numpy\core\fromnumeric.py", line 192, in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\emirc\ComfyUI\venv\Lib\site-packages\numpy\core\fromnumeric.py", line 59, in _wrapfunc
    return bound(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^
IndexError: cannot do a non-empty take from an empty axes.
Prompt executed in 7.61 seconds

Also same ss

[xliry]

Quote reply
Referenc

fixed with that way

class HalloNode:
@classmethod
def INPUT_TYPES(s):
def get_model_list():
model_path = os.path.join(ckpt_dir, "stable-diffusion-v1-5")
if os.path.exists(model_path):
model_file = "diffusion_pytorch_model.safetensors"
if os.path.exists(os.path.join(model_path, model_file)):
return [model_file]
return ["diffusion_pytorch_model.safetensors"]

    return {
        "required": {
            "source_image": ("IMAGEPATH",),
            "driving_audio":("AUDIOPATH",),
            "pose_weight" :("FLOAT",{
                "default": 1.
            }),
            "face_weight":("FLOAT",{
                "default": 1.
            }),
            "lip_weight":("FLOAT",{
                "default": 1.
            }),
            "face_expand_ratio":("FLOAT",{
                "default": 1.2
            })
        },
        "optional":{
            "sd_model": (get_model_list(),),
        }
    }

RETURN_TYPES = ("VIDEO",)
FUNCTION = "generate"
CATEGORY = "AIFSH_Hallo"
OUTPUT_NODE = True

def generate(self,source_image,driving_audio,pose_weight,face_weight,lip_weight,face_expand_ratio,sd_model=None):
    python_exec = sys.executable or "python"
    infer_py = os.path.join(now_dir,"Hallo","scripts","inference.py")
    default_yaml_path = os.path.join(now_dir,"Hallo","configs","inference","default.yaml")
    with open(default_yaml_path, 'r', encoding="utf-8") as f:
        yaml_data = yaml.load(f.read(),Loader=yaml.SafeLoader)
        
    yaml_data['save_path'] = os.path.join(output_dir, "hallo")
    yaml_data['audio_ckpt_dir'] = os.path.join(ckpt_dir,"hallo")
    
    model_base = os.path.join(ckpt_dir, "stable-diffusion-v1-5")
    if sd_model and os.path.exists(os.path.join(model_base, sd_model)):
        yaml_data['base_model_path'] = model_base
    else:
        yaml_data['base_model_path'] = os.path.join(ckpt_dir, "stable-diffusion-v1-5")
    
    print(f"Using model path: {yaml_data['base_model_path']}")
    
    yaml_data['motion_module_path'] = os.path.join(ckpt_dir,"motion_module","mm_sd_v15_v2.ckpt")
    yaml_data['face_analysis']['model_path'] = os.path.join(ckpt_dir,"face_analysis")
    yaml_data['wav2vec']['model_path'] = os.path.join(ckpt_dir,"wav2vec","wav2vec2-base-960h")
    yaml_data['audio_separator']['model_path'] = os.path.join(ckpt_dir,"audio_separator","Kim_Vocal_2.onnx")
    yaml_data['vae']['model_path'] = os.path.join(ckpt_dir,"sd-vae-ft-mse")
    
    yaml_data['unet_config'] = {
        "sample_size": 64,
        "in_channels": 4,
        "out_channels": 4,
        "down_block_types": ["DownBlock2D", "DownBlock2D", "DownBlock2D", "DownBlock2D"],
        "up_block_types": ["UpBlock2D", "UpBlock2D", "UpBlock2D", "UpBlock2D"],
        "block_out_channels": [128, 256, 512, 512],
        "layers_per_block": 2,
        "cross_attention_dim": 768
    }

    tmp_yaml_path = os.path.join(now_dir,'tmp.yaml')
    with open(tmp_yaml_path,'w', encoding="utf-8") as f:
        yaml.dump(data=yaml_data,stream=f,Dumper=yaml.Dumper)

    outfile = os.path.join(output_dir,f"hallo_{time.time_ns()}.mp4")
    os.environ["face_landmarker"] = os.path.join(ckpt_dir,"face_analysis","models","face_landmarker_v2_with_blendshapes.task")
    cmd = f"""{python_exec} {infer_py} --config "{tmp_yaml_path}" --source_image "{source_image}" --driving_audio "{driving_audio}" --output {outfile} --pose_weight {pose_weight} --face_weight {face_weight} --lip_weight {lip_weight} --face_expand_ratio {face_expand_ratio}"""
    
    print(cmd)
    os.system(cmd)
    os.remove(tmp_yaml_path)
    return (outfile, )

[emircanerkul]

@xliry please create pull request