[Non-issue] Model Available at Replicate

[xiankgx]

Dear author, thanks for your wonderful effort in creating this repo.

I've pushed this model on the Replicate website so that people can try it out easily.

Here is the model link on Replicate: https://replicate.com/xiankgx/lipsick

[xiankgx]

Using the examples from Video Retalking,

source video:
https://github.com/OpenTalker/video-retalking/blob/main/examples/face/1.mp4

driving audio:
https://github.com/OpenTalker/video-retalking/blob/main/examples/audio/1.wav

Here is the sample output with auto_mask enabled and others at default.
https://github.com/Inferencer/LipSick/assets/4113258/c686e8ea-8594-4bf7-8c3a-9c3ffa2c3b0f

[Inferencer]

oh nice i think im moving over to https://github.com/KwaiVGI/LivePortrait
although its image animation its truly sota in terms of quality and speed

[tonyabracadabra]

oh nice i think im moving over to https://github.com/KwaiVGI/LivePortrait although its image animation its truly sota in terms of quality and speed

LivePortrait looks really good, I suppose there are already many audio-driven 3D portrait animation algorithms available, by streaming the output from one of these algorithms and piping it into LivePortrait, we could potentially achieve real-time image animation, provided we handle the end-to-end streaming process correctly?

[tonyabracadabra]

Dear author, thanks for your wonderful effort in creating this repo.

I've pushed this model on the Replicate website so that people can try it out easily.

Here is the model link on Replicate: https://replicate.com/xiankgx/lipsick

thanks this looks really good, however the cold boosting of replicate was really one of the issue blocked me from using any of these "serverless" gpu platforms, I wonder if you have better ideas on how to serve these models, modal.com might be one that kinda fixed the problem but I am not sure about that neither.

[xiankgx]

Is LivePortrait for lip sync?

[Inferencer]

Is LivePortrait for lip sync?

yeah but it's video driven so you could use LipSick results as the driving video in LivePortrait providing you crop to 512x512 or just record footage of yourself to drive it, did a good vid with it here
https://youtube.com/shorts/hSdzJ4Orl8s

[Inferencer]

Dear author, thanks for your wonderful effort in creating this repo.
I've pushed this model on the Replicate website so that people can try it out easily.
Here is the model link on Replicate: https://replicate.com/xiankgx/lipsick

thanks this looks really good, however the cold boosting of replicate was really one of the issue blocked me from using any of these "serverless" gpu platforms, I wonder if you have better ideas on how to serve these models, modal.com might be one that kinda fixed the problem but I am not sure about that neither.

I think runpod has near instant cold boots

[tonyabracadabra]

Dear author, thanks for your wonderful effort in creating this repo.
I've pushed this model on the Replicate website so that people can try it out easily.
Here is the model link on Replicate: https://replicate.com/xiankgx/lipsick

thanks this looks really good, however the cold boosting of replicate was really one of the issue blocked me from using any of these "serverless" gpu platforms, I wonder if you have better ideas on how to serve these models, modal.com might be one that kinda fixed the problem but I am not sure about that neither.

I think runpod has near instant cold boots

Oh you meant their serverless worker or on demand pod? I’m also considering modal.com and would love to know people’s empirical experiences with these platforms.

[justswim]

how do i use the replicate model? it doesn't seem like it allows any inputs