Update lora_conversion_utils.py

[zhaowendao30]

x-flux single-blocks lora load

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[zhaowendao30]

I check the x-flux code, the single block lora should load after qkv attention，not norm,they are just has the same shape

[sayakpaul]

@raulmosa could you give this a look?

[raulmosa]

I've checked it and it's right, it shouldn't be norm. Looks good to me @sayakpaul .
Nice catch @zhaowendao30 , thanks! =)

[sayakpaul]

@zhaowendao30 thanks for your contribnutions!

Could you also do a side-by-side comparison of your changes applied and without your changes in the outputs? That would be very much appreciated.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zhaowendao30]

I've checked it and it's right, it shouldn't be norm. Looks good to me @sayakpaul . Nice catch @zhaowendao30 , thanks! =)

No thanks =)

[zhaowendao30]

@zhaowendao30 thanks for your contribnutions!

Could you also do a side-by-side comparison of your changes applied and without your changes in the outputs? That would be very much appreciated.

OK, the single block i just trained index 1,2,3,4, and the first image is load in qkv, the second is load in norm, the last is no lora

[sayakpaul]

@zhaowendao30 thanks, but please prefer not using human subjects in the public forums.

I will run the tests today and update the slices as needed because of the change.

[zhaowendao30]

@zhaowendao30 thanks, but please prefer not using human subjects in the public forums.

I will run the tests today and update the slices as needed because of the change.

OK, I've uploaded it again. However, this is a LoRA about people, and the panda looks a bit strange. =）

[sayakpaul]

Just ran

pytest tests/lora/ -k "test_flux_xlabs"

test_flux_xlabs is passing and test_flux_xlabs_load_lora_with_single_blocks is failing because of hardware change, which is expected. I will change the slices in https://github.com/huggingface/diffusers/pull/9845/files.

[sayakpaul]

Thanks!