Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Community] Add MagicTailor Personalization Training Script #9923

Open
1 of 2 tasks
clarencechen opened this issue Nov 13, 2024 · 0 comments
Open
1 of 2 tasks

[Community] Add MagicTailor Personalization Training Script #9923

clarencechen opened this issue Nov 13, 2024 · 0 comments

Comments

@clarencechen
Copy link
Contributor

Model/Pipeline/Scheduler description

Recent advancements in fine-tuning techniques for text-to-image (T2I) personalization still struggle to distill visual concepts from reference images when there are both image-wide and spatially localized concepts present in each reference image.

The techniques in this paper are designed to improve component-controllable personalization, a novel task that pushes the boundaries of T2I models by allowing users to reconfigure specific components when personalizing visual concepts. This task is particularly challenging due to two primary obstacles: semantic pollution, where unwanted visual elements corrupt spatially localized concepts, and semantic imbalance, which causes disproportionate learning between the custom image-wide concept and spatially localized component concepts.

To overcome these challenges, MagicTailor leverages Dynamic Masked Degradation (DM-Deg) to dynamically perturb undesired visual semantics and Dual-Stream Balancing (DS-Bal) to establish a balanced learning paradigm for desired visual semantics.

Open source status

  • The model implementation is available.
  • The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Paper: https://arxiv.org/pdf/2410.13370
Project Website: https://correr-zhou.github.io/MagicTailor/
Code: https://github.com/correr-zhou/MagicTailor
Contact: @Correr-Zhou

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants