Training Loss Values.

[mmderakhshani]

Hi, thank you for releasing this GitHub repository. I am trying to reproduce the stage 1 training on ImageNet. Could you please share the W&B log or let me know the initial and final loss values for that stage? I am getting the following loss, and it turns out the model is not converging. Thanks.

[Sierkinhane]

Hi, thanks for your interest. Your loss curve looks a bit weird. When I trained the first stage, the loss would quickly reduce to around 8, and then converge to around 7 (500k steps).

[Sierkinhane]

What's the resolution, are you training on 256x256?

[mmderakhshani]

I am training with 256x256 resolution. Can you tell at which training steps at stage 1, you are getting meaningful sampled images in class conditional manner?

One more thing, can you also tell me what is your total batch size and number of training steps in stage 1?

[Sierkinhane]

Here are the generated samples at 10k iterations. The total batch size of t2i, mmu, and language modeling is 1152, and I guess the batch size of t2i is around 700.

[mmderakhshani]

Thank you for your prompt response! I really appreciate it. Could you clarify what the generated images will look like at the end of Stage 1? I assume that during Stage 1, you are only using ImageNet and RefinedWeb. Is that correct?

…

On Tue, 12 Nov 2024 at 09:19, Jinheng Xie ***@***.***> wrote: Here are the generated samples at 10k iterations. The total batch size of t2i, mmu, and language modeling is 1152, and I guess the batch size of t2i is around 700. image.png (view on web) <https://github.com/user-attachments/assets/95f83469-a300-45ff-a63f-8c0f6c92ed5d> — Reply to this email directly, view it on GitHub <#50 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD7OT66RINXDI2QQIQRYSW32AG2ZDAVCNFSM6AAAAABRSVLPYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRZHA3DQMRWGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Sierkinhane]

mmu loss is also used in this stage. Here are samples generated without CFG.

[mmderakhshani]

Could you please clarify whether these samples were generated after the training was completed? Additionally, could you let me know how you adjusted the learning rate? Did you use a method similar to YOLO-style initialization, or did you conduct several experiments to determine the best approach? I want to increase the batch size and train a larger model, so I would appreciate some hints about the initialization of the learning rate.

[Sierkinhane]

"These samples were generated at 499,000 iterations. For the learning rate and adjustments, we just follow Muse[1] and you can refer to the configs, and we didn't put much effort into these configurations.

[1] muse: text-to-image generation via masked generative transformers

[Sierkinhane]

BTW, welcome to star our project :)

[mmderakhshani]

Happy to star! :D