Training Not Fitting Properly and Plot Display Issues

[alittlebitfun]

I noticed that when I used this repository, the example workflow couldn't achieve fitting. Even when I used the same parameters as I did in Kohya, the training results were still very poor. Despite repeatedly adjusting the parameters, I was still unable to achieve proper fitting.

Additionally, although the loss value changes normally, the plot often fails to display the line correctly and instead shows a horizontal straight line. It would also be great if the image inference during training could support weight adjustments.

[Andreas-Marx]

The example workflow does three training sessions (the blueish "LORA training xxx" groups). I even have expanded to four of these.

I then ran into problems, because I was tempted to have a first training session that trains the LoRA into a usable quality (450 steps), then three more sessions with 50-70 steps to find the sweet spot.
Turns out, that the shorter sessions actually decrease quality.

I think, this is due to cosine_with_restarts doing lr_scheduler_num_cycles restarts per training session.
You really should go for a reasonable number of epochs in each session. For 2 restarts, there should be at least 10 epochs.

Reduce the window-size parameter in the "Visualize Loss" nodes to see what's actually going on. I always set this value to average over roughly one epoch.