discriminator loss goes to infinity

[065294847]

Hi,

I'm trying to train the cvivit on a set of 10000 images. The vae-loss keeps going down, but the discriminator loss keeps rising infinity. It's easy to fool :)

Any idea what the problem is?

[lucidrains]

hmm maybe 10k images is not enough

are you doing any augmentations?

[065294847]

No augmentations.

[miguelvalente]

Check the gradients and output values at every step of the way to see where it starts to go wrong.

[065294847]

Thanks. It starts to jump dramatically already at step 500.

[lucidrains]

@065294847 what is your batch size and learning rate?

[065294847]

Just the defaults, batch size 4, lr 3e-4

[miguelvalente]

Try it with a dataset that's known to work and converge. Just to rule it out.

But try to log the grads and weights across steps to have a better understanding.

[065294847]

Thanks, I'll experiment a bit and report back

[065294847]

Thanks, I'll experiment a bit and report back

[lucidrains]

@065294847 try lowering your learning rate, or increase your effective batch size (either increase batch_size or grad_accum_every)

[lucidrains]

@065294847 get more data too, 10k is nothing... try 100k or a million. if that isn't possible, do some basic augmentations

[065294847]

Yes of course, just wanted to do a quick test first :)