Training Log 2022-06-12

[zh-zheng]

CPM-Live Training Log (June, 12)

Time: June, 12 2022 16:00

Recorder: @zh-zheng

Loss

• Begin: 2.90
• End: 2.68

Completed Data

• $\approx$ 108.38GB

Average Grad Norm

• 2.28

Progress

• 11.64%

Comment

The training has been resumed at 2 a.m. Today is a big day, as we have made the following changes to the model:

• According to our training plan, we increased the maximum input length from 512 to 1024, expecting the model to gradually learn to handle longer sequences.
• Optimizer states will also be saved to facilitate continual training from a checkpoint.
• We implemented a dynamic data expansion mechanism, so data can be added without interrupting training now!

It looks like the loss is dropping rapidly this time.