Training Log 2022-06-22

[zh-zheng]

CPM-Live Training Log (June, 22)

Time: June, 22 2022 16:00

Recorder: @zh-zheng

Loss

• Begin: 2.39
• End: 2.53

Completed Data

• $\approx$ 256.99GB

Average Grad Norm

• 0.81

Progress

• 27.59%

Comment

We have noticed that training loss is gradually increasing over the past day. In order to figure out the reason, we checked the code and found a bug in the dataloader🧐. The bug results in our model learning from the same data after each resumption of training. Since the model has seen new data in the past day, it is acceptable for the loss to increase.