Why do you disable Sophia G

[Practicinginhell]

First of all, thank you for the amazing repo. I'm curious about why SophiaG is disabled. Thank you for any information

[dlwh]

It's a good question. It's not so much as disabled as I didn't finish bringing it into mainline from the original branch. Honestly, I wasn't really sure the best way to refactor Trainer to work with it, because Sophia-G has requirements on the structure of the loss function. I have a goal to delete/invert trainer, at which point it'll become easier.

[Practicinginhell]

Thank you for your answer. I think the code right now is a little bit hard to modify. For example, I'm adding support for gemma but I have noticed that adding a new activation function requires modifying the code below
ACT2FN: Dict[str, Callable] = { "relu": hnn.relu, "silu": hnn.silu, "swish": hnn.swish, "gelu": partial(hnn.gelu, approximate=False), "gelu_new": partial(hnn.gelu, approximate=True), "quick_gelu": hnn.quick_gelu, }
which is in GPT2.py

[dlwh]

I'm sorry if it feels that way! I don't think you need to use the act2fn thing do you? I can see the HF implementation supports it being configurable for gemma, but... I don't think you actually need to make it configurable?
But if you feel strongly about it, it should be easy to move Act2fn somewhere else?

[Practicinginhell]

Indeed, that is precisely what I'm undertaking. I'm restructuring some folder arrangements. And thank you for the fantastic repository; I've gained a lot of knowledge from it.