StableLM Alpha Testing Issue

[MarkSchmidty]

@lhl could you shed some light on your testing regimen in this StableLM issue? Stability-AI/StableLM#30

The results seem unbelievably poor. It's shocking that a 7B GPT-NeoX trained on 800B tokens would score below GPT-2 700M and well below GPT-J, even if it is an early training checkpoint.

[MarkSchmidty]

Thanks!

[lhl]

You can see https://github.com/AUGMXNT/llm-experiments/blob/main/01-lm-eval.md to get an idea of how it's done. Feel free to test yourself. You'll want to use hf-causal as the model.