My implementation of the Gemma LLM.

Training data.

a) All English Wikipedia pages(6.5 million).

b) ~2 billion tokens.

a)RMS Normalization

b)ROPE Embedding

c)MultiQueryAttention

d)GeGLU Activations

e)Pre-Norm Transformers

a) 2 Million parameters

b) Contextual length of 64 tokens.

Name		Name	Last commit message	Last commit date
Latest commit History 89 Commits
.gitignore		.gitignore
LICENSE		LICENSE
LLM.ipynb		LLM.ipynb
Mixture_of_Experts.ipynb		Mixture_of_Experts.ipynb
README.md		README.md
backup_code.py		backup_code.py
config.py		config.py
data_loader.py		data_loader.py
data_preprocessing.ipynb		data_preprocessing.ipynb
gemma_model.ipynb		gemma_model.ipynb
gemma_train.ipynb		gemma_train.ipynb
generate.py		generate.py
main.py		main.py
model.py		model.py
rank_tensor_base_on_length.ipynb		rank_tensor_base_on_length.ipynb
requirements.txt		requirements.txt
train.py		train.py
utils.py		utils.py