Add resize_token_embeddings feature

[ccdv-ai]

⚠️ Please check that this feature request hasn't been suggested before.

• I searched previous Ideas in Discussions didn't find any similar feature requests.
• I searched previous Issues didn't find any similar feature requests.

🔖 Feature description

Add the option to resize the token embeddings:
PreTrainedModel has this method.

✔️ Solution

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(model_id)
model.resize_token_embeddings(num_tokens)

❓ Alternatives

No response

📝 Additional Context

No response

Acknowledgements

• My issue title is concise, descriptive, and in title casing.
• I have searched the existing issues to make sure this feature has not been requested yet.
• I have provided enough information for the maintainers to understand and evaluate this request.

[NanoCode012]

May I ask what is the use case? We currently resize to the tokenizer's length (or multiple of 32 to it if enabled).

[ccdv-ai]

Can be usefull for a tokenizer/custom tokenizer and vocab_size mismatch.
For instance, there is a mismatch for all qwen 2.5 models where they padded the embedding layer for distributed training.

[NanoCode012]

@ccdv-ai , just to clarify, would you want a config that lets you specify the new tokenizer vocab size or just to resize?

Axolotl does the latter* under the hood when you add new tokens.

axolotl/src/axolotl/utils/models.py

Lines 1039 to 1053 in 8c3a727

	embeddings_len = (
	math.ceil(len(self.tokenizer) / 32) * 32
	if self.cfg.resize_token_embeddings_to_32x
	else len(self.tokenizer)
	)
	if (
	hasattr(self.model, "get_input_embeddings")
	and self.model.get_input_embeddings().num_embeddings < embeddings_len
	):
	resize_kwargs = {}
	if self.cfg.mean_resizing_embeddings is not None:
	resize_kwargs["mean_resizing"] = self.cfg.mean_resizing_embeddings
	self.model.resize_token_embeddings(embeddings_len, **resize_kwargs)
	else:
	self.model.tie_weights()

If you enable resize_token_embeddings_to_32x: true, it will resize to next multiple of 32.

[ccdv-ai]

@NanoCode012 Only the option to choose and resize the token_embeddings to an arbitrary value.
For example, Qwen 2.5 7B tokenizer has 151665 tokens but the embedding layer has 152064.
resize_token_embeddings_to: 151665 should be possible.

if self.cfg.resize_token_embeddings_to < len(self.tokenizer):
    #Warning or stop
self.model.resize_token_embeddings(self.cfg.resize_token_embeddings_to, **resize_kwargs)

[NanoCode012]

@ccdv-ai thanks for clarifying. To add on to your point, we already do resize to tokenizer.

# above code but summarized

embeddings_len = len(self.tokenizer) 

if ( 
     self.model.get_input_embeddings().num_embeddings < embeddings_len 
 ): 
     self.model.resize_token_embeddings(embeddings_len)

For resizing to another value (!=len(self.tokenizer) ) , I'm not sure I understand the use case as the tokenizer would then mismatch the embedding length and cause an error during training.