Skip to content

Commit

Permalink
fix sft bug caused by exemplar
Browse files Browse the repository at this point in the history
tldr exemplar is a schema for data. I was storing the sequences and lengths with np arrays as schema objects and one of them had dim 0 which is a scalar and invalid for arrays
  • Loading branch information
ahmeda14960 committed Oct 29, 2024
1 parent 5343096 commit fd39828
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion src/levanter/data/text.py
Original file line number Diff line number Diff line change
Expand Up @@ -725,7 +725,8 @@ def mk_supervised_dataset(config: LMSupervisedDatasetConfig, tokenizer: PreTrain
input_field = config.input_field
output_field = config.output_field

output_exemplar = {"input_ids": np.zeros((0,), dtype=np.int32), "sources_len": np.zeros((), dtype=np.int32)}

output_exemplar = {"input_ids": np.zeros((0,), dtype=np.int32), "sources_len": np.zeros((0,), dtype=np.int32)}

# Use the same preprocessing as before
dataset = dataset.map_batches(
Expand Down

0 comments on commit fd39828

Please sign in to comment.