BUG when testing results of MTEB retrieval

[tianyumyum]

I was running mteb run -m McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp-supervised -t DBPedia ArguAna NFCorpus FiQA2018 --output_folder results_0 --device 0

But I got：
File "/mnt/ceph_home/zhongtianyun2023/miniconda3/envs/mteb_new/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
AttributeError: 'LlamaBiModel' object has no attribute 'rotary_emb'
Batches: 0%| | 0/1 [00:07<?, ?it/s]
[W729 12:01:59.244093981 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

[vaibhavad]

Hi @tianyumyum,

what version of transformers are you using? llm2vec supports transformers>=4.39.1,<=4.40.2

[lst627]

I encountered the same issue yesterday while using this model. It started working after I downgraded the transformers to version 4.40.2.

[tianyumyum]

Thanks！
I can run it now～@vaibhavad @lst627

[tianyumyum]

@vaibhavad Hello！ I have created a new conda environment with all the packages as in this issue.

I didn't get an error when using the command line, but I got an error Bus error (core dumped) when using python files.
I just ran the same code as you listed above in this site

import mteb

if __name__ == "__main__":
    model_name = "McGill-NLP/LLM2Vec-Sheared-LLaMA-mntp-supervised"
    model = mteb.get_model(model_name)
    tasks = mteb.get_tasks(tasks=["AmazonCounterfactualClassification"])
    evaluation = mteb.MTEB(tasks=tasks)
    results = evaluation.run(model, output_folder="results")

The bug will happen when running experiments/mteb_eval.py too.