You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, I met the following ImportError during generation evaluations.
generations were saved at ./output_humaneval/humaneval_humaneval.json
Evaluating generations...
Traceback (most recent call last):
File "/root/software/bigcode-evaluation-harness/main.py", line 414, in
main()
File "/root/software/bigcode-evaluation-harness/main.py", line 398, in main
results[task] = evaluator.evaluate(
File "/root/software/bigcode-evaluation-harness/bigcode_eval/evaluator.py", line 107, in evaluate
results = task.process_results(generations, references)
File "/root/software/bigcode-evaluation-harness/bigcode_eval/tasks/humaneval.py", line 98, in process_results
results, _ = compute_code_eval(
File "/root/software/bigcode-evaluation-harness/bigcode_eval/tasks/custom_metrics/code_eval.py", line 154, in compute_code_eval
result = future.result()
File "/root/miniconda3/envs/bigcode/lib/python3.9/concurrent/futures/_base.py", line 433, in result
return self.__get_result()
File "/root/miniconda3/envs/bigcode/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/root/miniconda3/envs/bigcode/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/root/software/bigcode-evaluation-harness/bigcode_eval/tasks/custom_metrics/execute.py", line 36, in check_correctness
manager = multiprocessing.Manager()
File "/root/miniconda3/envs/bigcode/lib/python3.9/multiprocessing/context.py", line 55, in Manager
from .managers import SyncManager
ImportError: cannot import name 'SyncManager' from partially initialized module 'multiprocessing.managers' (most likely due to a circular import) (/root/miniconda3/envs/bigcode/lib/python3.9/multiprocessing/managers.py)
Here is my script:
'''
#!/bin/bash
taskname=humaneval
mkdir -p output_$taskname
Hi, I met the following ImportError during generation evaluations.
generations were saved at ./output_humaneval/humaneval_humaneval.json
Evaluating generations...
Traceback (most recent call last):
File "/root/software/bigcode-evaluation-harness/main.py", line 414, in
main()
File "/root/software/bigcode-evaluation-harness/main.py", line 398, in main
results[task] = evaluator.evaluate(
File "/root/software/bigcode-evaluation-harness/bigcode_eval/evaluator.py", line 107, in evaluate
results = task.process_results(generations, references)
File "/root/software/bigcode-evaluation-harness/bigcode_eval/tasks/humaneval.py", line 98, in process_results
results, _ = compute_code_eval(
File "/root/software/bigcode-evaluation-harness/bigcode_eval/tasks/custom_metrics/code_eval.py", line 154, in compute_code_eval
result = future.result()
File "/root/miniconda3/envs/bigcode/lib/python3.9/concurrent/futures/_base.py", line 433, in result
return self.__get_result()
File "/root/miniconda3/envs/bigcode/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/root/miniconda3/envs/bigcode/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/root/software/bigcode-evaluation-harness/bigcode_eval/tasks/custom_metrics/execute.py", line 36, in check_correctness
manager = multiprocessing.Manager()
File "/root/miniconda3/envs/bigcode/lib/python3.9/multiprocessing/context.py", line 55, in Manager
from .managers import SyncManager
ImportError: cannot import name 'SyncManager' from partially initialized module 'multiprocessing.managers' (most likely due to a circular import) (/root/miniconda3/envs/bigcode/lib/python3.9/multiprocessing/managers.py)
Here is my script:
'''
#!/bin/bash
taskname=humaneval
mkdir -p output_$taskname
python /root/software/bigcode-evaluation-harness/main.py
--model /data00/models/meta-llama_Meta-Llama-3.1-8B-Instruct/
--trust_remote_code
--tasks $taskname
--limit 8
--max_length_generation 512
--temperature 0.8
--do_sample True
--n_samples 16
--batch_size 16
--precision fp16
--allow_code_execution
--save_generations
--save_generations_path ./output_$taskname/$taskname.json
'''
Could you share some tips on how to solve this problem?
The text was updated successfully, but these errors were encountered: