WorkflowError in file /opt/conda/lib/python3.9/site-packages/hippunfold/workflow/Snakefile, line 7: Config file must be given as JSON or YAML with keys at top level.

[VAMMGA]

I am a beginner of HippUnfold, I met some problems in the use and have not been effectively solved, I downloaded the container version of HippUnfold and wrote a script to run different tasks. However, during the running process, I found that running 50 tasks would fail 1 or 2 tasks, running the 1 or 2 tasks alone is no problem. However, I adjusted different tasks by using different calculation directories and output directories, but the situation still exists, and the errors are not the same. The multi-test experiment test did not find an effective solution. I would appreciate it if you could help me to have a look. I am still looking forward to your reply.
【error1】
INFO: Converting SIF file to temporary sandbox...
WorkflowError in file /opt/conda/lib/python3.9/site-packages/hippunfold/workflow/Snakefile, line 7:
Config file must be given as JSON or YAML with keys at top level.
File "/opt/conda/lib/python3.9/site-packages/hippunfold/workflow/Snakefile", line 7, in
INFO: Cleaning up image...

other error:

It can be confirmed that there is no problem in the calculation examples, and errors will be reported in separate calculation. It is not clear what content conflicts lead to abnormal operation and different errors will appear. I often get this error when I try to split the calculation between different directories.
【error2】
[Mon May 20 16:08:07 2024]
Error in rule cp_template_to_unfold:
jobid: 124
input: /opt/conda/lib/python3.9/site-packages/hippunfold/workflow/../resources/unfold_template_hipp/tpl-avg_space-unfold_den-1mm_midthickness.surf.gii
output: work/sub-N0013/ses-N00131/surf/sub-N0013_ses-N00131_hemi-R_space-unfold_den-1mm_label-hipp_midthickness.surf.gii
shell:
cp /opt/conda/lib/python3.9/site-packages/hippunfold/workflow/../resources/unfold_template_hipp/tpl-avg_space-unfold_den-1mm_midthickness.surf.gii work/sub-N0013/ses-N00131/surf/sub-N0013_ses-N00131_hemi-R_space-unfold_den-1mm_label-hipp_midthickness.surf.gii && wb_command -set-structure work/sub-N0013/ses-N00131/surf/sub-N0013_ses-N00131_hemi-R_space-unfold_den-1mm_label-hipp_midthickness.surf.gii CORTEX_RIGHT -surface-type FLAT -surface-secondary-type MIDTHICKNESS
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
【other errors】


【Sub.sh script】
#!/bin/bash
#SBATCH -p v6_384
#SBATCH -N 1
#SBATCH -n 64
#SBATCH --exclusive

module load singularity/3.9.9
module load anaconda/3.7.3-zyq
export PATH=/public1/wshome/ws221/sch0205/xumf/sci0048/install/bin:$PATH
export Brain_dir=/public1/wshome/ws221/sch0205/xumf/sci0048/BNU_HY_CBIRD
export out_dir=/public1/wshome/ws221/sch0205/xumf/sci0048/test1/output_BNU_HY_CBIRD
export tool_dir=/public1/wshome/ws221/sch0205/xumf/sci0048
start_time=$(date +%s)
mkdir ${out_dir}
doit() {
Subj_id="${1}"
if [ ! -f ${out_dir}/work/sub-${Subj_id}_work.tar.gz ]; then
rm -r ${out_dir}/work/sub-${Subj_id}
rm -r ${out_dir}/hippunfold/sub-${Subj_id}
echo "Processing ${Subj_id}"
singularity run --bind ${Brain_dir}:/bids,${out_dir}:/output -e ${tool_dir}/khanlab_hippunfold_latest.sif /bids /output participant --modality T1w --cores 64 --participant-label ${Subj_id} --output-density 1mm
Fi
export -f doit
Listfile=${1}
cat ${Listfile} |parallel -j 5 --tmpdir /public1/wshome/ws221/sch0205/xumf/sci0048/tmp doit
end_time=$(date +%s)
echo "It took $(($end_time - $start_time)) seconds for the seg of hipp..."

【Subm_jobs.sh script】
#!/bin/bash
job_info_file="job_info.txt"
rm -f "$job_info_file"
touch "$job_info_file"
sublist_dir=/public1/wshome/ws221/sch0205/xumf/sci0048/sublist_BNU_HY_CBIRD maxjobs=5
i=1 #新的任务需要从1开始
while (( ${i} < 3 )); do #总人数除以5，向上取整+1
JobN=$(squeue -u sci0048 | grep -c " R ")
if [ "$JobN" -lt "$maxjobs" ]; then
echo "正在运行的作业数量 ($JobN) 小于 $maxjobs，可以提交新作业。"
Listfile=${sublist_dir}/Subjlist_${i}

rm -f ${Listfile}

Start=$[i5-4]
Stop=$[i5]
sed -n "${Start},${Stop}p" ${sublist_dir}/Subjlist_all_BX >> ${Listfile} #需要修改，保留最后一行的回车
timestamp=$(date +%Y-%m-%d\ %H:%M:%S)
job_id=$(sbatch sub.sh ${Listfile}| awk '{print $NF}')
echo "Submitted job $job_id at $timestamp"
echo "JobID: $job_id, Submission Time: $timestamp" >> "$job_info_file"
i=$[i+1]
fi
sleep 20
done
echo "All jobs submitted."

[akhanf]

Looks lile the problem is in how you have setup your wrapper. By running multiple runs at once in the same folder you are creating conflicts. Hippunfold supports all the cluster execution profiles supported by snakemake, so there is a slurm executor that you can use instead. So you dont need the extra script you wrote. If you want multiple subjects to run on each job you can use --group-components subj=5 to achieve that. Have a look at the hippunfold docs for running on a cluster and also snakemake docs (snakemake version <8.0).

[Izaackk]

Hi,

I am running HippUnfold with Singularity. Are the options --cluster and --group-components available? If not, do I need to compile the source code to use these options?

Thank you!

Looks lile the problem is in how you have setup your wrapper. By running multiple runs at once in the same folder you are creating conflicts. Hippunfold supports all the cluster execution profiles supported by snakemake, so there is a slurm executor that you can use instead. So you dont need the extra script you wrote. If you want multiple subjects to run on each job you can use --group-components subj=5 to achieve that. Have a look at the hippunfold docs for running on a cluster and also snakemake docs (snakemake version <8.0).

[jordandekraker]

Yes they are available on both Singularity and Docker container version of hippunfold!