methylation calling issue on cluster

[wynstep]

I open this issue for flagging a problem I have when I run gemBS (in a singularity package with all dependencies satisfied) on a HPC cluster.
Saying more than 900 samples to be analysed, 850 succeed, 50 fails at the methylation calling step. No errors, no warnings, just killed.
If I relaunch the 50 that failed, let's say 40 succeed and 10 fail again. Then I launch the remaining for the third time and all are processed correctly.

It's obvious the issue is not related to the samples (otherwise they would have failed every time), so what could be the issue? Any suggestion?

Thanks

[wynstep]

@heathsc

This is the error that recurrently appears when it fails:
----------- Methylation Calling -------- : Reference : /usr/local/bin/ToTEM//src/ucsc.hg19.gemBS.ref : Species : Homo sapiens : Right Trim : 0 : Left Trim : 0 : Chromosomes : ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chrX', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr20', 'chrY', 'chr19', 'chr22', 'chr21', '@pool_1', '@pool_2'] : Threads : 4 : Sample: LNCaP_CelLin_BL_d0_BStg_R2_from_LNCaP-BL_diluted_0-065_41545.mapped.nodups Bam: /scratch/scratch/regmspi/results/totem/cellLine//LNCaP_CelLin_BL_d0_BStg_R2_from_LNCaP-BL_diluted_0-065_41545/LNCaP_CelLin_BL_d0_BStg_R2_from_LNCaP-BL_diluted_0-065_41545.mapped.nodups.bam : : Methylation Calling... 2020-11-06 20:57:12,735 ERROR: Process '/usr/local/lib/python3.8/dist-packages/gemBS/gemBSbinaries/bs_call' finished with 1 2020-11-06 20:57:12,736 ERROR: Additional threads: 2 1 1 2020-11-06 20:57:12,736 ERROR: Loading reference sequence index 2020-11-06 20:57:12,736 ERROR: Sequence index read in successfully 2020-11-06 20:57:12,736 ERROR: Opened /scratch/scratch/regmspi/results/totem/cellLine//LNCaP_CelLin_BL_d0_BStg_R2_from_LNCaP-BL_diluted_0-065_41545/LNCaP_CelLin_BL_d0_BStg_R2_from_LNCaP-BL_diluted_0-065_41545.mapped.nodups.bam for input (Index) 2020-11-06 20:57:12,736 ERROR: Processing region chr13:1-115169878 2020-11-06 20:57:12,736 ERROR: Processing chromosome chr13 (OK) 2020-11-06 20:57:12,736 ERROR: Loading reference for chr13 2020-11-06 20:57:12,736 ERROR: len = 115169878 2020-11-06 20:57:12,736 ERROR: Read in 96149878 bases on chr13 from 19020001 - 115169878, stored in 38459954 bytes 2020-11-06 20:57:12,736 ERROR: > System.Error::Signal raised (no=11) Exception in thread Thread-1: ValueError: Error while executing the bscall process. : Methylation call done, samples performed: LNCaP_CelLin_BL_d0_BStg_R2_from_LNCaP-BL_diluted_0-065_41545.mapped.nodups

Could you please help?