mcmapply memory error #9
Comments
|
probably a nested loop of some sort -- you may need to replace mcmapply --t On Sun, May 15, 2016 at 3:34 PM, Asif Zubair [email protected]
|
|
this is the exact invocation: command = paste("bash bamToBigWig_dedup.sh", bam.files)
tmp <- mcmapply(system, command = command)I was trying to call a script in parallel for converting |
|
better to take longer and complete successfully than to quickly fail :-) --t On Sun, May 15, 2016 at 10:36 PM, Asif Zubair [email protected]
|
|
also, this is happening when i have |
|
true. gc(,T) sometimes helps (run it 10 times if you're going to run it --t On Mon, May 16, 2016 at 1:20 AM, Asif Zubair [email protected]
|
|
For now, I'm going to do the following: rm(list=setdiff(ls(),
c("test", "dump_dir", "output_dir", "dedup", "bam.files",
"compute_library_complexity", "generate_BigWigs", "do_motif_analysis",
"is.sig", "diffAccRanges")))
gc()Once the significant regions have been called, I really only need the |
|
re: PWM: just make sure it's using openMP and not forking a zillion processes :-) --t On Mon, May 16, 2016 at 1:14 PM, Asif Zubair [email protected]
|
|
I think this might be the case:
|
|
gah, might want to skip it. People can wait longer if the bottleneck is on --t On Mon, May 16, 2016 at 5:04 PM, Asif Zubair [email protected]
|
|
this is still failing, despite memory clean up, when |
|
it looks like you are still parallelizing it at process level -- maybe just --t On Tue, May 17, 2016 at 12:39 AM, Asif Zubair [email protected]
|
|
I did try that, but even systems calls are failing: |
|
You may need to print out memory usage ( gc(TRUE, TRUE) will do it) along --t On Tue, May 17, 2016 at 9:23 AM, Asif Zubair [email protected]
|
|
60gb. Yes, I read somewhere that it is always advisable to call |
|
this is what |
|
this is bizarre... I'm not sure why an 8GB RAM footprint is segfaulting on --t On Tue, May 17, 2016 at 12:40 PM, Asif Zubair [email protected]
|
|
Personally, I think we should check with Jay about the instance. He told me that it is a |
|
this is even more weird, R is using |
|
agreed; you could also use system("free") to find out. --t On Tue, May 17, 2016 at 5:15 PM, Asif Zubair [email protected]
|
|
memory leak somewhere? try system("free") before and after running motif enrichment analysis --t On Tue, May 17, 2016 at 9:15 PM, Asif Zubair [email protected]
|
|
I'm not calling |
|
probably better anyways... test if the file exists && mv it --t On Wed, May 18, 2016 at 11:29 AM, Asif Zubair [email protected]
|
|
@ttriche what do you think about using the # ===================================================
# This function collects garbage
collectGarbage=function(){while (gc()[2,4] != gc()[2,4] | gc()[1,4] != gc()[1,4]){}} |
|
handy! It's not "supposed" to be needed for the GC but I do this by hand --t On Tue, Jul 19, 2016 at 1:27 PM, Asif Zubair [email protected]
|
|
you might want to consider the software SnakeMake which can actively On Tue, Jul 19, 2016 at 2:07 PM, Tim Triche, Jr. [email protected]
|
|
not clear how this solves the problem of running out of RAM -- if a forked On the other hand, for (e.g.) reassembling mt genomes, snakeMake could --t On Tue, Jul 19, 2016 at 3:00 PM, Anthony R. Colombo <
|
|
is this caused by / when / during sliding window calculation. from reading these messages, I'm not sure if this is caused by the window calculation, what exactly is the system() calling, more specifically what process is being computed in the algorithm if it is the sliding window, one cause may that in order for the downstream analysis to begin, one must wait for all the sliding windows to complete their calculations, storing a genome wide set of windows with computations. if this is true, then one solution may be to calculate the first k windows, push into a result(k), then calculate the k+1 windows and calculate result(k2) = result(k) + results(k+1), etc. then you can clear the memory from windows at the first k at each iteration. oooh this probably makes no sense. AC |
|
the other idea, assuming it is during a sliding window calculation, is that if the sliding_window at position k computations = alpha. then move to sliding_window computation result at position k+beta = gamma. if alpha is somewhat equivalent to gamma, then do not store gamma, but skip it. |
|
I call all my commands from inside the Rmarkdown, so I don't think I need SnakeMake. I eliminated all However, the App still crashes in the 8vs8 analysis and I am sort of running out of ideas. It needs to allocate a 8gb vector (when Could it be possible that R is not using all the memory ? I could possibly increase it by using A last question is. Do you folks know how to interpret this output of Doesn't this mean that R is using 600Mb of memory ? Thanks! |
|
It looks like this is due from Rsam Sent from my iPhone
|
|
The place it breaks is during |
|
One possibility (something that has happened in the past on HPC) is that It may be time to ask Aaron (csaw) and/or Martin (Rsamtools) about this As a bonus, Dan Tenenbaum sets up and runs AMIs and Docker images all the --t On Thu, Jul 28, 2016 at 3:40 AM, Asif Zubair [email protected]
|
|
also -- yes, it looks like R has allocated under 2GB (1932.132MB to be Try two things:
unlimited, for me
will verify what sort of image you really haveLast but not least, make sure you're using a 64-bit R image. --t On Thu, Jul 28, 2016 at 3:40 AM, Asif Zubair [email protected]
|
|
You might be able to use the mean value theorem. Instead of looking at a global filter you look at intervals along the global position. Now if you assume you have a continuous function, for every interval there exists a A Beta and a. Mu such that f(beta) = mu. Where mu is the meanover this interval. Then the property of means are additive. This can reduce a global filter into a segmented filter over smaller regions. I'm not sure if this helps at all Sent from my iPho
|
|
thank you, Anthony, but the function has been implemented in thank you, Tim. I checked that I am using 64-bit R. I'll also chat with Aaron about it. |
|
no prob. interesting for me to see what you've had problems with On Thu, Jul 28, 2016 at 11:42 AM, Asif Zubair [email protected]
|
|
This is the output from the following commands: output: any ideas why is the total memory used 40G when Also, I am going to switch default behaviour to |
|
Seems like a plan... I can't quite figure out what's using 40GB baseline, --t On Thu, Jul 28, 2016 at 6:22 PM, Asif Zubair [email protected]
|
|
but the R |
|
should I run a profiler ? does R have one ? |
|
R does have a profiler (several, actually; --t On Thu, Jul 28, 2016 at 8:48 PM, Asif Zubair [email protected]
|
|
Hi Folks, I might have this wrong but is it suspicious that the This also happens when I do a copy inside the docker container at the commandline: Is this behaviour expected ? These are the file sizes: Thanks! |
|
Also, is it suspicious that the Amazon EC2 instance has 0 swap ? @arcolombo , do you have this problem when you copy files to the scratch directory ? |
|
no an EC2 instance isn't going to have swap because it's pointless -- file.copy can eat up a tremendous amount of memory DURING the copy (run top --t On Fri, Jul 29, 2016 at 3:29 PM, Asif Zubair [email protected]
|
|
Thank you, Tim. That clears things up. The reason we did the copy is that we wanted to re-index the bam files. The ones generated by ILMN's bwa aligner were not working for some reason. I think copying large files is tricky. After the copy operation i don't see RAM usage go down. :/ |
|
oh crap, that's right. I wonder -- can you simply index them from wherever --t On Fri, Jul 29, 2016 at 4:07 PM, Asif Zubair [email protected]
|
|
Yes, that's brilliant. Simply link the bam files and then index!!! Thank you, Tim. Will try it out. |
|
Hmm, it doesn't seem to work. This from this seems pretty weird. |
|
This is bizarre. Seems like it has to be some sort of an FS issue -- is --t On Fri, Jul 29, 2016 at 7:41 PM, Asif Zubair [email protected]
|
|
Sorry, the linking wasn't causing the memory usage, it was the indexing. However, I checked that the indexs from the Aligner App are also working. I'll rejig the App so that those are now used directly and we don't have to copy/link bams. Also, it seems that other people have run into R memory issues. Please see this SO. I think the solution is to divide the Rmarkdown into different R files. But I think that will have to wait. I want to work on the App Note for now. |
|
Good plan. FWIW I'm submitting a separate app note today. If there are --t On Sat, Jul 30, 2016 at 9:24 PM, Asif Zubair [email protected]
|
When doing the 3vs8 analysis with default settings, getting the following error:
The text was updated successfully, but these errors were encountered: