Code

Resource optimisation#82

Code

Merged

muffato merged 50 commits intodevsanger-tol/readmapping:devfrom resource_optimisationsanger-tol/readmapping:resource_optimisationCopy head branch name to clipboardDec 18, 2023

+320-350

Commits

Commits on Dec 7, 2023

• Collect the read counts from the input files

  muffato

  committed
  2109bed

• Collect the size of the genome (file size is a good proxy)

  muffato

  committed
  e2e8ead

• Simply use the whole base name to name the channel

  muffato

  committed
  11678fc

• Deal with genomes > 4 Gbp

  Show description for 84caa10
  muffato

  committed
  84caa10

• Update the read count for SAMTOOLS_MERGE

  muffato

  committed
  8dca0f3

• Some values are large and need 64 bits

  muffato

  committed
  8c67daf

• Wrong information

  muffato

  committed
  e432d89

• Updated the error codes to the latest template versions. Covers all LSF exit codes

  muffato

  committed
  ae6942b

• Estimate the resource requirements based on the size of the inputs

  muffato

  committed
  46e9910

• bugfix: minimap2 uses the decimal system and understands floating point values !

  muffato

  committed
  5f2d786

• The output of SAMTOOLS_MERGE is sorted

  muffato

  committed
  35a646d

• Logically, SAMTOOLS_MERGE should happen in the calling sub-workflow

  muffato

  committed
  04b2660

• Skip SAMTOOLS_MERGE if there is a single file

  muffato

  committed
  f9b2763

• Explain why there is no SAMTOOLS_SORT

  muffato

  committed
  fb1cd6d

• Replaced the markduplicate workflow by a single module / bash pipeline

  muffato

  committed
  c00eee6

• The runtime of samtools sort depends on the number of reads

  muffato

  committed
  5c39de5

• Updated requirements for SAMTOOLS_SORMADUP

  muffato

  committed
  ff9ff03

• The MINIMAP2_ALIGN includes SAMTOOLS_SORT. Need some extra memory related to the number of reads

  muffato

  committed
  d54d46f

• In my latest tests, it seems BWAMEM2_MEM memory usage is correlated with the logarithm of the genome size

  muffato

  committed
  582bedd

• I don't need samtools cat

  muffato

  committed
  d884d47

• Alignment is nice

  muffato

  committed
  a581ae1

• Also use -I to decrease the memory requirement of MINIMAP2_ALIGN

  muffato

  committed
  d141493

• 20 minutes seems a bit too close to the real usage. We may want a larger buffer

  muffato

  committed
  613c4cb

• Don't increase the number of CPUs too high as there are diminishing returns

  muffato

  committed
  bc844bc

• Some samtools commands have a fixed memory usage per thread, so include the CPU count in the memory requirement

  muffato

  committed
  ec695e6

• There was supposed to be a +1 there, to provide the "* task.attempt" mechanism

  muffato

  committed
  d8043a5

• Introduced a helper method that clearly shows how the logarithm is modified

  muffato

  committed
  647a748

• Increasing the number of attempts to give more resilience

  muffato

  committed
  d0b305a

• New formula that is less greedy

  muffato

  committed
  2465d74

• Adjusted resource requirements for SAMTOOLS_SORMADUP

  muffato

  committed
  a9ec952

• Added a note about minimap2

  muffato

  committed
  a249a76

• Added some credit

  muffato

  committed
  99d289d

• More consistent comment

  muffato

  committed
  9afa763

• Tell it's a meta map

  muffato

  committed
  eae3261

• Indentation should be a multiple of 4

  muffato

  committed
  f211fb4

• SAMTOOLS_FLAGSTAT may take more time

  muffato

  committed
  5b70604

• typo

  muffato

  committed
  0c5e4f9

• Increased runtime, just in case

  muffato

  committed
  c27dd30

• Updated runtime and memory requirements

  muffato

  committed
  a2924b0

• Need these fields in the debug output

  muffato

  committed
  b2f22a2

• Like in the genome note pipeline, use the work directory instead of /tmp

  Show description for cfac7c8
  muffato

  committed
  cfac7c8

• Updated the settings for SORMADUP

  muffato

  committed
  9da9f3f

• Updated the BWA_MEM memory requirement

  muffato

  committed
  5e15589

• Increased the number of retries

  muffato

  committed
  bdbcbb4

• The default memory settings work just fine and make things easier to manipulate

  muffato

  committed
  3905219

• Usage is very close to the trend line. Smaller bins work fine

  muffato

  committed
  c15d923

• quay.io/ is now the default

  muffato

  committed
  4094724

Commits on Dec 8, 2023

• Added optimised settings for crumble

  muffato

  committed
  3ef0477

• Need to fake REF_PATH to force crumble to use the Fasta file defined in the UR field of the @sq headers

  muffato

  committed
  f0c1ba4

Commits on Dec 9, 2023

• There is a difference for ONT, which I assume would be there for CLR too

  muffato

  committed
  79bcbbc