-
Notifications
You must be signed in to change notification settings - Fork 4
Scratch Space
Nextflow allows you select the location of any temporary files which are produced
during the course of workflow execution. These files are not automatically deleted
when the workflow completes, which can be helpful for debugging errors or 'resuming'
partially completed runs. However, over time this can needlessly clog up your file
system. By default, Nextflow just writes these files to a folder called work/ inside
the folder where the workflow was launched. However, a useful practice is to keep
all of these files in a common location across all workflow runs so that they can be
more easily cleaned up to save space. Don't worry, these files will never conflict
with each other (Nextflow keeps track of that for you). In the Nextflow configuration
(described below) there is an option to specify a workDir, which can be used to
specify an absolute filepath where all temporary files will be placed. If running
on a local system, this path must point to a local folder. If running on a cloud
computing service, make sure to use a storage location within your cloud computing
account. Finally, if running on a shared computing cluster, make sure to use the
'scratch' storage volume which has been made available to you.
When running a workflow with Singularity, each software image must be saved as a file
object. Since these files can be quite large, it is often best to store them in
a less costly location. To help support this need for shared computing clusters,
the scratch space used for the work/ files can also be used for a separate folder
named singularity/. This folder will be populated by Nextflow with the Singularity
image files needed for any particular workflow. It can be easily deleted and re-created
by Nextflow, and so it does not need to be saved in a backed-up location on the filesystem.