Skip to content

Latest commit

 

History

History
82 lines (60 loc) · 1.65 KB

README.md

File metadata and controls

82 lines (60 loc) · 1.65 KB
Raw

Cromwell Output Transfer (Cromsfer)

Transfers output files from Cromwell/WDL workflows to a designated S3 locations with a human-friendly directory structure. It supports:

  • FastQC (FASTQ QC)
  • Sharp (Hashtag and CellPlex)
  • Sharp (CITE-seq)
  • Sharp (ASAP-seq)
  • Velopipe (RNA Velocity)
  • Cell Ranger V(D)J
  • Cell Ranger GEX
  • Cell Ranger ATAC
  • Cell Ranger ARC
  • Cell Ranger CellPlex
  • Space Ranger
  • Transgenes for Cell Ranger
  • ArchR Stand Alone (aka. ArchRSA)
  • mkref (Generating genome index for STAR aligner)

Prerequisites

  • AWS CLI
  • gsutil

Development Environment

Install Cromsfer

conda create -n cromsfer python=3.8 pip
conda activate cromsfer
git clone https://github.com/hisplan/cromsfer.git
pip install -e .[dev]

Run Redis

docker run --rm -d -p 6379:6379 redis:5.0.6

Run the Poller Service

cromsfer.poller picks up the workflows that have been completed, but not yet transferred.

cromsfer.poller --config config.aws-local-redis.yaml

Run the Transfer Service

cromsfer.transfer  --config config.aws-local-redis.yaml

Utilities

$ export JOB_MANAGER_USERNAME=johnDoe
$ export JOB_MANAGER_PWD=xyz123abc

Getting the metadata for a given workflow:

$ cd utils
$ ./get-metadata.sh -w 4bb895a2-dc44-4d6d-94ca-1294452e1bf8

Resetting the transfer status (i.e. "-")

$ cd utils
$ ./transfer-reset.sh -w 4bb895a2-dc44-4d6d-94ca-1294452e1bf8

Deployment

  1. Make sure you increment the version number (src/cromsfer/version.py).
  2. Push all the changes into the GitHub repository.
  3. Create a release tag in the GitHub repository.
  4. Build a docker image.