RNA Seq Analysis Pipeline for F. Nadeu

This is a pipeline written in Python and bash, and with Snakemake as a workflow manager, that will output kallisto files (abundances of transcripts) from RNA-Seq data. These samples can be in .fastq or compressed format.

The samples can be placed in any directory, but the path must be specified in the config/config.yaml file.

Table of Contents

1. Installation
2. Linux
3. Windows
4. macOS
5. Snakemake Environment
6. Clone the repository
7. Snakemake Usage
8. Run the pipeline in a HPC
9. Configuration of the pipeline
10. Cluster Configuration

Installation

Linux

Linux Installation

Install Miniconda 3

Open a Linux shell, then run these three commands to quickly and quietly download the latest 64-bit Linux miniconda 3 installer, rename it to a shorter file name, silently install, and then delete the installer.

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh

After installing, initialize your newly-installed Miniconda. The following commands initialize for bash and zsh shells:

~/miniconda3/bin/conda init bash

~/miniconda3/bin/conda init zsh

You should see (base) in the command line prompt. This tells you that you’re in your base conda environment. To learn more about conda environments, see Environments.

Check for a good installation with:

conda --version
# conda 24.X.X

conda list
# outputs a list of packages installed in the current environment (base)

---

Windows

Windows Installation

Since Windows does not have access to the majority of packages we need in the pipeline, we need to install Linux on Windows, also known as WSL. In the Windows Power Shell:

wsl --install
# This command will install the Ubuntu distribution of Linux.

If you run into an issue during the installation process, please check the installation section of the troubleshooting guide.

Set up your Linux user info

Once you have installed WSL, you will need to create a user account and password for your newly installed Linux distribution. See the Best practices for setting up a WSL development environment guide to learn more.

Install Miniconda 3

Once you have a working shell in your WSL, run these three commands to quickly and quietly download the latest 64-bit Linux miniconda 3 installer, rename it to a shorter file name, silently install, and then delete the installer.

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh

After installing, initialize your newly-installed Miniconda. The following commands initialize for bash and zsh shells:

~/miniconda3/bin/conda init bash

~/miniconda3/bin/conda init zsh

You should see (base) in the command line prompt. This tells you that you’re in your base conda environment. To learn more about conda environments, see Environments.

Check for a good installation with:

conda --version
# conda 24.X.X

conda list
# outputs a list of packages installed in the current environment (base)

---

macOS

macOS Installation

These four commands download the latest M1 version of the MacOS installer, rename it to a shorter file name, silently install, and then delete the installer:

mkdir -p ~/miniconda3
curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh -o ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh

After installing, initialize your newly-installed Miniconda. The following commands initialize for bash and zsh shells:

~/miniconda3/bin/conda init bash

~/miniconda3/bin/conda init zsh

You should see (base) in the command line prompt. This tells you that you’re in your base conda environment. To learn more about conda environments, see Environments.

Check for a good installation with:

conda --version
# conda 24.X.X

conda list
# outputs a list of packages installed in the current environment (base)

---

Snakemake Environment

Now, with miniconda installed in our machine, we can create a new environment with snakemake installed:

conda create -c conda-forge -c bioconda -n snakemake snakemake

In case you are downloading from the Clinic network, you will have trouble with the SLL certificate. To solve any problems, do the following:

1. You can usually get a copy by clicking on the padlock icon in your browser when visiting any https site, then click around to view certificate, and download in PEM format.

2. Then we will point conda to it in our system.

conda config --set ssl_verify <pathToYourFile>.pem

Once installed, we must activate and move into the snakemake environment with:

conda activate snakemake
snakemake --version
# 8.16.X

If at any time we want to exit the environment, we can withconda deactivate, and to get back in with conda activate snakemake. To see the packages we have currently installed in the environment, we can withconda list.

Clone the repository

1. Above the list of files, click Code.

2. Copy the URL for the repository. To clone the repository using HTTPS, under "HTTPS", copy the link provided.
3. Open a Terminal.
4. Change the current working directory to the location where you want the cloned directory. For example, cd rna_seq_fnadeu. Make sure that the directory exists before you move into it.
5. Type git clone [email protected]:lymphIDIBAPS/rnaseq_fnadeu.git.
6. Press Enter to create your local clone.

git clone [email protected]:lymphIDIBAPS/rnaseq_fnadeu.git
> Cloning into `rna_seq_fnadeu`...
> remote: Counting objects: 10, done.
> remote: Compressing objects: 100% (8/8), done.
> remove: Total 10 (delta 1), reused 10 (delta 1)
> Unpacking objects: 100% (10/10), done.

Snakemake Usage

When we have the cloned repository, we can proced and add our sample data to the FASTQ directory. This is not mandatory, as in config/config.yaml file we can edit and set any path to our sample data.

In the same file we can edit the number of threads our computer has, so it will run adapted to the current resources we have available.

The rulegraph for our pipeline at date 25/10 is the following:

# For a test run of the pipeline
snakemake --use-conda -np

# For a real run of the pipeline
snakemake --use-conda

Run the pipeline in a HPC

If we have many samples and our computer does not have enough computational power, we can run the pipeline in a cluster. This pipeline has been prepared to run in the StarLife cluster, in the BSC.

1. Make a new directory named /slgpfs/ in your computer and mount it to the same directory in StarLife:

mkdir /home/user/slgpfs
sshfs -o allow_other [email protected]:/slgpfs/ /home/user/slgpfs/

This will allow you to see and work on the custer from your computer system directly.

2. On your computer, navigate to the directory: /home/user/slgpfs/projects/group_folder

3. Download and extract the following file to the directory, in which we have a full conda environment ready to run snakemake: Snakemake Conda Environment

4. Clone this repository in the directory, following the steps from Clone the repository

5. Now, connect to the cluster:

ssh [email protected] # or
ssh [email protected]

6. In the cluster, navigate to the cloned repository: /slgpfs/projects/group_folder/rna_seq_fnadeu

7. Now, activate the snakemake_bsc environment:

source ../snakemake_bsc/bin/activate

In your terminal, you should now see something like: (snakemake_bsc) your_username@sllogin1

8. At this point, in your local machine, you can move your samples to the directory /slgpfs/projects/group_folder/rna_seq_fnadeu/FASTQ.

9. Now, you can run the pipeline from the cluster with the command:

# For a test run of the pipeline
snakemake --profile config/slurm/ --use-envmodules -np

# For a real run of the pipeline
snakemake --profile config/slurm/ --use-envmodules

This command above will run the pipeline with the pipeline configuration from the file located in /slgpfs/projects/group_folder/rna_seq_fnadeu/config/config.yaml. Be sure to check and modify the configuration file to alter the pipeline with your desired options.

The cluster configuration file is located in /home/oscar/rnaseq/config/slurm/config.yaml. Below you have all the options available to customize your cluster run.

Configuration of the pipeline

General Configuration

1. Pep File: path to a .yaml file, containing the PEP file info and additional metadata about our project.

2. Working directory: the directory where our analysis will be run.

3. Path to data and pipelines folder: where our data and other resources are located in the cluster.

4. Perform T-trimming: cut 1 base from the start of the read in trimmomatic. Yes or no, with default = no.

5. Adapters: adapters to be removed by trimmomatic, you can choose between illumina or bioskryb; default = illumina

6. Analysis name: which name you want your analysis to have

7. Remove fastqs: remove intermediate fastqs after QC; yes or no, default = no

8. Remove bams: remove bam files after QC; yes or no, default = no

9. rRNA database for sortmerna: which of the rRNA databases do you wish to use; fast or sensitive or default, default = default

10. Index file for kallisto: do ypu want to include only cDNA, ncDNA or both in the index; cDNA or ncRNA or both, default = cDNA

11. runQC: should BAM files be created and QC metrics done; yes or no, default = yes

12. Number of cpus per job: for some rules, specify the amount of cpus to use; default = 20

13. Transcription strand for rules kallisto and collectRNASeqMetrics; first, second, unstranded, default = first

Cluster Configuration

Remember to check the files in /config/slurm/config.yaml for the cluster configuration. Review all the items and in case something is not clear you can check in this website what each term means in the configuration.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Author

Developed by @obaeza16, based on a pipeline written by Ferran Nadeu.

Mantained by Lymphoid neoplasms program, IDIBAPS for Ferran Nadeu.

License

MIT