Set up the directory structure:
project_dir="/data/BIDS-HPC/private/projects/dmi2"
working_dir="/home/weismanal/notebook/2020-06-10/dmi"
mkdir "$project_dir" "$working_dir"
cd "$working_dir"
git clone [email protected]:andrew-weisman/target_classification.git "$project_dir/checkout"
mkdir "$project_dir/data"
Note: The effort using the data directly from the TARGET data website (as opposed to the GDC Data Portal) is in the target_data_website
branch of this repository.
Download the manifest for all the gene expression quantification files in the TARGET program (click on the blue "Manifest" button):
Place the downloaded manifest file as $project_dir/checkout/manifests/gdc_manifest.2020-06-10-all_gene_expression_files_in_target.txt
.
In addition, click on the blue "Add All Files to Cart" button, go to the cart (top right of page), click on the two blue buttons "Sample Sheet" and "Metadata", and save the resulting two files to $project_dir/data
. The two files will be named, e.g., gdc_sample_sheet.2020-07-02.tsv
and metadata.cart.2020-07-02.json
.
Note that these 5,149 files correspond to 1,192 cases (people [for sure that's what it means]).
Download the expression files from the manifest on Helix:
module load gdc-client
mkdir "$project_dir/data/all_gene_expression_files_in_target"
cd !!:1
gdc-client download -m "$project_dir/checkout/manifests/gdc_manifest.2020-06-10-all_gene_expression_files_in_target.txt"
Extract the resulting compressed files and link to them from a single folder $project_dir/data/all_gene_expression_files_in_target/links
:
mkdir links
cd !!:1
for file in $(find ../ -iname "*.gz"); do gunzip "$file"; done
for file in $(find ../ -type f | grep -v "/logs/\|/annotations.txt"); do ln -s $file; done
ln -s "$project_dir/checkout/manifests/gdc_manifest.2020-06-10-all_gene_expression_files_in_target.txt" MANIFEST.txt
Note that
for file in $(ls | grep -v MANIFEST.txt); do echo $file | awk -v FS="." '{print $1}'; done | sort -u | wc -l
shows that, ostensibly, there are 2,481 unique expression files (independent of normalization). This is just based on the filenames, and is not actually correct.
Start an interactive allocation, using, e.g.,
sinteractive --mem=40g # --mem=20g may be fine
Go through the Python Jupyter notebook /data/BIDS-HPC/private/projects/dmi2/checkout/main.ipynb
. Use the conda
environment /data/BIDS-HPC/public/software/conda/envs/r_env
. (Note this environment contains pandas
version 1.1.0
, whereas Biowulf's default python
module has pandas
version 0.24.2
, which is insufficient.) See here for more notes on the environment.