README.qmd

---
title: Viash project template
format: gfm
engine: knitr
toc: false
---

<!-- README.md is generated from README.qmd using quarto render. Please edit that file -->

This repository is a template for setting up a new Viash project, and is part of the [Quickstart](https://viash.io/quickstart) tutorial to learn how to get started with this repository.

## What is Viash?

**Viash** is your go-to script wrapper for building data pipelines from modular software components. All you need is your trusty script and a metadata file to embark on this journey.

Check out some of Viash's key features:

- Code in your [favorite scripting language](/guide/component/create-component.html). Mix and match scripting between multiple components to suit your needs. Viash supports a wide range of languages, including Bash, Python, R, Scala, JS, and C#.

- A **custom Docker container** is auto-generated based on the dependencies you've outlined in your metadata, meaning you don't need to be a Docker expert.

- Viash also generates a **Nextflow module** from your script, so no need to be a Nextflow guru either.

- Effortlessly combine Nextflow modules to design and run scalable, reproducible data pipelines.

- Test every component on your local workstation using the convenient built-in development kit.

```{mermaid}
graph LR
  subgraph component [Viash component]
    subgraph script [Script]
      rlang[R script]
      python[Python script]
      bash[Bash script]
      scriptetc[...]
    end

    config[Viash config]
  end

  viash_build[Viash build]

  docker_image[Docker image]
  executable[Executable]
  nextflow[Nextflow workflow]

  
  component --- viash_build --> executable & docker_image & nextflow
  docker_image -.-> executable & nextflow


  nextflow --dependency--> nextflow

  subgraph compute [Compute environment]
    direction LR
    local[Local execution]
    awsbatch[AWS Batch]
    googlebatch[Google Cloud Batch]
    hpc[HPC]
    infraetc[...]
  end

  nextflow --> compute
```

## Requirements

This guide assumes you've already installed [Viash](https://viash.io/installation), [Docker](https://docs.docker.com/engine/install). and [Nextflow](https://www.nextflow.io/index.html#GetStarted).


## Structure of this template project

To get up and running fast, we provide a [template project](https://github.com/viash-io/viash_project_template) for you to use. It contains four components from the same package as well, which are combined into a Nextflow pipeline as follows:

```{mermaid}
graph TD
   input1(file1.tsv) --> B1[/remove_comments/] --> C1[/take_column/] --> Y
   input2(file2.tsv)--> B2[/remove_comments/] --> C2[/take_column/] --> Y
   Y[combine] --> D[/combine_columns/]
   D --> output(output.tsv)
```

This pipeline takes one or more TSV files as input and stores its output in an output folder.

## Example usage

To run the pipeline, first create example input files.

Contents of `resources_test/file1.tsv`:

```{bash echo=FALSE}
cat resources_test/file1.tsv
```

Contents of `resources_test/file2.tsv`:

```{bash echo=FALSE}
cat resources_test/file2.tsv
```

Finally, we also need to create a `params.yaml` file to specify the input files for the pipeline:

Contents of `resources_test/params.yaml`:

```{bash echo=FALSE}
cat resources_test/params.yaml
```

Now run the pipeline:

```bash
nextflow run viash-io/viash_project_template \
  -main-script target/nextflow/template/workflow/main.nf \
  -r build/main \
  -latest \
  -profile docker \
  -params-file resources_test/params.yaml \
  --publish_dir output
```

<details>
<summary>Output</summary>

```{bash echo=FALSE}
# avoid ansi log for proper static output using -ansi-log false
nextflow run viash-io/viash_project_template \
  -main-script target/nextflow/template/workflow/main.nf \
  -r build/main \
  -latest \
  -profile docker \
  --publish_dir output \
  -params-file resources_test/params.yaml \
  -ansi-log false
```

</details>

If you have a [Seqera Cloud](https://cloud.seqera.io) compute environment already set up, you can also launch the workflow there:

```bash
cat > params.yaml <<EOF
param_list:
  - id: file1
    input: s3://my-bucket/file1.tsv
  - id: file2
    input: s3://my-bucket/file2.tsv
publish_dir: s3://my-bucket/output
EOF

tw launch viash-io/viash_project_template \
  --main-script target/nextflow/template/workflow/main.nf \
  --revision build/main \
  --pull-latest \
  --workspace 123456789 \
  --compute-env ABCDEFGHIJKLMNOP \
  --params-file params.yaml
```

## Extending this template

This template is a great starting point for building your own Viash project. Here's how you can extend it.


### Step 1: Get the template

First create a new repository by clicking the "Use this template" button. If you can't see the "Use this template" button, log into GitHub first.

Next, clone the repository using the following command.

```bash
git clone https://github.com/youruser/my_first_pipeline.git && cd my_first_pipeline
```

Your new repository should contain the following files:

```bash
tree my_first_pipeline
```

<details>
<summary>Output</summary>

```{bash echo=FALSE}
tree .
```

</details>

### Step 2: Build the Viash components

With Viash you can turn the components in `src/` into Dockerized Nextflow modules by running:

```bash
viash ns build --setup cachedbuild --parallel
```

<details>
<summary>Output</summary>

```{bash echo=FALSE}
viash ns build --setup cachedbuild --parallel
```

</details>

This command not only transforms the Viash components in `src/` to Nextflow modules but it also builds the containers when appropriate (starting from the Docker cache when available using the `cachedbuild` argument). Once everything is built, a new **target** directory has been created containing the executables and modules grouped per platform:

```bash
tree target
```

<details>
<summary>Output</summary>
```{bash echo=FALSE}
tree target
```
</details>

### Step 3: Run the pipeline

You can now run the locally built pipeline using the following command:

```bash
nextflow run . \
  -main-script target/nextflow/template/workflow/main.nf \
  -profile docker \
  -params-file resources_test/params.yaml \
  --publish_dir output
```

<details>
<summary>Output</summary>
```{bash echo=FALSE}
# avoid ansi log for proper static output using -ansi-log false
nextflow run . \
  -main-script target/nextflow/template/workflow/main.nf \
  -profile docker \
  -params-file resources_test/params.yaml \
  --publish_dir output \
  -ansi-log false
```
</details>

This will run the different stages of the workflow , with the final result result being stored in a file named **run.combine_columns.output** in the output directory `output`:

```bash
cat output/combined.workflow.output.tsv
```

<details>
<summary>Output</summary>
```{bash echo=FALSE}
cat output/combined.workflow.output.tsv
```
</details>


## What's next?

Congratulations, you've reached the end of this quickstart tutorial, and we're excited for you to delve deeper into the world of Viash!
Our comprehensive [guide](https://viash.io/guide) and [reference documentation](https://viash.io/reference) is here to help you explore various topics, such as:

* [Creating a Viash component and converting it into a standalone executable](https://viash.io/guide/component/create-component)
* [Ensuring reproducibility and designing customised Docker images](https://viash.io/guide/component/add-dependencies)
* [Ensuring code reliability with unit testing for Viash](https://viash.io/guide/component/unit-testing)
* [Streamlining your workflow by performing batch operations on Viash projects](https://viash.io/guide/project/batch-processing)
* [Building Nextflow pipelines using Viash components](https://viash.io/guide/nextflow_vdsl3)

So, get ready to enhance your skills and create outstanding solutions with Viash!