diff --git a/README.md b/README.md index 23934cb..8686054 100644 --- a/README.md +++ b/README.md @@ -1,147 +1,9 @@ # nft-utils -nf-test utility functions - This repository contains utility functions for nf-test. - These functions are used to help capture level tests using nf-test. -## `removeNextflowVersion()` - -nf-core pipelines create a yml file listing all the versions of the software used in the pipeline. - -Here is an example of this file coming from the rnaseq pipeline. - -```yaml -UNTAR: - untar: 1.34 -Workflow: - nf-core/rnaseq: v3.16.0dev - Nextflow: 24.04.4 -``` - -This function remove the Nextflow version from this yml file, as it is not relevant for the snapshot. Therefore for the purpose of the snapshot, it would consider this to be the contents of the YAML file: - -```yaml -UNTAR: - untar: 1.34 -Workflow: - nf-core/rnaseq: v3.16.0dev -``` - -Usage: - -```groovy -assert snapshot(removeNextflowVersion("$outputDir/pipeline_info/nf_core_rnaseq_software_mqc_versions.yml")).match() -``` - -The only argument is path to the file which must be a versions file in YAML format as per the nf-core standard. - -## `getAllFilesFromDir()` - -Files produced by a pipeline can be compared to a snapshot. -This function produces a list of all the content of a directory, and can exclude some files based on a glob pattern. -From there one can get just filenames and snapshot only the names when content is not stable. -Or snapshot the whole list of files that have stable content. - -In this example, these are the files produced by a pipeline: - -```bash -results/ -├── pipeline_info -│   └── execution_trace_2024-09-30_13-10-16.txt -└── stable - ├── stable_content.txt - └── stable_name.txt - -2 directories, 3 files -``` - -In this example, 1 file is stable with stable content (`stable_content.txt`), and 1 file is stable with a stable name (`stable_name.txt`). -The last file has no stable content (`execution_trace_2024-09-30_13-10-16.txt`) as its name is based on the date and time of the pipeline execution. - -For this example, we want to snapshot the files that have stable content, and the names of files and folders that are stable. -`stable_name` is a list of every file and folder except those matching the glob `pipeline_info/execution_*.{html,txt}`. -`stable_content` is a list of every file except those matching the two globs `pipeline_info/execution_*.{html,txt}` and `**/stable_name.txt` which is contained in the `tests/getAllFilesFromDir/.nftignore` file. -By using `stable_name*.name`, we extract the name of every file in `stable_name` and add them to the snapshot. -`stable_content` can be used in the snapshot directly to include the hash of the file contents. - -```groovy -def stable_name = getAllFilesFromDir(params.outdir, true, ['pipeline_info/execution_*.{html,txt}'], null ) -def stable_content = getAllFilesFromDir(params.outdir, false, ['pipeline_info/execution_*.{html,txt}'], 'tests/getAllFilesFromDir/.nftignore' ) -assert snapshot( - stable_name*.name, - stable_content -).match() -``` - -First argument is the pipeline `outdir` directory path, second is a boolean to include folders, and the third is a list of glob patterns to ignore, and the fourth is a file containing a list of glob patterns to ignore. - -## `getRelativePath()` - -This function is used to get the relative path from a list of files compared to a given directory. - -```bash -results/ -├── pipeline_info -│   └── execution_trace_2024-09-30_13-10-16.txt -└── stable - ├── stable_content.txt - └── stable_name.txt - -2 directories, 3 files -``` - -Following the previous example, we want to get the relative path of the stable paths in the `results` directory. - -```groovy -def stable_name = getAllFilesFromDir(params.outdir, true, ['pipeline_info/execution_*.{html,txt}'], null ) -``` - -The `stable_name` variable contains the list of stable files and folders in the `results` directory. - -```groovy -assert snapshot( - getRelativePath(stable_name, outputDir) -).match() -``` - -By using `getRelativePath()` we generate in the snapshot: - -```text -"content": [ - [ - "pipeline_info", - "stable", - "stable/stable_content.txt", - "stable/stable_name.txt" - ] -] -``` - -A reduced list can be generated by using the `getAllFilesFromDir()` not to output the folders. - -```text -"content": [ - [ - "stable/stable_content.txt", - "stable/stable_name.txt" - ] -] -``` - -While only a flat structure would be generated without using `getRelativePath()` and using `*.name` to capture the file names. - -```text -"content": [ - [ - "pipeline_info", - "stable", - "stable_content.txt", - "stable_name.txt" - ] -] -``` +Please read the [documentation](./docs) for more information. ## Credits @@ -152,6 +14,7 @@ We'd like to thank the following people: - [Adam Talbot](https://github.com/adamrtalbot) - [Edmund Miller](https://github.com/edmundmiller) - [Jonathan Manning](https://github.com/pinin4fjords) +- [Lukas Forer](https://github.com/lukfor) - [Matthias Zepper](https://github.com/MatthiasZepper) - [Nicolas Vannieuwkerke](https://github.com/nvnieuwk) - [Sateesh Peri](https://github.com/sateeshperi) diff --git a/docs/index.md b/docs/index.md index 38065b1..82fcd87 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,21 +1,12 @@ # nft-utils -nft-utils is an nf-test plugin to provide additional functions and assertions that fall outside of the typical nf-test features. They were primarily developed by the nf-core community but should be applicable to any nf-tests. +nft-utils is an nf-test plugin to provide additional functions and assertions that fall outside of the typical nf-test features. +They were primarily developed by the nf-core community but should be applicable to any nf-tests. ## Start using the plugin To start using the plugin please add it to your `nf-test.config` file: -```groovy title="nf-test.config" -config { - plugins { - load "nft-utils" - } -} -``` - -It's a good idea to specify a version which you can do using the `@` symbol, however currently there are no releases: - ```groovy title="nf-test.config" config { plugins { diff --git a/docs/usage.md b/docs/usage.md index 54bb8c2..cd2d50e 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -1,3 +1,155 @@ -# Usage +# Functions usage -TBC! +The plugin adds the following functions to assist with managing pipeline-level nf-test snapshots: + +## `removeNextflowVersion()` + +nf-core pipelines create a yml file listing all the versions of the software used in the pipeline. + +Here is an example of this file coming from the rnaseq pipeline. + +```yaml +UNTAR: + untar: 1.34 +Workflow: + nf-core/rnaseq: v3.16.0dev + Nextflow: 24.04.4 +``` + +This function remove the Nextflow version from this yml file, as it is not relevant for the snapshot. Therefore for the purpose of the snapshot, it would consider this to be the contents of the YAML file: + +```yaml +UNTAR: + untar: 1.34 +Workflow: + nf-core/rnaseq: v3.16.0dev +``` + +Usage: + +```groovy +assert snapshot(removeNextflowVersion("$outputDir/pipeline_info/nf_core_rnaseq_software_mqc_versions.yml")).match() +``` + +The only argument is path to the file which must be a versions file in YAML format as per the nf-core standard. + +## `getAllFilesFromDir()` + +This function generates a list of all the contents within a directory, allowing for the exclusion of specific files using a glob pattern. It can be used to obtain filenames alone, enabling snapshotting of just the names when the content is not stable. Alternatively, it can snapshot the entire list of files with stable content. + +In this example, below are the files produced by a pipeline: + +```bash +results/ +├── pipeline_info +│ └── execution_trace_2024-09-30_13-10-16.txt +└── stable + ├── stable_content.txt + └── stable_name.txt + +2 directories, 3 files +``` + +In this example, 1 file is stable with stable content (`stable_content.txt`), and 1 file is stable with a stable name (`stable_name.txt`). +The last file has no stable content (`execution_trace_2024-09-30_13-10-16.txt`) as its name is based on the date and time of the pipeline execution. + +In this example, we aim to snapshot files with stable content, along with the names of files and folders that are stable. + +•`stable_name` is a list of all files and folders, excluding those matching the glob pattern `pipeline_info/execution_*.{html,txt}`. +•`stable_content` is a list of all files, excluding those that match the two glob patterns: `pipeline_info/execution_*.{html,txt}` and `**/stable_name.txt`. The latter is specified in the `tests/getAllFilesFromDir/.nftignore` file. +By using `stable_name*.name`, we extract the name of every file in `stable_name` and add them to the snapshot. +`stable_content` can be used in the snapshot directly to include the hash of the file contents. + +```groovy +def stable_name = getAllFilesFromDir(params.outdir, true, ['pipeline_info/execution_*.{html,txt}'], null ) +def stable_content = getAllFilesFromDir(params.outdir, false, ['pipeline_info/execution_*.{html,txt}'], 'tests/getAllFilesFromDir/.nftignore' ) +assert snapshot( + stable_name*.name, + stable_content +).match() +``` + +• The first argument is the pipeline’s `outdir` directory path. +• The second argument is a boolean indicating whether to include folders. +• The third argument is a list of glob patterns to ignore. +• The fourth argument is a file containing additional glob patterns to ignore. + +`getAllFilesFromDir()` also supports named parameters: + +```groovy +def stable_name = getAllFilesFromDir(params.outdir, ignore: ['pipeline_info/execution_*.{html,txt}']) +def stable_content = getAllFilesFromDir(params.outdir, includeDir: false, ignore: ['pipeline_info/execution_*.{html,txt}'], ignoreFile: 'tests/getAllFilesFromDir/.nftignore') +``` + +## `getRelativePath()` + +This function is used to get the relative path from a list of files compared to a given directory. + +```bash +results/ +├── pipeline_info +│ └── execution_trace_2024-09-30_13-10-16.txt +└── stable + ├── stable_content.txt + └── stable_name.txt + +2 directories, 3 files +``` + +Following the previous example, we want to get the relative path of the stable paths in the `results` directory. + +```groovy +def stable_name = getAllFilesFromDir(params.outdir, true, ['pipeline_info/execution_*.{html,txt}'], null ) +``` + +The `stable_name` variable contains the list of stable files and folders in the `results` directory. + +```groovy +assert snapshot( + getRelativePath(stable_name, outputDir) +).match() +``` + +By using `getRelativePath()` we generate in the snapshot: + +```text +"content": [ + [ + "pipeline_info", + "stable", + "stable/stable_content.txt", + "stable/stable_name.txt" + ] +] +``` + +A reduced list can be generated by using `getAllFilesFromDir()` without including the folders in the output. + +```text +"content": [ + [ + "stable/stable_content.txt", + "stable/stable_name.txt" + ] +] +``` + +Without using `getRelativePath()` and by using `*.name` to capture the file names, only a flat structure would be generated, as shown below: + +```text +"content": [ + [ + "pipeline_info", + "stable", + "stable_content.txt", + "stable_name.txt" + ] +] +``` + +`getAllFilesFromDir()` named parameters `relative` can also be used to combine the two functions: + +```groovy +def stable_name = getAllFilesFromDir(params.outdir, relative: true, ignore: ['pipeline_info/execution_*.{html,txt}'] ) +def stable_file = getAllFilesFromDir(params.outdir, relative: true, includeDir: false, ignore: ['pipeline_info/execution_*.{html,txt}'] ) +``` diff --git a/tests/getRelativePath/main.nf.test b/tests/getRelativePath/main.nf.test index 9ee7771..19da6dc 100644 --- a/tests/getRelativePath/main.nf.test +++ b/tests/getRelativePath/main.nf.test @@ -22,4 +22,22 @@ nextflow_pipeline { ).match() } } + + test("getAllFilesFromDir & getRelativePath combined") { + when { + params { + outdir = "$outputDir" + } + } + + then { + // Use getAllFilesFromDir() with named params to get a list of all files and folders from the output directory, minus the timestamped files + def stable_name = getAllFilesFromDir(params.outdir, relative: true, ignore: ['pipeline_info/execution_*.{html,txt}'] ) + def stable_file = getAllFilesFromDir(params.outdir, relative: true, includeDir: false, ignore: ['pipeline_info/execution_*.{html,txt}'] ) + assert snapshot( + stable_name, + stable_file + ).match() + } + } } diff --git a/tests/getRelativePath/main.nf.test.snap b/tests/getRelativePath/main.nf.test.snap index 60938ab..b5edf4b 100644 --- a/tests/getRelativePath/main.nf.test.snap +++ b/tests/getRelativePath/main.nf.test.snap @@ -1,4 +1,23 @@ { + "getAllFilesFromDir & getRelativePath combined": { + "content": [ + [ + "pipeline_info", + "stable", + "stable/stable_content.txt", + "stable/stable_name.txt" + ], + [ + "stable/stable_content.txt", + "stable/stable_name.txt" + ] + ], + "meta": { + "nf-test": "0.9.0", + "nextflow": "24.09.0" + }, + "timestamp": "2024-10-03T20:14:09.332332" + }, "getRelativePath": { "content": [ [