Merge pull request #70 from Workflomics/add-test-scripts

Add test scripts

.github/workflows/test.yml

@@ -0,0 +1,28 @@
+name: Run test script
+
+on: 
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  run-test:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.x'
+
+      - name: Install cwltool
+        run: pip install cwltool
+
+      - name: Run tests
+        run: ./test.sh

README.md

@@ -3,12 +3,18 @@
 
 This repository contains the domain and tool descriptions needed to generate and execute workflows in the [Workflomics](https://github.com/Workflomics/workflomics-frontend) interface.
 
-To add new domains or tools to the `Workflomics` environment, CWL files and other files needed to run these tools should be added to this repository. See the [domain annotation guide](https://workflomics.readthedocs.io/en/domain-creation/developer-guide/domain-development.html) for more information.
+To add new domains or tools to the `Workflomics` environment, CWL files and other files needed to run these tools should be added to this repository. For a detailed description of the steps that this requires, have a look at the [Workflomics documentation](https://workflomics.readthedocs.io/en/latest/index.html), under 'Domain Expert Guide'
 
 The repository is organized in the following way:
 
-- `domains/`: Contains the domain descriptions. The descriptions are used by the Workflomics environment to generate workflows. Each domain comprises a set of tools (e.g., described in `tools.json` file) and a configuraion file (e.g., `config.json`) that specifies the domain-specific parameters. See the [documentation of the APE engine](https://ape-framework.readthedocs.io/en/latest/docs/specifications/setup.html#configuration-file) to learn more about the configuration file.
-- `cwl-tools/`: Contains the CWL CommandLineTool descriptions of the tools used in the workflows (similar to the [bio-cwl-tools](https://github.com/common-workflow-library/bio-cwl-tools) repo). The CWL files are used by the Workflomics environment to execute each step of the workflow. Within the Workflomics ecosystem these workflows are executed using the [Workflomics Benchmarker](https://github.com/Workflomics/workflomics-benchmarker) which utilizes [cwltool](https://github.com/common-workflow-language/cwltool).
+- `domains/`: Contains the domain descriptions. The descriptions are used by the Workflomics environment to generate workflows. Each domain comprises a set of tools (e.g., described in `tools.json` file) and a configuraion file (e.g., `config.json`) that specifies the domain-specific parameters. See the [domain annotation guide](https://workflomics.readthedocs.io/en/latest/domain-expert-guide/domain-development.html) to learn more about these files.
+
+- `cwl-tools/`: Contains the CWL CommandLineTool descriptions of the tools used in the workflows (similar to the [bio-cwl-tools](https://github.com/common-workflow-library/bio-cwl-tools) repo). The CWL files are used by the Workflomics environment to execute each step of the workflow. Within the Workflomics ecosystem these workflows are executed using the [Workflomics Benchmarker](https://github.com/Workflomics/workflomics-benchmarker) which utilizes [cwltool](https://github.com/common-workflow-language/cwltool). For more information about adding new tools, see the [adding tools section](https://workflomics.readthedocs.io/en/latest/domain-expert-guide/adding-tools.html) of the documentation.
+
 - `examples/`: Contains example workflows that can be executed using the [Workflomics Benchmarker](https://github.com/Workflomics/workflomics-benchmarker). The workflows were generated by the Workflomics platform are written in the [Common Workflow Language (CWL)](https://www.commonwl.org/).
 
 When using Workflomics web interface, workflows are referencing this repository directly. They are downloaded during the workflow execution, so you don't need to clone this repository for normal usage in the Workflomics environment.
+
+## Testing
+
+To test the CWL annotations, run `test_cwl_annotations.cwl` in the repository root. This script runs the test scripts in the 'test' directory of each tool, testing whether the CWL annotations pass as stand-alone workflow steps. This requires `cwltool` and `docker` to be installed.

cwl-tools/Sage-proteomics/test/debug-in-docker.sh

@@ -3,7 +3,7 @@
 docker run -it --rm \
   --entrypoint /bin/bash \
   --mount=type=bind,source=/Users/peter/repos/bakeoff/containers/cwl-tools/Sage/test/data,target=/data/ \
-  sage:latest 
+  ghcr.io/lazear/sage:v0.14.7
 
 # sage -o /data/output -f /data/small.fasta /data/config.json /data/small.mzML
 

test_cwl_annotations.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+
+# This script runs the test scripts in the 'test' directory of each tool,
+# testing whether the CWL annotations pass as stand-alone workflow steps.
+# This requires `cwltool` and `docker` to be installed.
+
+# Set the base directory
+base_dir="cwl-tools"
+
+# Create arrays and counters for passed, failed tests, and missing scripts
+declare -a failed_tests
+passed_tests_count=0
+failed_tests_count=0
+missing_script_count=0
+
+# Iterate over all directories (tool-names) in the base directory
+for tool_dir in "$base_dir"/*/; do
+    # Define the path to the 'test' directory for the current tool
+    script_dir="${tool_dir}test"
+
+    # Check if the 'run-cwl.sh' script exists
+    if [ -f "$script_dir/run-cwl.sh" ]; then
+        echo "Testing $script_dir..."
+
+        (cd "$script_dir" && bash "./run-cwl.sh")
+        if [ $? -eq 0 ]; then
+            ((passed_tests_count++))  # Increment passed tests counter
+        else
+            failed_tests+=("$script_dir/run-cwl.sh")
+            ((failed_tests_count++))  # Increment failed tests counter
+        fi
+
+    else
+        echo "❗ Script not found in directory: $script_dir"
+        ((missing_script_count++))  # Increment the missing script counter
+    fi
+done
+
+# Print missing test script count
+if [ $missing_script_count -gt 0 ]; then
+    echo "❗ $missing_script_count tools did not have a test script"
+fi
+
+# Print a summary of test results
+if [ $failed_tests_count -eq 0 ]; then
+    echo "✅ $passed_tests_count tests passed successfully"
+    exit 0
+else
+    echo "The following tests failed:"
+    for test in "${failed_tests[@]}"; do
+        echo "🚨 $test"
+    done
+    exit 1
+fi