Merge pull request #298 from BU-ISCIII/develop

Release 1.0.0

.github/CONTRIBUTING.md

@@ -0,0 +1,72 @@
+# relecov-tools: Contributing Guidelines
+
+## Contribution workflow
+
+If you'd like to write or modify some code for relecov-tools, the standard workflow is as follows:
+
+1. Check that there isn't already an issue about your idea in the [relecov-tools issues](https://github.com/BU-ISCIII/relecov-tools/issues) to avoid duplicating work. **If there isn't one already, please create one so that others know you're working on this**.
+2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [relecov-tools repository](https://github.com/BU-ISCIII/relecov-tools/) to your GitHub account.
+3. Make the necessary changes / additions within your forked repository following the [code style guidelines](#code-style-guidelines).
+4. Modify the [`CHANGELOG`](../CHANGELOG.md) file according to your changes in the appropiate section ([X.X.Xdev]), you should register your changes regarding:
+   1. Added enhancements
+   2. New modules
+   3. Fixes
+   4. Removed stuff
+   5. Requirements added or version update
+5. Update any documentation as needed.
+6. [Submit a Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) against the `develop` branch and send the url to the #pipelines-dev channel in slack (if you are not in the slack channel just wait fot the PR to be reviewed and rebased).
+
+If you're not used to this workflow with git, you can start with:
+
+- Some [docs in the bu-isciii wiki](https://github.com/BU-ISCIII/BU-ISCIII/wiki/Github--gitflow).
+- [some slides](https://docs.google.com/presentation/d/1PruqGxPQVxtNcuEbOd86mylXorgYIU5a/edit?pli=1#slide=id.p1) (in spanish).
+- some github generic docs [docs from GitHub](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests).
+- even their [excellent `git` resources](https://try.github.io/).
+
+### relecov-tools repo branches
+
+relecov-tools repo works with a two branching scheme: `main` and `develop`.
+
+- `main`: stable code only for releases.
+- `develop`: new code development for the different modules.
+
+You need to submit your PR always against `develop`. Once approbed, this changes must be **`rebased`** so we do not create empty unwanted merges.
+
+## Tests
+
+When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests.
+Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.
+
+There are typically two types of tests that run:
+
+### Lint tests
+
+We use black and flake8 linting based on PEP8 guidelines for python coding. You can check more information [here](https://github.com/BU-ISCIII/BU-ISCIII/wiki/Python#linting).
+
+### Code tests
+
+Download, read-lab-metadata, map and validate modules are executed using a test dataset.
+
+Anyhow you should always submit locally tested code!!
+
+### New version bumping and release
+
+In order to create a new release you need to follow the next steps:
+
+1. Set the new version according to [semantic versioning](https://semver.org/), in our particular case, changes in the `hotfix` branch will change the PATCH version (third one), and changes in develop will typicaly change the MINOR version, unless the developing team decides otherwise.
+2. Create a PR bumping the new version against `develop`. For bumping a new version just change [this line](https://github.com/BU-ISCIII/relecov-tools/blob/09c00c1ddd11f7489de7757841aff506ef4b7e1d/setup.py#L5) with the new version.
+3. Once that PR is merged, create via web another PR against `main` (origin `develop`). This PR would need 2 approvals.
+4. [Create a new release](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository) copying the appropiate notes from the `CHANGELOG`.
+5. Once the release is approved and merged, you're all set!
+
+PRs from one branch to another, like in a release should be **`merged`** not rebased, so we avoid conflicts and the branch merge is correctly visualize in the commits history.
+
+### Code style guidelines
+
+We follow PEP8 conventions as code style guidelines, please check [here](https://github.com/BU-ISCIII/BU-ISCIII/wiki/Python#pep-8-guidelines-read-the-full-pep-8-documentation) for more detail.
+
+When developing new code, we strongly recommend to implement LogSum() functions from log_summary.py instead of the classic python logging in order to keep track of all the warnings and errors that may appear during any of the processes.
+
+## Getting help
+
+For further information/help, please ask on the  `#pipelines-dev` slack channel or write us an email! ([[email protected]](emailto:[email protected])).

.github/workflows/pypi_publish.yml

@@ -0,0 +1,88 @@
+name: Publish package python distribution to Pypi
+
+on:
+  push:
+    branches: "main"
+
+jobs:
+    build:
+      name: Build distribution
+      runs-on: ubuntu-latest
+      steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.9.16"
+      - name: Install pypi/build
+        run: >-
+          python3 -m
+          pip install
+          build
+          --user
+      - name: Build a binary wheel and a source tarball
+        run: python3 -m build
+      - name: Store the distribution packages
+        uses: actions/upload-artifact@v3
+        with:
+          name: python-package-distributions
+          path: dist/
+
+    publish-to-pypi:
+      name: Publish dist to PyPI
+      if: startsWith(github.ref, 'refs/tags/')  # only publish to PyPI on tag pushes
+      needs:
+      - build
+      runs-on: ubuntu-latest
+      environment:
+        name: pypi
+        url: https://pypi.org/p/relecov-tools
+      permissions:
+        id-token: write
+      steps:
+      - name: Download all the dists
+        uses: actions/download-artifact@v3
+        with:
+          name: python-package-distributions
+          path: dist/
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+
+    github-release:
+        name: Sign dist with Sigstore and upload to GitHub Release
+        needs:
+        - publish-to-pypi
+        runs-on: ubuntu-latest
+        permissions:
+          contents: write
+          id-token: write
+        steps:
+        - name: Download all the dists
+          uses: actions/download-artifact@v3
+          with:
+            name: python-package-distributions
+            path: dist/
+        - name: Sign the dists with Sigstore
+          uses: sigstore/[email protected]
+          with:
+            inputs: >-
+              ./dist/*.tar.gz
+              ./dist/*.whl
+        - name: Create GitHub Release
+          env:
+            GITHUB_TOKEN: ${{ github.token }}
+          run: >-
+            gh release create
+            '${{ github.ref_name }}'
+            --repo '${{ github.repository }}'
+            --notes ""
+        - name: Upload artifact signatures to GitHub Release
+          env:
+            GITHUB_TOKEN: ${{ github.token }}
+          # Upload to GitHub Release using the `gh` CLI.
+          # `dist/` contains the built packages, and the
+          # sigstore-produced signatures and certificates.
+          run: >-
+            gh release upload
+            '${{ github.ref_name }}' dist/**
+            --repo '${{ github.repository }}'

.github/workflows/python_lint.yml

@@ -0,0 +1,35 @@
+name: python_lint
+
+on:
+  push:
+    paths:
+      - '**.py'
+  pull_request:
+    paths:
+      - '**.py'
+
+jobs:
+  flake8_py3:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Setup Python
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.9.x
+          architecture: x64
+      - name: Checkout PyTorch
+        uses: actions/checkout@master
+      - name: Install flake8
+        run: pip install flake8
+      - name: Run flake8
+        run: flake8 --ignore E501,W503,E203,W605
+
+  black_lint:
+    runs-on: ubuntu-latest
+    steps:
+        - name: Setup
+          uses: actions/checkout@v2
+        - name: Install black in jupyter
+          run: pip install black[jupyter]
+        - name: Check code lints with Black
+          uses: psf/black@stable

.github/workflows/test_modules.yml

@@ -0,0 +1,79 @@
+name: test_modules
+
+on:
+  push:
+    branches: "**"
+  pull_request:
+    types: [opened, reopened, synchronize, closed]
+    branches: "**"
+
+jobs:
+  test_map:
+    runs-on: ubuntu-latest
+    strategy:
+      max-parallel: 2
+      matrix:
+        map_args: ["-d 'ENA' -f relecov_tools/schema/ena_schema.json", "-d 'GISAID' -f relecov_tools/schema/gisaid_schema.json"]
+    steps:
+    - name: Set up Python 3.9.16
+      uses: actions/setup-python@v3
+      with:
+        python-version: '3.9.16'
+    - name: Checkout code
+      uses: actions/checkout@v3
+      with:
+        ref: ${{ github.event.pull_request.head.sha }}
+        fetch-depth: 0
+    - name: Install package and dependencies
+      run: |
+        pip install -r requirements.txt
+        pip install .
+    - name: Run each module tests
+      run: |
+        relecov-tools map -j tests/data/map_validate/processed_metadata_lab_test.json -p relecov_tools/schema/relecov_schema.json ${{ matrix.map_args }} -o .
+      env:
+        OUTPUT_LOCATION: ${{ github.workspace }}/tests/
+    - name: Upload output file
+      uses: actions/upload-artifact@v2
+      with:
+        name: test-output
+        path: output.txt
+
+  test_all_modules:
+    runs-on: ubuntu-latest
+    strategy:
+      max-parallel: 3
+      matrix:
+        modules: ["read-lab-metadata", "read-bioinfo-metadata", "validate"]
+    steps:
+
+    - name: Set up Python 3.9.16
+      uses: actions/setup-python@v3
+      with:
+        python-version: '3.9.16'
+
+    - name: Checkout code
+      uses: actions/checkout@v3
+      with:
+        ref: ${{ github.event.pull_request.head.sha }}
+        fetch-depth: 0
+
+    - name: Install package and dependencies
+      run: |
+        pip install -r requirements.txt
+        pip install .
+
+    - name: Run each module tests
+      run: |
+        if [ "${{ matrix.modules }}" == "read-lab-metadata" ]; then
+          relecov-tools read-lab-metadata -m tests/data/read_lab_metadata/metadata_lab_test.xlsx -s tests/data/read_lab_metadata/samples_data_test.json -o $OUTPUT_LOCATION
+        elif [ "${{ matrix.modules }}" == "validate" ]; then
+          relecov-tools validate -j tests/data/map_validate/processed_metadata_lab_test.json -s relecov_tools/schema/relecov_schema.json -m tests/data/map_validate/metadata_lab_test.xlsx -o $OUTPUT_LOCATION
+        fi
+      env:
+        OUTPUT_LOCATION: ${{ github.workspace }}/tests/
+    - name: Upload output file
+      uses: actions/upload-artifact@v2
+      with:
+        name: test-output
+        path: output.txt

.github/workflows/test_sftp_handle.yml

@@ -0,0 +1,65 @@
+name: test_sftp_handle
+
+on:
+  push:
+    branches: "**"
+  pull_request_target:
+    types: [opened, reopened, synchronize, closed]
+    branches: "**"
+
+jobs:
+  security_check:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Get User Permission
+      id: checkAccess
+      uses: actions-cool/check-user-permission@v2
+      with:
+        require: write
+        username: ${{ github.triggering_actor }}
+    - name: Check User Permission
+      if: steps.checkAccess.outputs.require-result == 'false'
+      run: |
+        echo "${{ github.triggering_actor }} does not have permissions on this repo."
+        echo "Current permission level is ${{ steps.checkAccess.outputs.user-permission }}"
+        echo "Job originally triggered by ${{ github.actor }}"
+        exit 1
+      
+  test_sftp_handle:
+    needs: security_check
+    concurrency:
+      group: ${{ github.repository }}-test_sftp_handle
+      cancel-in-progress: false
+    runs-on: ubuntu-latest
+    strategy:
+      max-parallel: 1
+      matrix:
+        download_options: ["download_only", "download_clean", "delete_only"]
+        target_folders: ["", "-t COD-test-1"]
+
+    steps:
+    - name: Set up Python 3.9.16
+      uses: actions/setup-python@v3
+      with:
+        python-version: '3.9.16'
+
+    - name: Checkout code
+      uses: actions/checkout@v3
+      with:
+        ref: ${{ github.event.pull_request.head.sha }}
+        fetch-depth: 0
+
+    - name: Install package and dependencies
+      run: |
+        pip install -r requirements.txt
+        pip install .
+      
+    - name: Run sftp_handle tests
+      run: |
+        python3 tests/test_sftp_handle.py --download_option ${{ matrix.download_options }} ${{ matrix.target_folders }}
+      env:
+        TEST_USER: ${{ secrets.TEST_USER }}
+        TEST_PASSWORD: ${{ secrets.TEST_PASSWORD }}
+        TEST_PORT: ${{ secrets.TEST_PORT }}
+        OUTPUT_LOCATION: ${{ github.workspace }}/tests/
+

.gitignore

@@ -127,3 +127,6 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# (Ad-hoc files)
+.nextflow*

CHANGELOG.md

@@ -0,0 +1,38 @@
+# relecov-tools Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.1.0dev] - 2024-0X-0X : https://github.com/BU-ISCIII/relecov-tools/releases/tag/1.1.X
+
+### Credits
+
+Code contributions to the hotfix:
+
+### Modules
+
+#### Added enhancements
+
+#### Fixes
+
+#### Changed
+
+#### Removed
+
+### Requirements
+
+## [1.0.0] - 2024-0X-0X : https://github.com/BU-ISCIII/relecov-tools/releases/tag/1.0.0
+
+### Credits
+
+Code contributions to the inital release:
+
+- [Sara Monzón](https://github.com/saramonzon)
+- [Sarai Varona](https://github.com/svarona)
+- [Guillermo Gorines](https://github.com/GuilleGorines)
+- [Pablo Mata](https://github.com/Shettland)
+- [Luis Chapado](https://github.com/luissian)
+- [Erika Kvalem](https://github.com/ErikaKvalem)
+- [Alberto Lema](https://github.com/Alema91)
+- [Daniel Valle](https://github.com/Daniel-VM)

MANIFEST.in

@@ -0,0 +1,6 @@
+include LICENSE
+include README.md
+graft relecov_tools/assets
+graft relecov_tools/schema
+graft relecov_tools/conf
+include requirements.txt