Run tests in Slurm test env (#88)

* Adds CI for running tests (#15)

* Adds CI for running tests

* Fix CI typo

* Adds setup step for python

* Adds minimal pyproject.toml file

* Update pyproject.toml

* Adds missing --with directive

* Adds missing description field

* Adds missing docstring [skip ci]

* Updates the way coverage is run

* Expand .gitignore

* Exceptions top (#16)

* Moves exception pattern checking into dedicated method

* Update top of file comments for clarity

* Deletes test placeholder

* Fix boolean typo

* Add codacy badges to readme (#17)

* Adds tests for piping shell commands (#20)

* Drop accumulator pattern for node names (#21)

* Use builtin support for logging (#22)

* Switch to builtin logging

* Explicitly set logging level

* Moves logging to /var

* Log directly to syslog

* Add missing import

* Replace accumulator (#23)

* Replace accumulator patterns with single iteration

* Fix typo in list unpacking

* Eod cleanup (#24)

* Drop support for regex ignore patterns

* Increase number/detail of log messages

* Drop Python2 support and impliment Python3 features

* Delete test_check_ignore_node.py

* Simplify set logic

* Adds ability to whitelist UID and UID ranges (#25)

* Adds basic whitelist

* Replaces usernames with uid

* Updates docs and comments

* Ignore uid from 0 to 15000

* Formally package source code for pip installability (#28)

* Move source into a package

* Add GPL3 license

* Update pyproject.toml

* Removes 'if __name__' conditional

* Drop /home/djperrefort/GitHub/pitt-crc/shinigami reference in testing CI

* Update test suite structure to match package structure

* Add CI workflow for publishing to PyPI (#30)

* Updates Readme and Package Docs (#31)

* Updates README

* Updates package docstring

* Adds ability to load settings from settings file (#32)

* Introduces settings module

* Configure settings to load from disk

* Add dedicated method for loading settings from disk

* Adds ability to skip FileNotFoundError error

* Adds basic CLI parsing (#34)

* Initial commit of cli module

* Fixes outdated import signature

* Mves main function into Application class

* Mid-flight cleanup pass

* Updates import signatures in test suite

* Adds support for async execution across nodes in the same cluster (#35)

* Execute SSH commands asynchronously

* Run async functins using asyncio.gather

* Fix typo in SSH connection

* Run all remote commands through SSH object

* Imposes limit on max SSH connections

* PEP8

* Drops old tests

* Adds dummy test coverage for ci

* Drops Settings.load_from_disk method (#36)

* Adds debug option to CLI (#37)

* Build out test coverage (#38)

* Adds tests for parser debug option

* Adds tests for Settings class

* Adds tests for id_in_whitelist function

* Adds verbosity argument to CLI (#39)

* Adds configurable logging

* Adds verbosity argument

* Adds tests for logging configuration

* Adds tests for CLI --debug option (#40)

* Delete lock file

* Fix syntax typo in async generator

* Eliminate global semaphre and glbal settings to address cncurrency locks

* Logs errors collected by asyncio.gather

* Move semaphore inside event loop

* Adds nde name to SSH related logs

* Pre-review cleanup

* Minor capitalization typos

* Drop yaml for json

* Update setup instructions README.md

There was a typo in the cron job example.

* Run AI linter (#50)

* Makes Application.settings private

* Runs AI linter

* Adds ssh timeout option (#49)

* Drop whitelisting functionality for GIDs (#51)

* Drops GID whitelist

* Drops GID related tests

* Move logic for loading settings into Settings class (#53)

* Move logic for loading settings into Settings class

* Revert changes to parser class

* Raise error on file not found

* Revert last commit

* Terminate orphaned processes matching blacklisted user IDs (#55)

* Terminate orphaned processes in userlist

* Rename whitelist to blacklist

* Kill processes using group ID

* Fix bug in fetching of remote process data

* updates tests

* Lower pandas requirement

* Adds test coverage for settings file parsing (#56)

* Adds test coverage for settings file parsing

* PEP8

* Update testing CI to use slurm

* Adds rudementary test to try out slurm support

* Replace settings file with CLI arguments (#80)

* Outline CLI options to replace settings module

* Print help text on error

* Parse UID list as json string

* Dynamically determine valid arguments from func signature

* Add log message when there are no processes to terminate

* Add nargs='+' to uid-whitelist arg

* Updates docstring

* Adds missing type hints

* Introduce dedicat ssh argument group

* Abstract away SSH options in function signatures

* Adds tests for argument parsing

* Drops application tests

* Makes logging setup private

* Shorten line lengths

* Merge V0.3.3 updates into v0.4.x (#86)

* Bump actions/checkout from 3 to 4 (#81)

Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Updates CI (#82)

* Fix termination signal (#84)

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix conflicts in workflow files

* Add docker test env back into testing matrix

* Drops applciation logging tests

* Updates test node names

* Adds newer python versions to CI

* Updats expected node names

* Run codacy report from bash shell

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

.github/workflows/PackageTest.yml

@@ -14,17 +14,29 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ "3.8", "3.9", "3.10", "3.11" ]
+        slurm_version:
+          - 20.11.9.1
+          - 22.05.2.1
+          - 23.02.5.1
+        python_version:
+          - 3.8
+          - 3.9
+          - 3.10
+          - 3.11
+
+    container:
+      image: ghcr.io/pitt-crc/test-env:${{ matrix.slurm_version }}
+      credentials:
+        username: ${{ github.actor }}
+        password: ${{ secrets.GITHUB_TOKEN }}
 
     steps:
+      - name: Setup environment
+        run: /usr/local/bin/entrypoint.sh
+
       - name: Checkout repository
         uses: actions/checkout@v4
 
-      - name: Setup Python
-        uses: actions/setup-python@v4
-        with:
-          python-version: ${{ matrix.python-version }}
-
       - name: Install Poetry
         uses: snok/install-poetry@v1
         with:
@@ -43,6 +55,7 @@ jobs:
       - name: Report partial coverage results
         if: github.event_name != 'release'
         run: bash <(curl -Ls https://coverage.codacy.com/get.sh) report --partial -l Python -r coverage.xml
+        shell: bash
         env:
           CODACY_PROJECT_TOKEN: ${{ secrets.CODACY_PROJECT_TOKEN }}
 

pyproject.toml

@@ -31,7 +31,7 @@ shinigami = "shinigami.cli:Application.execute"
 python = ">=3.8"
 pydantic = "^2.0.3"
 pydantic-settings = "^2.0.2"
-asyncssh = {extras = ["bcrypt", "fido2"], version = "^2.13.2"}
+asyncssh = { extras = ["bcrypt", "fido2"], version = "^2.13.2" }
 pandas = "1.5.3"  # Newer versions are incompatible with Python 3.8
 
 [tool.poetry.group.tests]

tests/utils/test_get_nodes.py

@@ -0,0 +1,19 @@
+"""Tests for the ``utils.get_nodes`` function"""
+
+from unittest import TestCase
+
+from shinigami import utils
+
+# For information on resources defined in the testing environment,
+# see https://github.com/pitt-crc/Slurm-Test-Environment/
+TEST_CLUSTER = 'development'
+TEST_NODES = set(f'c{i}' for i in range(1, 11))
+
+
+class NodesMatchTestEnvironment(TestCase):
+    """Test the returned node list matches values defined in the testing environment"""
+
+    def test_returned_nodes(self) -> None:
+        """Test returned nodes match hose defined in the slurm test env"""
+
+        self.assertSequenceEqual(TEST_NODES, utils.get_nodes(TEST_CLUSTER, []))