Merge branch 'main' into functional_testing_ci

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,20 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+---
+
+## Describe the Bug
+> A clear and concise description of what the bug is.
+
+## Steps to Reproduce
+> Steps to reproduce the behavior.
+> Please include the version information where the bug was observed.
+
+## Expected Behavior
+> A clear and concise description of what you expected to happen.
+
+## Screenshots
+> If applicable, add screenshots to help explain your problem.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,13 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+---
+
+## Problem Related to the Feature
+> A clear and concise description of what the problem is.
+
+## Proposed Solution
+> A clear and concise description of what you want to happen.

.github/ISSUE_TEMPLATE/general_question.md

@@ -0,0 +1,9 @@
+---
+name: General question
+about: Ask a question or seek clarification about the project
+title: ''
+labels: 'question'
+assignees: ''
+---
+
+> Please provide a detailed description of your question or the information you seek.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,13 @@
+## Summary
+Provide a concise summary of the changes introduced by this pull request. Detail the purpose and scope of the changes, referencing any relevant issues or discussions. Explain how these changes address the problem or improve the project.
+
+## Test Plan
+In this section, describe the testing you have performed to verify the changes. Include:
+- A clear description of the testing environment.
+- The steps you followed to test the new features or bug fixes.
+- Any specific commands used during testing, along with their outputs.
+- A description of the results and observations from your testing.
+This information is crucial for reviewers to understand how the changes have been validated.
+
+## Additional Notes
+Include any other notes or comments about the pull request here. This can include challenges faced, future considerations, or context that reviewers might find helpful.

.github/workflows/ci.yml

@@ -0,0 +1,55 @@
+name: CI
+
+on:
+  pull_request:
+
+jobs:
+
+  lint:
+    name: Linting
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.9
+
+      - name: Install dependencies
+        run: pip install -r requirements-dev.txt
+
+      - name: Run ruff linter
+        run: ruff check .
+
+      - name: Run ruff formatter
+        run: ruff format --check --diff .
+
+      - name: Run pyright
+        run: pyright .
+
+      - name: Run vulture check
+        run: vulture src/ tests/ ci_tools/ --min-confidence 100
+
+  test:
+    name: Run pytest
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.9
+
+      - name: Install dependencies
+        run: pip install pytest -r requirements-dev.txt
+
+      - name: Run pytest
+        run: pytest -vv

.gitignore

@@ -0,0 +1,91 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+# Usually these files are written by a python script from a template
+# before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# pytype static type analyzer
+.pytype/
+
+# pycharm
+.idea/
+
+# VSCode
+.vscode/
+
+# Editors and IDEs
+*.swp
+*.bak
+*.tmp
+*~
+*.sublime-project
+*.sublime-workspace
+
+# OS generated files
+.DS_Store
+.DS_Store?
+._*
+.Spotlight-V100
+.Trashes
+ehthumbs.db
+Thumbs.db
+
+*.log
+install/
+results/
+.*

README.md

@@ -33,43 +33,43 @@ Cloud AI supports five modes: install, dry-run, run, generate-report, and uninst
 * Use the generate-report mode to generate reports under the test directories alongside the raw data.
 * Use the uninstall mode to remove installed test templates.
 
-To install test templates, run main.py in install mode.
+To install test templates, run Cloud AI CLI in install mode.
 Please make sure to use the correct system configuration file that corresponds to your current setup for installation and experiments.
 ```bash
-python main.py\
+cloudai\
     --mode install\
     --system_config_path conf/v0.6/general/system/example_slurm_cluster.toml
 ```
 
 To simulate running experiments without execution, use the dry-run mode:
 ```bash
-python main.py\
+cloudai\
     --mode dry-run\
     --system_config_path conf/v0.6/general/system/example_slurm_cluster.toml\
     --test_scenario_path conf/v0.6/general/test_scenario/sleep/test_scenario.toml
 ```
 
-To run experiments, execute main.py in run mode:
+To run experiments, execute Cloud AI CLI in run mode:
 ```bash
-python main.py\
+cloudai\
     --mode run\
     --system_config_path conf/v0.6/general/system/example_slurm_cluster.toml\
     --test_scenario_path conf/v0.6/general/test_scenario/sleep/test_scenario.toml
 ```
 
-To generate reports, execute main.py in generate-report mode:
+To generate reports, execute Cloud AI CLI in generate-report mode:
 ```bash
-python main.py\
+cloudai\
     --mode generate-report\
     --system_config_path conf/v0.6/general/system/example_slurm_cluster.toml\
     --output_path /path/to/output_directory
 ```
 In the generate-report mode, use the --output_path argument to specify a subdirectory under the result directory.
 This subdirectory is usually named with a timestamp for unique identification.
 
-To uninstall test templates, run main.py in uninstall mode:
+To uninstall test templates, run Cloud AI CLI in uninstall mode:
 ```bash
-python main.py\
+cloudai\
     --mode uninstall\
     --system_config_path conf/v0.6/general/system/example_slurm_cluster.toml
 ```

ci_tools/functional_tests/run_functional_test.sh

@@ -1,6 +1,7 @@
 #!/bin/bash
 
 VERBOSE=true
+CLOUDAI_PATH="./src/"
 
 if [ $# -ne 2 ]; then
     echo "Usage: $0 <test_scenario_path> <expected_output_path>"
@@ -51,9 +52,12 @@ dirs_diff() {
     fi
 }
 
+export PYTHONPATH=$PYTHONPATH:$CLOUDAI_PATH
+scenario_path=$1
+expected_output_path=$2
 
-scenario_path="$1"
-expected_output_path="$2"
+$VERBOSE && echo "Scenario dir: $scenario_path"
+$VERBOSE && echo "Expected output dir: $expected_output_path"
 
 if [ ! -f "$scenario_path" ]; then
     >&2 echo "Error: Scenario $scenario is not valid, can't find path $scenario_path."
@@ -64,19 +68,24 @@ fi
 
 last_result_before=$(ls results/ -la -X | tail -n 3 | head -n 1 | awk '{print $NF}')
 
-python main.py \
+python -m cloudai \
     --mode dry-run\
     --system_config_path "ci_tools/functional_tests/system_config.toml" \
     --test_scenario_path $scenario_path
 
+if [ $? -ne 0 ]; then
+    echo "Tests failed"
+    exit 1
+fi
+
 last_result=$(ls results/ -la -X | tail -n 3 | head -n 1 | awk '{print $NF}')
 
 if [ "$last_result_before" == "$last_result" ]; then
     >&2 echo "No new result added after running cloudai dry run."
     exit 1
 fi
 
-last_result_path="results/$last_result"
+last_result_path="results/$last_result/"
 
 dirs_diff "$expected_output_path" "$last_result_path"
 is_diff=$?

pyproject.toml

@@ -1,9 +1,16 @@
 [project]
 name = "cloudai"
 version = "0.6"
+dependencies = [
+    "bokeh==3.4.1",
+    "pandas==2.2.1",
+    "requests==2.31.0",
+    "tbparse==0.0.8",
+    "toml==0.10.2",
+]
 
-[tool.setuptools.packages.find]
-where = ["cloudai"]
+[project.scripts]
+cloudai = "cloudai.__main__:main"
 
 [tool.ruff]
 target-version = "py39"

requirements-dev.txt

@@ -3,3 +3,6 @@ pytest==8.1.1
 ruff==0.3.7
 pandas-stubs==2.2.*
 pyright==1.1.359
+build==1.2.*
+vulture==2.11
+

cloudai/schema/test_template/jax_toolbox/report_generation_strategy.py → src/cloudai/schema/test_template/jax_toolbox/report_generation_strategy.py

@@ -49,38 +49,41 @@ def generate_report(self, directory_path: str, sol: Optional[float] = None) -> N
                 "max": max(times),
                 "average": sum(times) / len(times),
                 "median": statistics.median(times),
+                "stdev": statistics.stdev(times) if len(times) > 1 else 0,
             }
             self._write_report(directory_path, stats)
 
     def _extract_times(self, directory_path: str) -> List[float]:
         """
         Extracts elapsed times from all error files matching the pattern in the directory,
-        excluding the first time value recorded in each file.
+        starting after the 10th occurrence of a line matching the "[PAX STATUS]: train_step() took" pattern.
 
         Args:
             directory_path (str): Directory containing error files.
 
         Returns:
-            List[float]: List of extracted times as floats, after excluding the first time from each file.
+            List[float]: List of extracted times as floats, starting from the epoch after the 10th occurrence.
         """
         times = []
         error_files = glob.glob(os.path.join(directory_path, "error-*.txt"))
         for stderr_path in error_files:
             file_times = []
+            epoch_count = 0
             with open(stderr_path, "r") as file:
                 for line in file:
-                    if "Elapsed time for" in line and "run" in line and ":434" in line:
-                        parts = line.split()
-                        time_str = parts[parts.index("<run>:") + 1]
-                        try:
-                            time_value = float(time_str.split("seconds")[0])
-                            file_times.append(time_value)
-                        except ValueError:
-                            continue  # Skip any lines where conversion fails
+                    if "[PAX STATUS]: train_step() took" in line:
+                        epoch_count += 1
+                        if epoch_count > 10:  # Start recording times after 10 epochs
+                            # Extract the time value right after the keyword
+                            parts = line.split("took")
+                            time_str = parts[1].strip().split("seconds")[0].strip()
+                            try:
+                                time_value = float(time_str)
+                                file_times.append(time_value)
+                            except ValueError:
+                                continue  # Skip any lines where conversion fails
 
-            # Exclude the first time record from each file if it exists
-            if file_times:
-                times.extend(file_times[1:])
+            times.extend(file_times)
 
         return times