Merge branch 'branch-25.02' of github.com:nv-morpheus/Morpheus into d…

…avid-automate-dep-fetch

.github/workflows/pr.yaml

@@ -20,6 +20,7 @@ on:
     branches:
       - 'pull-request/**'
       - 'branch-*'
+      - 'main'
 
 # This allows a subsequently queued workflow run to interrupt previous runs
 concurrency:

CHANGELOG.md

@@ -15,6 +15,19 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
+# Morpheus 24.10.01 (22 Nov 2024)
+
+## 🐛 Bug Fixes
+
+- Pin mlflow version to avoid breaking changes in v2.18 ([#2067](https://github.com/nv-morpheus/Morpheus/pull/2067)) [@dagardner-nv](https://github.com/dagardner-nv)
+- Execute CI on the main branch ([#2064](https://github.com/nv-morpheus/Morpheus/pull/2064)) [@dagardner-nv](https://github.com/dagardner-nv)
+
+## 📖 Documentation
+
+- Remove references to pipeline-ae in docs ([#2063](https://github.com/nv-morpheus/Morpheus/pull/2063)) [@dagardner-nv](https://github.com/dagardner-nv)
+- Document location of third party source repository ([#2059](https://github.com/nv-morpheus/Morpheus/pull/2059)) [@dagardner-nv](https://github.com/dagardner-nv)
+- Update DFP class and file paths ([#2052](https://github.com/nv-morpheus/Morpheus/pull/2052)) [@dagardner-nv](https://github.com/dagardner-nv)
+
 # Morpheus 24.10.00 (01 Nov 2024)
 
 ## 🚨 Breaking Changes

ci/conda/recipes/morpheus-libs/meta.yaml

@@ -91,7 +91,7 @@ outputs:
         - feedparser =6.0.*
         - grpcio =1.62.*
         - lxml
-        - mlflow>=2.10.0,<3
+        - mlflow>=2.10.0,<2.18
         - mrc
         - networkx=2.8.8
         - numpydoc =1.5.*
@@ -239,7 +239,7 @@ outputs:
         - scripts/fetch_data.py
         - tests/*
       script: morpheus_llm_test.sh
- 
+
 about:
   home: https://github.com/nv-morpheus/Morpheus
   license: Apache-2.0

ci/conda/recipes/morpheus/meta.yaml

@@ -97,7 +97,7 @@ outputs:
         - feedparser =6.0.*
         - grpcio =1.62.*
         - libwebp>=1.3.2 # Required for CVE mitigation: https://nvd.nist.gov/vuln/detail/CVE-2023-4863
-        - mlflow>=2.10.0,<3
+        - mlflow>=2.10.0,<2.18
         - mrc
         - networkx=2.8.8
         - numpydoc =1.5.*

cmake/dependencies_core.cmake

@@ -19,6 +19,10 @@ list(APPEND CMAKE_MESSAGE_CONTEXT "dep_core")
 # =========
 morpheus_utils_configure_cccl()
 
+# indicators
+# ==========
+morpheus_utils_configure_indicators()
+
 # matx
 # ====
 morpheus_utils_configure_matx()

conda/environments/all_cuda-125_arch-x86_64.yaml

@@ -53,6 +53,7 @@ dependencies:
 - gxx=12.1
 - huggingface_hub=0.20.2
 - include-what-you-use=0.20
+- indicators=2.3
 - ipython
 - isort
 - jsonpatch>=1.33

conda/environments/dev_cuda-125_arch-x86_64.yaml

@@ -45,6 +45,7 @@ dependencies:
 - gtest=1.14
 - gxx=12.1
 - include-what-you-use=0.20
+- indicators=2.3
 - ipython
 - isort
 - libcublas-dev

dependencies.yaml

@@ -277,6 +277,7 @@ dependencies:
           - cmake=3.27
           - cuda-cudart-dev=12.5
           - cuda-version=12.5
+          - indicators=2.3 # C++ library for displaying progress bars
           - libtool # Needed for DOCA build
           - ninja=1.11
           - pkg-config=0.29 # for mrc cmake

docs/source/developer_guide/guides/2_real_world_phishing.md

@@ -482,7 +482,6 @@ To explicitly set the output format we could specify the `file_type` argument to
 ```python
 import logging
 import os
-import tempfile
 
 import click
 
@@ -542,7 +541,7 @@ MORPHEUS_ROOT = os.environ['MORPHEUS_ROOT']
 @click.option("--server_url", default='localhost:8000', help="Tritonserver url.")
 @click.option(
     "--output_file",
-    default=os.path.join(tempfile.gettempdir(), "detections.jsonlines"),
+    default=".tmp/output/phishing_detections.jsonlines",
     help="The path to the file where the inference output will be saved.",
 )
 def run_pipeline(use_stage_function: bool,
@@ -633,7 +632,7 @@ morpheus --log_level=debug --plugin examples/developer_guide/2_1_real_world_phis
   monitor --description="Inference Rate" --smoothing=0.001 --unit=inf \
   add-scores --label=is_phishing \
   serialize \
-  to-file --filename=/tmp/detections.jsonlines --overwrite
+  to-file --filename=.tmp/output/phishing_detections_cli.jsonlines --overwrite
 ```
 
 ## Stage Constructors

docs/source/getting_started.md

@@ -376,35 +376,6 @@ Commands:
   validate        Validate pipeline output for testing.
 ```
 
-And for the AE pipeline:
-
-```
-$ morpheus run pipeline-ae --help
-Usage: morpheus run pipeline-ae [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...
-
-<Help Paragraph Omitted>
-
-Commands:
-  add-class        Add detected classifications to each message.
-  add-scores       Add probability scores to each message.
-  buffer           (Deprecated) Buffer results.
-  delay            (Deprecated) Delay results for a certain duration.
-  filter           Filter message by a classification threshold.
-  from-azure       Source stage is used to load Azure Active Directory messages.
-  from-cloudtrail  Load messages from a CloudTrail directory.
-  from-duo         Source stage is used to load Duo Authentication messages.
-  inf-pytorch      Perform inference with PyTorch.
-  inf-triton       Perform inference with Triton Inference Server.
-  monitor          Display throughput numbers at a specific point in the pipeline.
-  preprocess       Prepare Autoencoder input DataFrames for inference.
-  serialize        Includes & excludes columns from messages.
-  timeseries       Perform time series anomaly detection and add prediction.
-  to-file          Write all messages to a file.
-  to-kafka         Write all messages to a Kafka cluster.
-  train-ae         Train an Autoencoder model on incoming data.
-  trigger          Buffer data until the previous stage has completed.
-  validate         Validate pipeline output for testing.
-```
 > **Note**: The available commands for different types of pipelines are not the same. This means that the same stage may have different options when used in different pipelines. Check the CLI help for the most up-to-date information during development.
 
 ## Next Steps

examples/abp_nvsmi_detection/README.md

@@ -140,7 +140,7 @@ morpheus --log_level=DEBUG \
    `# 7th Stage: Convert from objects back into strings. Ignore verbose input data` \
    serialize --include 'mining' \
    `# 8th Stage: Write out the JSON lines to the detections.jsonlines file` \
-   to-file --filename=detections.jsonlines --overwrite
+   to-file --filename=.tmp/output/abp_nvsmi_detections.jsonlines --overwrite
 ```
 
 If successful, the following should be displayed:
@@ -217,15 +217,15 @@ Added stage: <add-class-5; AddClassificationsStage(threshold=0.5, labels=[], pre
   └─ morpheus.ControlMessage -> morpheus.ControlMessage
 Added stage: <serialize-6; SerializeStage(include=['mining'], exclude=['^ID$', '^_ts_'], fixed_columns=True)>
   └─ morpheus.ControlMessage -> morpheus.MessageMeta
-Added stage: <to-file-7; WriteToFileStage(filename=detections.jsonlines, overwrite=True, file_type=FileTypes.Auto)>
+Added stage: <to-file-7; WriteToFileStage(filename=.tmp/output/abp_nvsmi_detections.jsonlines, overwrite=True, file_type=FileTypes.Auto)>
   └─ morpheus.MessageMeta -> morpheus.MessageMeta
 ====Building Pipeline Complete!====
 Starting! Time: 1656353254.9919598
 Inference Rate[Complete]: 1242inf [00:00, 1863.04inf/s]
 ====Pipeline Complete====
 ```
 
-The output file `detections.jsonlines` will contain a single boolean value for each input line. At some point the values will switch from `0` to `1`:
+The output file `.tmp/output/detections.jsonlines` will contain a single boolean value for each input line. At some point the values will switch from `0` to `1`:
 
 ```
 ...

examples/abp_pcap_detection/README.md

@@ -97,7 +97,7 @@ python examples/abp_pcap_detection/run.py
 ```
 Note: Both Morpheus and Triton Inference Server containers must have access to the same GPUs in order for this example to work.
 
-The pipeline will process the input `abp_pcap_dump.jsonlines` sample data and write it to `pcap_out.jsonlines`.
+The pipeline will process the input `abp_pcap_dump.jsonlines` sample data and write it to `.tmp/output/pcap_out.jsonlines`.
 
 ### CLI Example
 The above example is illustrative of using the Python API to build a custom Morpheus Pipeline.
@@ -118,6 +118,6 @@ morpheus --log_level INFO --plugin "examples/abp_pcap_detection/abp_pcap_preproc
     monitor --description "Add classification rate" --unit "add-class" \
     serialize \
     monitor --description "Serialize rate" --unit ser \
-    to-file --filename "pcap_out.jsonlines" --overwrite \
+    to-file --filename ".tmp/output/pcap_out_cli.jsonlines" --overwrite \
     monitor --description "Write to file rate" --unit "to-file"
 ```

examples/abp_pcap_detection/run.py

@@ -65,7 +65,7 @@
 )
 @click.option(
     "--output_file",
-    default="./pcap_out.jsonlines",
+    default="./.tmp/output/pcap_out.jsonlines",
     help="The path to the file where the inference output will be saved.",
 )
 @click.option(

examples/cpu_only/README.md

@@ -53,7 +53,7 @@ Options:
 To launch the configured Morpheus pipeline with the sample data that is provided in `examples/data`, run the following:
 
 ```bash
-python examples/cpu_only/run.py --use_cpu_only --in_file=examples/data/email.jsonlines --out_file=.tmp/out.jsonlines
+python examples/cpu_only/run.py --use_cpu_only --in_file=examples/data/email.jsonlines --out_file=.tmp/output/cpu_only_out.jsonlines
 ```
 
 ### CLI Example
@@ -68,5 +68,5 @@ morpheus --log_level INFO \
     deserialize \
     monitor --description "deserialize" \
     serialize \
-    to-file --filename=.tmp/out.jsonlines --overwrite
+    to-file --filename=.tmp/output/cpu_only_cli_out.jsonlines --overwrite
 ```

examples/cpu_only/run.py

@@ -61,7 +61,7 @@
     "--out_file",
     help="Output file",
     type=click.Path(dir_okay=False),
-    default="output.csv",
+    default=".tmp/output/cpu_only_out.csv",
     required=True,
 )
 def run_pipeline(log_level: int, use_cpu_only: bool, in_file: pathlib.Path, out_file: pathlib.Path):

examples/developer_guide/2_1_real_world_phishing/run.py

@@ -17,7 +17,6 @@
 
 import logging
 import os
-import tempfile
 
 import click
 from recipient_features_stage import RecipientFeaturesStage
@@ -77,7 +76,7 @@
 @click.option("--server_url", default='localhost:8000', help="Tritonserver url.")
 @click.option(
     "--output_file",
-    default=os.path.join(tempfile.gettempdir(), "detections.jsonlines"),
+    default=".tmp/output/phishing_detections.jsonlines",
     help="The path to the file where the inference output will be saved.",
 )
 def run_pipeline(use_stage_function: bool,

examples/gnn_fraud_detection_pipeline/README.md

@@ -97,7 +97,7 @@ Added stage: <serialize-8; SerializeStage(include=None, exclude=None, fixed_colu
   └─ morpheus.ControlMessage -> morpheus.MessageMeta
 Added stage: <monitor-9; MonitorStage(description=Serialize rate, smoothing=0.05, unit=messages, delayed_start=False, determine_count_fn=None, log_level=LogLevels.INFO)>
   └─ morpheus.MessageMeta -> morpheus.MessageMeta
-Added stage: <to-file-10; WriteToFileStage(filename=output.csv, overwrite=True, file_type=FileTypes.Auto, include_index_col=True, flush=False)>
+Added stage: <to-file-10; WriteToFileStage(filename=.tmp/output/gnn_fraud_detection_output.csv, overwrite=True, file_type=FileTypes.Auto, include_index_col=True, flush=False)>
   └─ morpheus.MessageMeta -> morpheus.MessageMeta
 ====Building Segment Complete!====
 Graph construction rate[Complete]: 265 messages [00:00, 1016.18 messages/s]
@@ -128,5 +128,5 @@ morpheus --log_level INFO \
 	gnn-fraud-classification --model_xgb_file examples/gnn_fraud_detection_pipeline/model/xgb.pt \
 	monitor --description "Add classification rate" \
 	serialize \
-	to-file --filename "output.csv" --overwrite
+	to-file --filename "gnn_fraud_detection_cli_output.csv" --overwrite
 ```

examples/gnn_fraud_detection_pipeline/run.py

@@ -84,7 +84,7 @@
 @click.option(
     "--output_file",
     type=click.Path(dir_okay=False),
-    default="output.csv",
+    default=".tmp/output/gnn_fraud_detection_output.csv",
     help="The path to the file where the inference output will be saved.",
 )
 def run_pipeline(num_threads,

examples/log_parsing/README.md

@@ -119,6 +119,6 @@ morpheus --log_level INFO \
 	monitor --description "Inference rate" --unit inf \
 	log-postprocess --vocab_path ./models/training-tuning-scripts/sid-models/resources/bert-base-cased-vocab.txt \
 		--model_config_path=./models/log-parsing-models/log-parsing-config-20220418.json \
-	to-file --filename ./log-parsing-output.jsonlines --overwrite  \
+	to-file --filename .tmp/output/log-parsing-cli-output.jsonlines --overwrite  \
 	monitor --description "Postprocessing rate"
 ```

examples/log_parsing/run.py

@@ -60,7 +60,7 @@
 )
 @click.option(
     "--output_file",
-    default="log-parsing-output.jsonlines",
+    default=".tmp/output/log-parsing-output.jsonlines",
     help="The path to the file where the inference output will be saved.",
 )
 @click.option('--model_vocab_hash_file',

examples/nlp_si_detection/README.md

@@ -131,8 +131,8 @@ morpheus --log_level=DEBUG \
    filter --filter_source=TENSOR \
    `# 8th Stage: Convert from objects back into strings` \
    serialize --exclude '^_ts_' \
-   `# 9th Stage: Write out the JSON lines to the detections.jsonlines file` \
-   to-file --filename=detections.jsonlines --overwrite
+   `# 9th Stage: Write out the JSON lines to the nlp_si_detections.jsonlines file` \
+   to-file --filename=.tmp/output/nlp_si_detections.jsonlines --overwrite
 ```
 
 If successful, the following should be displayed:
@@ -187,7 +187,7 @@ Added stage: <add-class-5; AddClassificationsStage(threshold=0.5, labels=[], pre
   └─ morpheus.ControlMessage -> morpheus.ControlMessage
 Added stage: <serialize-6; SerializeStage(include=[], exclude=['^_ts_'], fixed_columns=True)>
   └─ morpheus.ControlMessage -> morpheus.MessageMeta
-Added stage: <to-file-7; WriteToFileStage(filename=detections.jsonlines, overwrite=True, file_type=FileTypes.Auto)>
+Added stage: <to-file-7; WriteToFileStage(filename=.tmp/output/nlp_si_detections.jsonlines, overwrite=True, file_type=FileTypes.Auto)>
   └─ morpheus.MessageMeta -> morpheus.MessageMeta
 ====Building Pipeline Complete!====
 Starting! Time: 1656352480.541071
@@ -196,7 +196,7 @@ Inference Rate[Complete]: 93085inf [00:07, 12673.63inf/s]
 
 ```
 
-The output file `detections.jsonlines` will contain the original PCAP messages with the following additional fields added:
+The output file `.tmp/output/nlp_si_detections.jsonlines` will contain the original PCAP messages with the following additional fields added:
 * `address`
 * `bank_acct`
 * `credit_card`

examples/nlp_si_detection/run.sh

@@ -29,4 +29,4 @@ morpheus --log_level=DEBUG \
     add-class \
     filter --filter_source=TENSOR \
     serialize --exclude '^_ts_' \
-    to-file --filename=detections.jsonlines --overwrite
+    to-file --filename=.tmp/output/nlp_si_detections.jsonlines --overwrite

examples/ransomware_detection/README.md

@@ -72,7 +72,7 @@ python examples/ransomware_detection/run.py --server_url=localhost:8000 \
               --sliding_window=3 \
               --model_name=ransomw-model-short-rf \
               --input_glob=./examples/data/appshield/*/snapshot-*/*.json \
-              --output_file=./ransomware_detection_output.jsonlines
+              --output_file=.tmp/output/ransomware_detection_output.jsonlines
 ```
 
 Input features for a short model can be taken from every three snapshots sequence, such as (1, 2, 3), or (2, 3, 4). The sliding window represents the number of subsequent snapshots that need to be taken into consideration when generating the input for a model. Sliding window for the medium model is `5` and for the long model it is `10`.