[NIGHTLY] v24.10.00

🔗 Links

• Development Branch
• Compare with main branch

🚨 Breaking Changes

• Support LLM pipelines in CPU-only mode (#1906) @dagardner-nv
• Remove Starter Digital Fingerprinting (DFP) (#1903) @efajardo-nv
• Finalize removing MultiMessage from Morpheus (#1886) @yczhang-nv
• Add support for a CPU-only Mode (#1851) @dagardner-nv
• Removing support for MultiMessage from stages (#1803) @yczhang-nv

🐛 Bug Fixes

• Pin mlflow version to avoid breaking changes in v2.18 (#2067) @dagardner-nv
• Execute CI on the main branch (#2064) @dagardner-nv
• Various Lint fixes (#2041) @dagardner-nv
• Pin boto3 and s3fs to compatible versions to resolve access denied errors (#2039) @dagardner-nv
• Fix phishing Python API example to match CLI example (#2037) @dagardner-nv
• Model updates and cleanup following upgrade to to triton 24.09 (#2036) @AnuradhaKaruppiah
• Copy data files needed by root_cause_analysis to examples/data (#2032) @AnuradhaKaruppiah
• Fix for duplicate row IDs in log_parsing output (#2031) @dagardner-nv
• Fix log_parsing example pipeline null output issue (#2024) @yczhang-nv
• Fixup file paths in the modular digital fingerprinting documentation. (#2016) @AnuradhaKaruppiah
• Fix DeserializeStage to ensure output messages correctly contain the correct rows for each batch (#2015) @dagardner-nv
• Fix offset calculation when taking a slice of a SlicedMessageMeta (#2006) @dagardner-nv
• Fix CUDF's Column.from_column_view by copying it and adjusting. (#2004) @cwharris
• Fix up file paths in the DFP README (#2003) @AnuradhaKaruppiah
• Fix AttributeError: 'int' object has no attribute 'item' (#1995) @dagardner-nv
• Fix utilities submodule commit (#1987) @cwharris
• Update val-run-all.sh to run cpp pipeline only (#1986) @yczhang-nv
• Fix onnx-to-trt utility (#1984) @dagardner-nv
• Update Utilities submodule and fix compilation with latest build of MRC (#1981) @mdemoret-nv
• Fix missing dependency in DFP Grafana example (#1977) @efajardo-nv
• Populate all the LFS data needed for running examples within the release container (#1976) @AnuradhaKaruppiah
• Ensure timestamps are copied in LLMEngineStage (#1975) @dagardner-nv
• Install sentence-transformers via pip to avoid CPU-torch conda dependencies (#1974) @efajardo-nv
• Add **kwargs back to NVFoundationLLMClient.generate_batch() and generate_batch_async() (#1967) @ashsong-nv
• Benchmark updates/fixes (#1958) @efajardo-nv
• Improve test performance (#1953) @dagardner-nv
• Adopt updated utilities fix in-place Python installs (#1952) @dagardner-nv
• Update cuda version for docker containers (#1941) @dagardner-nv
• Multiple fixes related to SharedProcessPool & MultiProcessingStage (#1940) @yczhang-nv
• Fix dask error in DFP Integrated training pipeline (#1931) @dagardner-nv
• Remove SharedProcessPool.terminate() related tests to avoid stack traces and blocking remote-ci (#1929) @yczhang-nv
• Provide a timeout to the queue.get call in HttpServerSourceStage to avoid spinlocking (#1928) @dagardner-nv
• Ensure that pytest is able to run without optional dependencies (#1927) @dagardner-nv
• Better handle exceptions generated in the LLMEngine to not show the stoul error (#1922) @mdemoret-nv
• Fixing the docker build when Morpheus is a submodule (#1914) @mdemoret-nv
• Build morpheus_llm by default (#1911) @AnuradhaKaruppiah
• Fix conda path for missing llm packages (#1907) @dagardner-nv
• Update WriteToVectorDBStage to re-raise errors from the underlying database (#1905) @dagardner-nv
• Avoid memory leak warnings from pypdfium2 (#1902) @dagardner-nv
• Remove redundant copy of the load_labels_file method (#1901) @dagardner-nv
• Fix Can't find 'action.yml' CI error (#1896) @dagardner-nv
• Fix DFP integrated training Azure pipeline (#1894) @yczhang-nv
• Drop 'CI Pipeline / Check' dependency from the 'package-core' job (#1885) @AnuradhaKaruppiah
• Python source stages now optionally receive a reference to mrc.Subscription (#1881) @dagardner-nv
• Fix Unregistered type : mrc::pymrc::coro::BoostFibersMainPyAwaitable error (#1869) @dagardner-nv
• Revert PR_1736 "Always run the PR builder step even if others are cancelled" (#1860) @AnuradhaKaruppiah
• ensure columns are strings before concatenation (#1857) @cwharris
• Update Kafka DL script to 2.13-3.8.0 (#1856) @cwharris
• Update isort settings file path in fix_all.sh (#1855) @yczhang-nv
• Move isort settings into pyproject.toml (#1854) @dagardner-nv
• Update location of morpheus setup and data files in VS settings (#1843) @AnuradhaKaruppiah
• Fix isort config marking _utils as known first party (#1842) @dagardner-nv
• Fix usage of the C++ impl of write_df_to_file (#1840) @dagardner-nv
• Fix shutdown on Ctrl+C for Python source stages (#1839) @dagardner-nv
• Improved type-hints for stage and source decorators (#1831) @dagardner-nv
• Add tests to confirm that a mis-configured MultiPortModulesStage will raise an exception rather than segfaulting (#1829) @dagardner-nv
• Ensure proper initialization of CMAKE_INSTALL_PREFIX if needed (#1815) @dagardner-nv
• Fix merging of CLI args and Yaml configs in vdb_upload example (#1813) @dagardner-nv
• Fix log parsing undefined variable and duplicate sequence id errors (#1809) @dagardner-nv
• Remove obsolete version string from compose yamls (#1808) @dagardner-nv
• Ensure the release container does not contain any unintended files (#1807) @dagardner-nv
• Update ci/release/update-version.sh to include missed files (#1801) @dagardner-nv

📖 Documentation

• Remove references to pipeline-ae in docs (#2063) @dagardner-nv
• Document location of third party source repository (#2059) @dagardner-nv
• Update DFP class and file paths (#2052) @dagardner-nv
• Add known issue for dask shutdown (#2027) @cwharris
• Set the version in the conda packages docs (#2017) @AnuradhaKaruppiah
• Fix mis-leading deserialize stage comments (#2009) @dagardner-nv
• Update Morpheus docs to use cuda 12.5 (#2008) @yczhang-nv
• Fix minor issues with LLM example documentation (#1992) @dagardner-nv
• Incorporate review comments in the conda packages documentation (#1982) @AnuradhaKaruppiah
• Add CPU-only documentation (#1969) @dagardner-nv
• Document each of the Conda environment files (#1932) @dagardner-nv
• Update documentation to reflect CPU-only execution mode (#1924) @dagardner-nv
• Remove TODO statements from documentation (#1879) @dagardner-nv
• Remove automock for merlin as we no longer have/use merlin (#1830) @dagardner-nv
• Add documentation checks to CI (#1821) @dagardner-nv
• Fix documentation links to work in both source repo and documentation builds (#1814) @dagardner-nv
• Update documentation for vdb_upload to use realistic source data with the --file_source flag (#1800) @dagardner-nv

🚀 New Features

• Install morpheus-dfp conda package in the DFP container (#1971) @AnuradhaKaruppiah
• Morpheus docs update post compartmentalization (#1964) @AnuradhaKaruppiah
• Adding implementation of Router Nodes (#1963) @mdemoret-nv
• Include requirements files in the morpheus packages (#1957) @AnuradhaKaruppiah
• Unit tests for the namespace update script (#1954) @AnuradhaKaruppiah
• Script for updating the namespace due to compartmentalization changes (#1946) @AnuradhaKaruppiah
• Move tests/common to tests/morpheus/common (#1942) @AnuradhaKaruppiah
• Refactor Morpheus unit tests and plugin to the conda recipe for per-lib testing (#1933) @AnuradhaKaruppiah
• Remove debug log in HttpServerSourceStage when the queue is empty (#1921) @dagardner-nv
• Refactor digital_fingerprinting stages and add morpheus-split conda recipe (core, dfp, llm) (#1897) @AnuradhaKaruppiah
• Move vector db stages to morpheus-llm (#1889) @AnuradhaKaruppiah
• Scripts for building and uploading the morpheus-core conda package (#1883) @AnuradhaKaruppiah
• Implement MultiProcessingStage (#1878) @yczhang-nv
• Update to RAPIDS 24.10 (#1874) @cwharris
• Add support for a CPU-only Mode (#1851) @dagardner-nv
• [morpheus-refactor] Move morpheus source to python/morpheus (#1836) @AnuradhaKaruppiah
• Support for ControlMessage as an output type for HttpServerSourceStage and HttpClientSourceStage (#1834) @dagardner-nv
• Remove NVTabular (#1825) @cwharris
• Create a Docker image for Morpheus models (#1804) @dagardner-nv
• Add unique column to output of the log_parsing pipeline (#1795) @dagardner-nv

🛠️ Improvements

• Update to Triton Inference Server container version 24.09 (#2001) @dagardner-nv
• Remove temporary DFP todo list (#1998) @AnuradhaKaruppiah
• devcontainer: replace VAULT_HOST with AWS_ROLE_ARN (#1962) @jjacobelli
• Reduce the number of warnings emitted (#1947) @dagardner-nv
• Set lower CPU usage for test_shared_process_pool.py to avoid slowing down the test (#1935) @yczhang-nv
• Remove unused pymysql dependency from DFP mlflow container (#1930) @dagardner-nv
• Support LLM pipelines in CPU-only mode (#1906) @dagardner-nv
• Remove Starter Digital Fingerprinting (DFP) (#1903) @efajardo-nv
• Finalize removing MultiMessage from Morpheus (#1886) @yczhang-nv
• Run pre-commit on all files, not just python (#1880) @cwharris
• Prefer len(os.sched_getaffinity(0)) over os.cpu_count() (#1866) @cwharris
• Remove cloudtrail debug log from autoencoder source stage (#1865) @cwharris
• Run yapf, flake8, isort as part of pre-commit (#1859) @cwharris
• Warn when Config's pipeline_batch_size < model_max_batch_size (#1858) @cwharris
• Breakout morpheus_llm (#1853) @AnuradhaKaruppiah
• Install built documentation into release container (#1806) @dagardner-nv
• Removing support for MultiMessage from stages (#1803) @yczhang-nv
• Batch incoming DOCA raw packet data (#1731) @dagardner-nv