[FEA]: Pipeline error alerting and progress monitoring

[efajardo-nv]

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

High

Please provide a clear description of problem this feature solves

We have a long running Morpheus DFP pipeline which we run in a SSH session to our DGX host. We currently use screen so that we can disconnect from the host and later reattach to the screen to monitor progress and check for errors/failures. We are looking for a more practical, real-time way of being alerted of errors and multi-user access to progress monitoring.

Describe your ideal solution

An example of how to use existing functionality to implement this. If this functionality does not already exist, request necessary hooks/stages and example/documentation of how to set it up in our environment.

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request

[mdemoret-nv]

As discussed with @efajardo-nv, the best option here will be to create a logging handler which can tie into the existing morpheus logger. For example, if you wanted to report errors and warnings to Graphana, you could use this logger: https://pypi.org/project/python-logging-loki/.

import logging
import logging_loki

# Configure the morpheus logger first
configure_logging()

handler = logging_loki.LokiHandler(
    url="https://my-loki-instance/loki/api/v1/push", 
    tags={"application": "my-app"},
    auth=("username", "password"),
    version="1",
)

# Get the morpheus root logger
logger = logging.getLogger("morpheus")

# Add the loki handler to the morpheus logger
logger.addHandler(handler)

# At any point in the morpheus application, you can call something like the following which will be reported to Graphana
logger.error(
    "Something happened", 
    extra={"tags": {"service": "my-service"}},
)