Running long tasks on MMODA

[dsavchenko]

Following questions of @okolo

• is there any way to monitor/report progress of long tasks

• is there a standard way to view debug info (e.g. console output) for developers like me

Perhaps one can use ODATextProduct to save logs? Right?

[dsavchenko]

Concerning logs, there is no standard way to return the stdout/stderr of the nb2service
For subprocesses, one can save the ouptut and return as TextProduct. The downside is that it will only be available upon successfull completion of the job.

[dsavchenko]

We don't have any progress monitoring for the notebook-based workflows (as it is for integral).
There is a related #25
But this wouldn't help much in your case, as far as I understand, there is one cell that takes most of the time. And this would often be the case. So we need to think about some machinery to do it.

[volodymyrss]

Following questions of @okolo

• is there any way to monitor/report progress of long tasks

You mean incremental progress, like by cell? That we plan and could add now.

We could also let oda api send custom callback progress messages, if useful.

• is there a standard way to view debug info (e.g. console output) for developers like me

We return the entire notebook for debug, it should contain stdout, does it not?

You can also fetch the notebook before or is completed, with the same endpoint, it will return partial output but still valid.

We could add URL to fetch this notebook to dispatcher response if useful.

Perhaps one can use ODATextProduct to save logs? Right?

[volodymyrss]

We don't have any progress monitoring for the notebook-based workflows (as it is for integral).
There is a related #25
But this wouldn't help much in your case, as far as I understand, there is one cell that takes most of the time. And this would often be the case. So we need to think about some machinery to do it.

Here we need to think about how you want to express these updates.
If you just mean stdout-err lines, there is an endpoint on the backend, it is already used by the dispatcher, and it can be made available to the user with a simple dispatcher proxy. Would it be useful?

[okolo]

You mean incremental progress, like by cell? That we plan and could add now.

This could be useful but unfortunately doesn't cover the case of running an external program which takes long time

We could also let oda api send custom callback progress messages, if useful.

Custom callback messages would perhaps solve the problem. If one uses an external program like CRbeam one could implement parsing of the program output and sending callback progress messages.

• is there a standard way to view debug info (e.g. console output) for developers like me

We return the entire notebook for debug, it should contain stdout, does it not?

You can also fetch the notebook before or is completed, with the same endpoint, it will return partial output but still valid.

We could add URL to fetch this notebook to dispatcher response if useful.

Thank you! I think this is perfect solution for debugging.

[volodymyrss]

You can also fetch the notebook before or is completed, with the same endpoint, it will return partial output but still valid.
We could add URL to fetch this notebook to dispatcher response if useful.

Thank you! I think this is perfect solution for debugging.

oda-hub/dispatcher-app#594

[volodymyrss]

Discussed with @okolo that one of the other options, sending custom updates, would be also useful like so