-
Notifications
You must be signed in to change notification settings - Fork 575
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Docs] Treat delegated execution separately #5184
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -594,8 +594,8 @@ operator can be invoked like so: | |
dataset = foz.load_zoo_dataset("quickstart") | ||
compute_metadata = foo.get_operator("@voxel51/utils/compute_metadata") | ||
|
||
# Schedule a delegated operation to (re)compute metadata | ||
compute_metadata(dataset, overwrite=True, delegate=True) | ||
# (Re)compute the dataset's metadata | ||
compute_metadata(dataset, overwrite=True) | ||
|
||
.. note:: | ||
|
||
|
@@ -614,13 +614,11 @@ follows: | |
sample_collection, | ||
overwrite=False, | ||
num_workers=None, | ||
delegate=False, | ||
): | ||
ctx = dict(view=sample_collection.view()) | ||
params = dict( | ||
overwrite=overwrite, | ||
num_workers=num_workers, | ||
delegate=delegate, | ||
) | ||
return foo.execute_operator(self.uri, ctx, params=params) | ||
|
||
|
@@ -655,71 +653,63 @@ data, you can access it via the ``result`` property of the returned | |
result = await op(...) | ||
print(result.result) # {...} | ||
|
||
.. _delegating-function-calls: | ||
|
||
Delegating function calls | ||
------------------------- | ||
Requesting delegation | ||
~~~~~~~~~~~~~~~~~~~~~ | ||
|
||
The | ||
`@voxel51/utils/delegate <https://github.com/voxel51/fiftyone-plugins/tree/main/plugins/utils>`_ | ||
operator provides a general purpose utility for | ||
:ref:`delegating execution <delegated-operations>` of an arbitrary function | ||
call that can be expressed in any of the following forms: | ||
Operators that support :ref:`delegated execution <delegated-operations>` can | ||
support this via the `__call__()` syntax by passing the | ||
`request_delegation=True` flag to | ||
:func:`execute_operator() <fiftyone.operators.execute_operator>`. | ||
|
||
- Execute an arbitrary function: `fcn(*args, **kwargs)` | ||
- Apply a function to a dataset or view: | ||
`fcn(dataset_or_view, *args, **kwargs)` | ||
- Call an instance method of a dataset or view: | ||
`dataset_or_view.fcn(*args, **kwargs)` | ||
|
||
Here's some examples of delegating common tasks that can be expressed in the | ||
above forms: | ||
In fact, the | ||
`@voxel51/utils/compute_metadata <https://github.com/voxel51/fiftyone-plugins/tree/main/plugins/utils>`_ | ||
operator does just that: | ||
|
||
.. code-block:: python | ||
:linenos: | ||
|
||
import fiftyone as fo | ||
import fiftyone.operators as foo | ||
import fiftyone.zoo as foz | ||
class ComputeMetadata(foo.Operator): | ||
return foo.OperatorConfig( | ||
... | ||
allow_immediate_execution=True, | ||
allow_delegated_execution=True, | ||
) | ||
|
||
dataset = foz.load_zoo_dataset("quickstart") | ||
delegate = foo.get_operator("@voxel51/utils/delegate") | ||
def __call__( | ||
self, | ||
sample_collection, | ||
overwrite=False, | ||
num_workers=None, | ||
delegate=False, | ||
): | ||
ctx = dict(view=sample_collection.view()) | ||
params = dict( | ||
overwrite=overwrite, | ||
num_workers=num_workers, | ||
) | ||
return foo.execute_operator( | ||
self.uri, | ||
ctx, | ||
params=params, | ||
request_delegation=delegate, | ||
) | ||
|
||
# Compute metadata | ||
delegate("compute_metadata", dataset=dataset) | ||
which means that it can be invoked like so: | ||
|
||
# Compute visualization | ||
delegate( | ||
"fiftyone.brain.compute_visualization", | ||
dataset=dataset, | ||
brain_key="img_viz", | ||
) | ||
.. code-block:: python | ||
:linenos: | ||
|
||
# Export a view | ||
delegate( | ||
"export", | ||
view=dataset.to_patches("ground_truth"), | ||
export_dir="/tmp/patches", | ||
dataset_type="fiftyone.types.ImageClassificationDirectoryTree", | ||
label_field="ground_truth", | ||
) | ||
compute_metadata = foo.get_operator("@voxel51/utils/compute_metadata") | ||
|
||
# Load the exported patches into a new dataset | ||
delegate( | ||
"fiftyone.Dataset.from_dir", | ||
dataset_dir="/tmp/patches", | ||
dataset_type="fiftyone.types.ImageClassificationDirectoryTree", | ||
label_field="ground_truth", | ||
name="patches", | ||
persistent=True, | ||
) | ||
# Schedule a delegated operation to (re)compute metadata | ||
compute_metadata(dataset, overwrite=True, delegate=True) | ||
|
||
.. _direct-operator-execution: | ||
|
||
Direct execution | ||
---------------- | ||
|
||
You can also programmatically execute any operator by directly calling | ||
You can programmatically execute any operator by directly calling | ||
:func:`execute_operator() <fiftyone.operators.execute_operator>`: | ||
|
||
.. code-block:: python | ||
|
@@ -738,25 +728,11 @@ You can also programmatically execute any operator by directly calling | |
dataset_type="COCO", | ||
labels_path=dict(absolute_path="/tmp/coco/labels.json"), | ||
label_field="ground_truth", | ||
delegate=False, # False: execute immediately, True: delegate | ||
) | ||
} | ||
|
||
foo.execute_operator("@voxel51/io/export_samples", ctx) | ||
|
||
In the above example, the `delegate=True/False` parameter controls whether | ||
execution happens immediately or is | ||
:ref:`delegated <operator-delegated-execution>` because the operator implements | ||
its | ||
:meth:`resolve_delegation() <fiftyone.operators.operator.Operator.resolve_delegation>` | ||
as follows: | ||
|
||
.. code-block:: python | ||
:linenos: | ||
|
||
def resolve_delegation(self, ctx): | ||
return ctx.params.get("delegate", False) | ||
|
||
.. note:: | ||
|
||
In general, to use | ||
|
@@ -787,13 +763,11 @@ data, you can access it via the ``result`` property of the returned | |
result = await foo.execute_operator("@an-operator/with-results", ctx) | ||
print(result.result) # {...} | ||
|
||
.. _requesting-operator-delegation: | ||
|
||
Requesting delegation | ||
--------------------- | ||
~~~~~~~~~~~~~~~~~~~~~ | ||
|
||
If an operation supports both immediate and delegated execution as specified | ||
either by its :ref:`configuration <operator-delegation-configuration>` or | ||
If an operation supports both immediate and | ||
:ref:`delegated execution <delegated-operations>` as specified by its | ||
:ref:`execution options <operator-execution-options>`, you can request | ||
delegated execution by passing the `request_delegation=True` flag to | ||
:func:`execute_operator() <fiftyone.operators.execute_operator>`: | ||
|
@@ -823,6 +797,65 @@ operator's input modal when executing it from within the App: | |
delegation_target="overnight", | ||
) | ||
|
||
.. _delegating-function-calls: | ||
|
||
Delegating function calls | ||
------------------------- | ||
|
||
The | ||
`@voxel51/utils/delegate <https://github.com/voxel51/fiftyone-plugins/tree/main/plugins/utils>`_ | ||
operator provides a general purpose utility for | ||
:ref:`delegating execution <delegated-operations>` of an arbitrary function | ||
call that can be expressed in any of the following forms: | ||
|
||
- Execute an arbitrary function: `fcn(*args, **kwargs)` | ||
- Apply a function to a dataset or view: | ||
`fcn(dataset_or_view, *args, **kwargs)` | ||
- Call an instance method of a dataset or view: | ||
`dataset_or_view.fcn(*args, **kwargs)` | ||
|
||
Here's some examples of delegating common tasks that can be expressed in the | ||
above forms: | ||
|
||
.. code-block:: python | ||
:linenos: | ||
|
||
import fiftyone as fo | ||
import fiftyone.operators as foo | ||
import fiftyone.zoo as foz | ||
|
||
dataset = foz.load_zoo_dataset("quickstart") | ||
delegate = foo.get_operator("@voxel51/utils/delegate") | ||
|
||
# Compute metadata | ||
delegate("compute_metadata", dataset=dataset) | ||
|
||
# Compute visualization | ||
delegate( | ||
"fiftyone.brain.compute_visualization", | ||
dataset=dataset, | ||
brain_key="img_viz", | ||
) | ||
|
||
# Export a view | ||
delegate( | ||
"export", | ||
view=dataset.to_patches("ground_truth"), | ||
export_dir="/tmp/patches", | ||
dataset_type="fiftyone.types.ImageClassificationDirectoryTree", | ||
label_field="ground_truth", | ||
) | ||
|
||
# Load the exported patches into a new dataset | ||
delegate( | ||
"fiftyone.Dataset.from_dir", | ||
dataset_dir="/tmp/patches", | ||
dataset_type="fiftyone.types.ImageClassificationDirectoryTree", | ||
label_field="ground_truth", | ||
name="patches", | ||
persistent=True, | ||
) | ||
|
||
.. _delegated-operations: | ||
|
||
Delegated operations | ||
|
@@ -893,7 +926,7 @@ delegated operations and execute them serially in its process. | |
|
||
You must also ensure that the | ||
:ref:`allow_legacy_orchestrators <configuring-fiftyone>` config flag is set | ||
in the environment where you run the App, e.g. by setting: | ||
in the environment where you run the App/SDK, e.g. by setting: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🛠️ Refactor suggestion Consider highlighting this requirement more prominently. This environment configuration is crucial for using delegated operations. Consider moving this information to the beginning of the delegated operations section or adding it to a "Prerequisites" subsection. |
||
|
||
.. code-block:: shell | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we need a kitchen sink example of
__call__
,request_delegation
,allow_delegated_execution
, adding adelegate
property, and determining if an operator was scheduled (which we know is missing). In other words, an example operator that illustrates everything that you should do when implementingallow_delegated_execution
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I fixed this here: it is available via
ctx.delegated
This is now effectively deprecated. Plugin devs should always use
execution_options()
as pictured here, since the plugin framework can be trusted as authoritative on which execution modes the current deployment supportsThat's what I tried to do here. There are multiple examples of increasing complexity here, including a last one that links over to here for additional context
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
voxel51/fiftyone-plugins#182