diff --git a/README.md b/README.md new file mode 100644 index 0000000..5d86e89 --- /dev/null +++ b/README.md @@ -0,0 +1,78 @@ +# Xpublish + +Publish Xarray Datasets to the web + + + +[![PyPI](https://img.shields.io/pypi/v/xpublish)](https://pypi.org/project/xpublish/) +[![Conda](https://img.shields.io/conda/v/conda-forge/xpublish)](https://anaconda.org/conda-forge/xpublish) +![PyPI - Python Version](https://img.shields.io/pypi/pyversions/xpublish) +[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/xpublish-community/xpublish/main) + +[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/xpublish-community/xpublish/main.yaml?logo=github)](https://github.com/xpublish-community/xpublish/actions/workflows/main.yaml) +[![Documentation Status](https://readthedocs.org/projects/xpublish/badge/?version=latest)](https://xpublish.readthedocs.io/en/latest/?badge=latest) +[![](https://codecov.io/gh/xpublish-community/xpublish/branch/main/graph/badge.svg)](https://codecov.io/gh/xpublish-community/xpublish) +[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/xpublish-community/xpublish/main.svg)](https://results.pre-commit.ci/latest/github/xpublish-community/xpublish/main) + + + +## A quick example + +**Serverside: Publish a Xarray Dataset through a rest API** + + + +```python +ds.rest.serve(host="0.0.0.0", port=9000) +``` + + + +**Client-side: Connect to a published dataset** + + + +The published datasets can be accessed from various kinds of client applications, e.g., from within Python using Zarr and fsspec. + +```python +import xarray as xr +import zarr +from fsspec.implementations.http import HTTPFileSystem + +fs = HTTPFileSystem() +http_map = fs.get_mapper("http://0.0.0.0:9000/zarr/") + +# open as a zarr group +zg = zarr.open_consolidated(http_map, mode="r") + +# or open as another Xarray Dataset +ds = xr.open_zarr(http_map, consolidated=True) +``` + +Or to explore other access methods, open [http://0.0.0.0:9000/docs](http://0.0.0.0:9000/docs) in a browser. + + + +## Why? + +Xpublish lets you serve/share/publish Xarray Datasets via a web application. + +The data and/or metadata in the Xarray Datasets can be exposed in various forms through [pluggable REST API endpoints](https://xpublish.readthedocs.io/en/latest/user-guide/plugins.html). +Efficient, on-demand delivery of large datasets may be enabled with Dask on the server-side. + +Xpublish's [plugin ecosystem](https://xpublish.readthedocs.io/en/latest/ecosystem/index.html#plugins) has capabilities including: + +- publish on-demand or derived data products +- turning xarray objects into streaming services (e.g. OPeNDAP) + +## How? + +Under the hood, Xpublish is using a web app (FastAPI) that is exposing a +REST-like API with builtin and/or user-defined endpoints. + +For example, Xpublish provides by default a minimal Zarr compatible REST-like API with the following endpoints: + +- `zarr/.zmetadata`: returns Zarr-formatted metadata keys as json strings. +- `zarr/var/0.0.0`: returns a variable data chunk as a binary string. + +Futher endpoints can be added by installing or writing plugins. diff --git a/README.rst b/README.rst deleted file mode 100644 index 7ad5fea..0000000 --- a/README.rst +++ /dev/null @@ -1,77 +0,0 @@ -Xpublish -======== - -Publish Xarray Datasets via a REST API. - -.. image:: https://github.com/xpublish-community/xpublish/actions/workflows/main.yaml/badge.svg - :target: https://github.com/xpublish-community/xpublish/actions/workflows/main.yaml - :alt: GitHub Workflow Status - -.. image:: https://readthedocs.org/projects/xpublish/badge/?version=latest - :target: https://xpublish.readthedocs.io/en/latest/?badge=latest - :alt: Documentation Status - -.. image:: https://mybinder.org/badge_logo.svg - :target: https://mybinder.org/v2/gh/xarray-contrib/xpublish/main - :alt: Binder - -.. image:: https://codecov.io/gh/xarray-contrib/xpublish/branch/main/graph/badge.svg - :target: https://codecov.io/gh/xarray-contrib/xpublish - -.. image:: https://results.pre-commit.ci/badge/github/xpublish-community/xpublish/main.svg - :target: https://results.pre-commit.ci/latest/github/xpublish-community/xpublish/main - :alt: pre-commit.ci status - -**Serverside: Publish a Xarray Dataset through a rest API** - -.. code-block:: python - - ds.rest.serve(host="0.0.0.0", port=9000) - - -**Client-side: Connect to a published dataset** - -The published dataset can be accessed from various kinds of client applications. -Here is an example of directly accessing the data from within Python: - -.. code-block:: python - - import xarray as xr - import zarr - from fsspec.implementations.http import HTTPFileSystem - - fs = HTTPFileSystem() - http_map = fs.get_mapper("http://0.0.0.0:9000") - - # open as a zarr group - zg = zarr.open_consolidated(http_map, mode="r") - - # or open as another Xarray Dataset - ds = xr.open_zarr(http_map, consolidated=True) - - -Why? -^^^^ - -Xpublish lets you serve/share/publish Xarray Datasets via a web application. - -The data and/or metadata in the Xarray Datasets can be exposed in various forms -through pluggable REST API endpoints. Efficient, on-demand delivery of large -datasets may be enabled with Dask on the server-side. - -We are exploring applications of Xpublish that include: - -* publish on-demand or derived data products -* turning xarray objects into streaming services (e.g. OPeNDAP) - -How? -^^^^ - -Under the hood, Xpublish is using a web app (FastAPI) that is exposing a -REST-like API with builtin and/or user-defined endpoints. - -For example, Xpublish provides by default a minimal Zarr compatible REST-like -API with the following endpoints: - -* ``.zmetadata``: returns Zarr-formatted metadata keys as json strings. -* ``var/0.0.0``: returns a variable data chunk as a binary string. diff --git a/docs/source/getting-started/tutorial/using-plugins.md b/docs/source/getting-started/tutorial/using-plugins.md index 0b5ace3..067fb34 100644 --- a/docs/source/getting-started/tutorial/using-plugins.md +++ b/docs/source/getting-started/tutorial/using-plugins.md @@ -1,5 +1,9 @@ # Using Plugins +```{versionadded} 0.3.0 +Plugins were released in [0.3.0](https://github.com/xpublish-community/xpublish/releases/tag/0.3.0). +``` + Much of the power of Xpublish comes from it's [ecosystem](../../ecosystem/index) of plugins, which can quickly extend Xpublish with new capabilities. ```{note} diff --git a/docs/source/index.md b/docs/source/index.md index 14e1e3d..e02c3f0 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -2,6 +2,13 @@ **Useful links:** [Installation](getting-started/installation) | [Source Repository](https://github.com/xpublish-community/xpublish/) | [Issue Tracker](https://github.com/xpublish-community/xpublish/issues) | [Q&A Support](https://github.com/xpublish-community/xpublish/discussions/categories/q-a?discussions_q=category%3AQ%26A+) | [Slack Channel](./ecosystem/index.md#slack) +```{include} ../../README.md +--- +start-after: +end-before: +--- +``` + ## Xpublish is ````{grid} 3 @@ -73,37 +80,27 @@ The contributing guidelines will guide you through the process of improving Xpub ```` -### Xpublish lets you easily publish Xarray Datasets via a REST API. +## A quick intro _You can run a short example application in a live session here:_ [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/xpublish-community/xpublish/master) On the server-side, one or more datasets can be published using the {class}`xpublish.Rest` class or the {attr}`xarray.Dataset.rest` accessor, e.g., -```python -ds.rest.serve(host="0.0.0.0", port=9000) +```{include} ../../README.md +--- +start-after: +end-before: +--- ``` -Those datasets can be accessed from various kinds of client applications, e.g., -from within Python using Zarr and fsspec. - -```python -import xarray as xr -import zarr -from fsspec.implementations.http import HTTPFileSystem - -fs = HTTPFileSystem() -http_map = fs.get_mapper("http://0.0.0.0:9000/zarr/") - -# open as a zarr group -zg = zarr.open_consolidated(http_map, mode="r") - -# or open as another Xarray Dataset -ds = xr.open_zarr(http_map, consolidated=True) +```{include} ../../README.md +--- +start-after: +end-before: +--- ``` -Or to explore other access methods, open [http://0.0.0.0:9000/docs](http://0.0.0.0:9000/docs) in a browser. - ```{toctree} --- caption: Documentation Contents diff --git a/docs/source/user-guide/plugins.md b/docs/source/user-guide/plugins.md index 14fde78..a2afcf4 100644 --- a/docs/source/user-guide/plugins.md +++ b/docs/source/user-guide/plugins.md @@ -1,5 +1,9 @@ # Plugins +```{versionadded} 0.3.0 +Plugins were released in [0.3.0](https://github.com/xpublish-community/xpublish/releases/tag/0.3.0). +``` + While {py:class}`fastapi.APIRouter` can get you started building new endpoints for datasets quickly, the real extendability of Xpublish comes from it's plugin system.