Basic question about the opendap plugin

[noaaroland]

Hi,

I'm just getting started with xpublish. I have created and published a data set successfully which is to say I can read the data set with xarray using xarray zarr. E.g.

url = 'http://cefi1.pmel.noaa.gov:8970/datasets/B10K-K20_CORECFS_Level1_collection/zarr'
fs = HTTPFileSystem()

# A thing
http_map = fs.get_mapper(url)

# open as a zarr group
zg = zarr.open_consolidated(http_map, mode="r")

# or open as another Xarray Dataset
ds = xr.open_zarr(http_map, consolidated=True)
ds

xarray.Dataset
Dimensions:
s_rho: 30 ocean_time: 775 eta_rho: 258 xi_rho: 182
blah blah blah

All good so far.

I have installed the opendap plugin so I expected to be able to also access the data set via opendap. However, I'm not entirely sure of the correct URL.

When I try http://cefi1.pmel.noaa.gov:8970/datasets/B10K-K20_CORECFS_Level1_collection/opendap.das in my browser, I get Internal server error in the browser, and this in the logs.

  File "/home/rhs/miniforge3/envs/xp/lib/python3.12/site-packages/xpublish_opendap/plugin.py", line 56, in get_dap_dataset
    dataset = dap_xarray.dap_dataset(ds, dataset_id)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rhs/miniforge3/envs/xp/lib/python3.12/site-packages/xpublish_opendap/dap_xarray.py", line 100, in dap_dataset
    data_grid = dap_grid(ds[var], dims)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rhs/miniforge3/envs/xp/lib/python3.12/site-packages/xpublish_opendap/dap_xarray.py", line 78, in dap_grid
    data=da.astype(da.encoding["dtype"]).data,
                   ~~~~~~~~~~~^^^^^^^^^
KeyError: 'dtype'

When I look at the loop which is doing the data_grid = dap_grid(ds[var], dims), it's in a loop that looks like:

    for var in ds.data_vars:
        data_grid = dap_grid(ds[var], dims)
        dataset.append(data_grid)

When in turn does the access to da.encoding["dtype"]

Of course, if I just open the data set and do the same thing it works fine:

for var in ds.data_vars:
    print(var)
    print(ds[var].encoding["dtype"])

Cs_r
float64
Jel
float32
hc
float64
salt
float32
temp
float32
zeta
float32

Which make me feel like maybe I just don't quite understand the right URL to use to tell xpublish to open the data set with opendap since I can do what the opendap plugin wants to do when I open the data set myself.

I'm sorry that the data set is behind our fire wall, but hopefully somebody can answer this basic question without access. If not, I'll try to figure out a way to give access.

[jhamman]

Without seeing how you are populating your dataset before serving it, this is just a guess, but I would think the opendap router should be robust to missing encoding. I would probably want to see something like:

if "dtype" in da.encoding:
    data = da.astype(da.encoding["dtype"]).data
else:
    data = da.data

(honestly, I'm not sure we should be type-casting this variable at all)

[noaaroland]

Thank you for the response. I'm not well versed in the details of how xpublish works, but I made a change along the lines you suggested.

    if 'dtype' in da.encoding:
        da_grid = da.astype(da.encoding["dtype"]).data
    else:
        da_grid = da.data
    data_grid = dap.Grid(
        name=da.name,
        data=da_grid,
        dtype=dap_dtype(da),
        dimensions=[dims[dim] for dim in da.dims],
    )

Now, the .das returns successfully in the browser.

When I open the dataset,

url = 'http://cefi1.pmel.noaa.gov:8970/datasets/B10K-K20_CORECFS_Level1_collection/opendap'
ds = xr.open_dataset(url, decode_cf=True, decode_times=True)
ds

There is an error from the DAP library:

syntax error, unexpected WORD_WORD, expecting ';' or ','

But, I can access a data point in the xarray just fine.

temp = ds.temp
temp[45,20,35,9].values

array(3.7376397, dtype=float32)

The times do not get translated from Doubles to datetime even though I asked. I'm not sure who's job it is (xarray or the plugin) to translate the times, but maybe that's why the cast is in place.

[abkfenris]

Sorry you hit a bug with OpenDAP. I think you are accessing things the right way, so the issue is probably on the libraries end. The OpenDAP plugin is largely something that I hacked together in an evening, and since it worked suprisingly well, I threw it out into the world so that it could encounter more data, and find things that broke it.

I think this is due to a change in more recent versions of Xarray in how encoding is handled. I believe that the encoding dict is reset for more variables when loading data in, or on various transformations, but I'm not finding anything in a quick search of what's new for recent releases. Are you using different versions of Xarray between the environment where Xpublish is running, and when you loaded it in locally?

We're probably not getting the dtype the most reliable way anyways, so I've added a new issue to address that xpublish-community/xpublish-opendap#45

Is there any chance that you could carve off a small chunk of your dataset that you could share and we could test against? It would also be helpful to get a pip/conda/mamba list of your environment(s) so we could compare Xarray and other versions.

[noaaroland]

No worries. Happy to help to make improvements and debug.

I am using these on the server:

(xp) [rhs ~]$ mamba list | grep xarray
cf_xarray                 0.8.6              pyhd8ed1ab_0    conda-forge
xarray                    2023.10.1          pyhd8ed1ab_0    conda-forge

I am using these on the client:

(cf) schweitze:~$ mamba list | grep xarray
cf_xarray                 0.8.6              pyhd8ed1ab_0    conda-forge
xarray                    2023.10.1          pyhd8ed1ab_0    conda-forge

I will get to work on a small sample data set.

The full lists are attached.
client.txt
server.txt

[noaaroland]

Again thanks for looking at this. Here is a small self-contained example which illustrates all the steps I took to serve the data via xpublish and the xpublish-opendap.

https://github.com/noaaroland/xpublish_example/tree/main.

Edit: syntax

[abkfenris]

Hi @noaaroland , while I haven't gotten time to dig in to your example, @xaviernogueira made a PR that I think will address the underlying issue, and I just released v0.1.2 for xpublish-opendap with those fixes (PyPI done, Conda-Forge in progress).

Any chance you could test and see if that did the trick?

[noaaroland]

I'm having a look at it now.... Thanks for the update.

[noaaroland]

I put in a pull request to fix some issues I had with attributes in the two examples I've tried.

It's appropriate to close this, however, there is a more fundamental problem with the OPeNDAP objects that get built for one of my test data sets which became apparent once I got past the issue with the attributes. This concerns how the OPeNDAP objects for a data variable with 2D coordinates gets constructed. I will start a new conversation for that issue.