structuring an Xpublish application

[mhaberler]

I want to serve partial weather forecast data from one or a few forecasting models, and want to keep a history for a few weeks, maybe months. The purpose is to serve subset of forecast parameters for trajectory forecast and hindcast, which is used for stuff like this or this.

The forecast happens in runs, from maybe 2 to 8 runs per day and so have a timestamp like 2020121712 run 00, 03 etc. Clients use data from a single run.

Serving a single run works fine for me but as of now is a singleton. I run uvicorn behind an nginx reverse proxy.

What I have: https://example.com/datasets/u/points/47.0/14.7/47.05/14.8/1/4 (eg the northern wind component, samples from a bounding box and selected height levels)

What I need is something like https://example.com/icon-d2/2020121712/00/u/points/47.0/14.7/47.05/14.8/1/4

How should I go about it?

One idea would be to subclass xpublish.Rest and override get_dataset_collection_keys(). Making sense?

thanks for any suggestions,

Michael

edit: reading up on the FastAPI docs this seems a viable solution for the setup above:

• /datasets/models -> return a list of models, like 'icon-d2', 'gfs' etc
• /datasets/models/{model}/runs -> returns a list of runs archived for this model
• /dataset/{model}/{run}/{dataset_id} -> serve dataset(model, run)

I'd prefer to do this without modifying Xpublish. Feasible?

[benbovy]

Would merging all models and runs into a single dataset (with model and run extra dimensions) before serving it with Xpublish be a viable option? You could then create a FastAPI dependency for all your API endpoints in order to get model and run from the merged dataset using Dataset.sel.

I wouldn't recommend subclassing xpublish.Rest, as it may not work with future versions of Xpublish. However, it would probably be useful to provide some mechanism built in Xpublish to allow more flexible dataset collection "keys" beyond single strings. Rest would accept a dictionary of dataset objects where keys can be any hashable object (e.g., a tuple). Users would then need to provide:

• a custom FastAPI dependency function that is used to get the dataset object from the collection being served. This function may have an arbitrary signature with multiple arguments, e.g., (model: str, run: str).
• a custom route prefix that is applied to all API endpoints, parametrized like the custom dependency function.

Not sure yet how to expose this feature to Xpublish users. Another tricky thing is to properly update OpenAPI specs according to the custom dependency function and route prefix, as FastAPI currently doesn't seem to update the specs when overriding a dependency with a different signature (see fastapi/fastapi#1594).