Live dataset updates

[alexander0042]

Hi,

I'm the developer behind Pirate Weather, and lately I've been running into some issues trying to serve updating Zarr files via FastAPI, and was wondering how XPublish managed it? My current approach saves the arrays as .zips and the copies them over to updating LMDB databases; however, the database seems to grow unstable pretty quickly, the chunk layouts being identical. Does xpublish support updating datasets online, or would I need to restart the server and point it at a new file?

Thank you in advance,
Alexander

[mpiannucci]

Hi Alexander, great to hear from you, I've read through pirate weathers blog posts and repo and you are doing cool stuff.

In the xpublish world we use dataset provider plugins to accomplish this. This allows us to customize all our dataset loading so you have full control of when to reload a dataset or when to use a cached version.