You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In v1 we had the idea of a "dataset", a curated set of files that get ingested into the system in order to have something available for the user to play with. Ingesting the dataset should:
be quick
easy
lead to the same state of the system for every import (given the same code/commit/version is being used)
In v1, this was orchestrated through Helm charts, and the process wasn't pretty. In v2, we don't really have that, although we could use the same Helm chart approach.
However, I think we could do better. What about having a .zip file structure (yes, I come from the world or .jar, .war, .ear, …) which includes all the files, and which can be uploaded via the Rest API with our generic "upload anything" endpoint? That should also make it possible to upload this via the UI.
In the background, the "loader" demuxes this archive into the different actual loaders.
Given the system was empty before, it should lead to the same state once the upload is complete.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
In v1 we had the idea of a "dataset", a curated set of files that get ingested into the system in order to have something available for the user to play with. Ingesting the dataset should:
In v1, this was orchestrated through Helm charts, and the process wasn't pretty. In v2, we don't really have that, although we could use the same Helm chart approach.
However, I think we could do better. What about having a
.zip
file structure (yes, I come from the world or.jar
,.war
,.ear
, …) which includes all the files, and which can be uploaded via the Rest API with our generic "upload anything" endpoint? That should also make it possible to upload this via the UI.In the background, the "loader" demuxes this archive into the different actual loaders.
Given the system was empty before, it should lead to the same state once the upload is complete.
Beta Was this translation helpful? Give feedback.
All reactions