Upload data

[mjaquiery]

It would be great if we had functionality to upload data via the frontend. Ideally, this could enable sharing data between collaborators or individual users without needing harvesters running locally. I think this would be most beneficial when collaborators wanted to share few data files, otherwise it would make more sense to have a harvester running locally. This could also enable sharing of simulated data in addition to experimental data.

Original issue by @BradyPlanden Battery-Intelligence-Lab/galv#66 (comment)

[mjaquiery]

This is going to be very challenging to implement with our current architecture. The way we have things set up, Harvesters upload files in a two-stage process that involves editing the data so that they can map columns appropriately for meta-analysis. This makes it difficult to replicate the Harvester's behaviour on the frontend or backend without duplicating the Harvester code.

Some approaches we might try for implementing this functionality:

Let the Harvester code do the work

We could allow users to upload files to temporary directories on the backend server, and run a Harvester::harvest process on those files. This would require spoofing quite a lot of stuff on both Harvester and Backend side, though, and would likely break frequently when trying to update either service.

Let the Frontend do the work

We could hold uploaded files in memory and require the user to manipulate them within the Frontend until they are satisfactory data files for backend storage. We could then allow them to be uploaded and adjust the Backend code to allow uploaded files to come from non-Harvester sources (or create a dummy Harvester that represents user-uploaded files).

This requires a lot of the Harvester code to be reimplemented in React/TypeScript.

[mjaquiery]

Should be solved by #24

The harvester is integrated as a dependency for the backend and access control is managed directly by Files instead of inheriting from Monitored Paths.

For files with a known mapping, they can be uploaded in one go. For files with no known mapping, they must be uploaded twice. Once to allow a mapping to be defined and applied, and once to ingest the actual data. This avoids having to store data on the server temporarily and therefore having to manage cleanup, etc.