Computational output: moving from save-to-disk to DuckDB

[SamuelBrand1]

At the moment, the analysis workflow relies on saving to disk for both serialising results and checkpointing (via DrWatson.produce_or_load). Similarly to discussion #221 this is fine for low/moderate computational workloads but isn't obviously scalable and relies on local file structure.

A (IMO) better alternative is open a connection to a DuckDB instance using the Julia front-end to stream results at; this also makes it easier to run the post-processing as results arrive.

[seabbs]

Does DuckDB not have an interface to DataFrames.jl or similar to avoid needing to use the SQL syntax?

[SamuelBrand1]

Does DuckDB not have an interface to DataFrames.jl or similar to avoid needing to use the SQL syntax?

It looks like the DataFrames interface is via an API to Appender and you can add row by row; not sure about load back from the DB.

[seabbs]

Status of this idea?