Version replaying

[asmeurer]

We need a way to "replay" versions, i.e., to reconstruct the raw data for a dataset and rebuild the data in each version. This will enable several features

• Changing certain metadata about a dataset, which currently cannot be changed across versions, including ndim, chunks, and dtype Support for changing chunk size #110
• Support for deleting a version. Currently we can delete the version group, but it would not remove the data for that version from the raw data. Deleting versions and garbage collecting #143
• It opens the possibility of doing some tricks like inserting rows without actually rewriting all the data, by leaving some data in the raw data empty insert method #47. The issue with this is that you need a way to garbage collect the empty data after a while, or else it becomes more inefficient than just doing it the naive way.

[asmeurer]

For changing metadata, there are two points.

1. If we want to only change the metadata for new versions, but keep old versions as they are, I think this issue isn't what we need. Rather, we need some way to point the new dataset to a different raw dataset. Currently the raw dataset is referenced by name, but I think this wouldn't be hard to change, as we already are storing the path to the raw dataset in the metadata ever since Continuing work on sparse datasets #144

2. If we want to change it for all old versions, this is what is needed. This is a potentially destructive action, so it may be worth having some checks that the new metadata isn't lossy against the old without some sort of force flag (e.g., the new ndim should be larger, the new dtype should cast from the old losslessly, etc.).

[ArvidJB]

Sounds good. Let's make this as simple as possible. It would be great if we could have some API which takes a callback and then calls it for each version to write a new HDF5 file:

• Input would be the InMemoryGroup for that version (including attrs)
• Output would be None (if we want to skip this version) or a new InMemoryGroup with added/removed/modified datasets/attrs

Do you think this would work?

[asmeurer]

I hadn't considered doing everything in a new file. That would take care of the atomicity concerns, though it also means we would need to copy everything.

[asmeurer]

Version replaying is now implemented in versioned_hdf5/replay.py, and the modify_metadata and delete_version functions in particular. The insertion idea has not yet been implemented, but we can use #47 to track that, so I am closing this issue.