Add disk caching

[benjeffery]

WIP. Currently the test tree sequence has no UUID so trying to find a way to get one.

[jeromekelleher]

LGTM - does it work?

We'll want to integrate with the CLI, have you thought about how to configure the cache dir? (I was think appdirs might be a good approach, ultimately?)

[jeromekelleher]

see comment about cache version

[jeromekelleher]

Easiest way to get a uuid is to save to tempfile and read back in

[benjeffery]

I've modified this to use appdirs and to cache based on the hash of files of functions in the calling tree of the cached function. This means that changes to pages code won't trigger an invalidation, which will help a lot with dev.

[jeromekelleher]

I think we should merge this so we have it, but it's probably overkill in a few different ways.

1. diskcache is really quite complex. After spending 10 mins with the docs, it looks like it's using relational DB plus a bunch of stuff around timing out keys. We really don't need this, and it also means we'd have to be much more careful about managing opening and closing the cache than we are here. We could just store the files (as CSVs, even) based on their keys and it would be much simpler and easier to maintain.

2. The key generation policy is clever, but expensive to run and hard to test. Just incrementing a number when we change the dataframe format isn't that hard, and it'll make key generation fast.

[jeromekelleher]

This could also happen, e.g. if there wasn't sufficient permissions, or if it already existed and was a file, right? So, the log error will be misleading.

Why not use the pathlib API directly?

cache_dir.mkdir(exist_ok=True, parents=True)

I think this takes care of all the required corner cases and raises a sensible error on weirdnesses.

[benjeffery]

Nice, much simpler. Fixed.

[jeromekelleher]

right, so there's two layers of caching here. I had to think a bit about what this means - so the first time the property is called, we fall through to the disk_cache version, and then afterwards (within that process) it'll be cached by functools.

[benjeffery]

1. diskcache is really quite complex. After spending 10 mins with the docs, it looks like it's using relational DB plus a bunch of stuff around timing out keys. We really don't need this, and it also means we'd have to be much more careful about managing opening and closing the cache than we are here. We could just store the files (as CSVs, even) based on their keys and it would be much simpler and easier to maintain.

From the diskcache docs: "Cache objects are thread-safe and may be shared between threads. Two Cache objects may also reference the same directory from separate threads or processes. In this way, they are also process-safe and support cross-process communication." So I don't think we need to worry about opening and closing

2. The key generation policy is clever, but expensive to run and hard to test. Just incrementing a number when we change the dataframe format isn't that hard, and it'll make key generation fast.

Yeah, fair point. I replaced it with a version string.

[jeromekelleher]

Did you push the changes here @benjeffery?

[benjeffery]

Whoops, should be there now!

[jeromekelleher]

From the diskcache docs: "Cache objects are thread-safe and may be shared between threads. Two Cache objects may also reference the same directory from separate threads or processes. In this way, they are also process-safe and support cross-process communication." So I don't think we need to worry about opening and closing

In principle - in practise, sqlite hasn't worked well in my experience without carefully handling the context managers. Let's try it out and see anyway.