Composite Stores

[thorwhalen]

There's a pattern I've seen pop up regularly, that I'll call "Composite Stores" (not my name) or "Store Meshes" (my name -- but prefer the former).

This discussion is meant to discuss this patterns (what do we mean by it?), as well as the relevant tools, applications, and designs.

What I, personally mean by it, is "A store that is made of other stores". This fits perfectly with the definition of the Composite Pattern at least the one of wikipedia: The composite pattern describes a group of objects that are treated the same way as a single instance of the same type of object.. The alignment may, or may not go further than that. We should definitely brush up on the famous design pattern: Making sure it helps us get the design right though -- not detract us from designing the actual solution we need.

What do we need? Some design patterns, and if possible actual code, to help us solve the problems that fit this pattern.

Sometimes the problem is simply, we need a container to contain multiple stores. Note that without going further, it's not really a composite pattern yet. A composite patterns happens when we also need this container to act as a store (i.e. a (collections.abc...) Collection, Mapping, or MutableMapping)) itself.

Still, let's start with that: We need a container of stores. Ideas (and possibly designs and tools) can be found in i2.MultiObject.

Just containing the stores, we do have a composition, but what do we need to do with that store container? That's the question that may, or may not, lead us to a "composite" in the "composite pattern" sense. One thing that the i2.MultiObject tools do is make what is called, there, a fanout; where we want to do some kind of action on several objects. For example, feed some (single) input to (multiple) functions to get the (multiple) outputs they produce, or enter/exit multiple context managers "simultaneously".

So, if it's something like that we want to do with our multiple stores,i2.MultiObject could be a place to look, and possibly use to make a "MultiStore", or "StoreFanout". Note that the resulting object is not necessarily a store type (nor do we need it to be).

An actualy i2-free implementation of the "StoreFanout" reader (no writes/deletes) can be found in dol, it's called FanoutReader.

On the store compositions that are stores themselves, we have one very common situation, so common that there's a builtin for it: collections.ChainMap. It's (one way of doing) a union of stores (Mappings). Often, ChainMap fits the bill, but sometimes it's behavior is not exactly how you'd want it. For example, when you write on a ChainMap object, it will write on the first store of the list -- but maybe you want to write on all of them, or some of them; or maybe when you find a key in the ith store, you want to write it's value in all the stores up to the i-1th.
The caching module of dol has some tools regarding these kinds of patterns.

Another mentionable implementation of a specific choice for a store aggregate is the FlatReader, which for example, is used to make
FlatZipFilesReader.

So the question here, again, is what do you want to do with your composition of stores?. And if you actually want the composite to be a store itself (this, have an actual "composite pattern"), how should the composite interface related to the contained stores?

[thorwhalen]

Example interfaces:

s[k] = (blob, meta)  # with s[k] = blob possible?
# is the meta explicit or is there an implicit meta that is merged with it?

# ... several abstractions later, other proposal
s[k] = v  # but with functional parameters that tell s how to extract blob and meta from v

# ... and then
s.append(v)  # where s knows how to get k, meta and blob from v

[valentin-feron]

My commit here regarding the architecture and interface