Present and Future of dol

[valentin-feron]

A general discussion to describe the actual state of dol, but also to align on its goal and fundamentals.

Definition

As described in the README file, dol is a set of "tools to make your interface with data be domain-oriented, simple, and isolated from the underlying data infrastructure".

What does that mean?

Domain-oriented

The tool adapts to the context it is applied to, meaning that it provides objects that can deal with a specific domain model.

@dataclass
class User:
    username: str
    password: str
    email: str

user = User('alice', 'mysecretpassword', '[email protected]')

# This is NOT domain-oriented
user_store[user_id] = {
    'username': user.username,
    'password': encrypt(user.password),
    'email': user.email,
}

# Whereas this IS domain-oriented
user_store[user_id] = user  # The store takes care of interpreting the user object and transforming it accordingly

Simple

The interface is intuitive and normalized. dol uses the python built-in interface for Mappings (key/value) for CRUD operations with data.

# This is NOT simple neither normalized. It requires to define a specific language to interact with the data
user_ids = user_store.list_ids()
users = user_store.list()
user_store.create(user_id, user)
assert user_store.contains(user_id)
user = user_store.retrieve(user_id)
user.email = '[email protected]'
user_store.update(user_id, user)
user_store.delete(user_id)

# When this is simple and normalized. The developer interacts with the store as it were a dict
user_ids = list(user_store)
users = list(user_store.values())
user_store[user_id] = user
assert user_id in user_store
user = user_store[user_id]
user.email = '[email protected]'
user_store[user_id] = user
del user_store[user_id]

Isolated from the underlying data infrastructure

dol totally hide the storage logic and infrastructure, making the interface the same regardless how and where the data is stored

class User:
    username: str
    password: str
    email: str
    photo: bytes

# This is NOT isolated from the underlying data infrastructure
user_store.add_to_db(user_id, user)
photo_store.upload_to_cloud(user.photo)

# When this is isolated from the underlying data infrastructure
user_store[user_id] = user  # The store takes care of dispatching the right information to the right target (db and cloud)

State of the Art

So far, we have a bunch of concrete dol solutions. However, 2 groups stand out:

• Filesystem-like solutions
  Examples: file stores, s3dol, azuredol. They all organize nested data using paths that can be used in a flatten way so any child element is reachable from any node, regardless the number of intermediary nodes between them.

store['some/path/to/data'] = data
sub_store = store['some']
assert assert store['path/to/data'] == data

• DB solutions
  Examples: mongodol, sqldol. They interact with DBMS.

Now, are the 3 main principles well followed?

Simple and Normalized

This principle has been our main concern. We've tried to stick to the python built-in Mapping interface. But sometimes, it is hard to keep the train on rails when the underlying technology's interface differs too much from the key-value one, or when it provides more complex features that are beyond the Mapping interface.

The most relevant example would be mongodol (even if I think sqldol would have been an even better example if it were more developed and used in production, because of relational db). MongoDB provides a bunch of extra features (like distinct, aggregators, etc.) that can hardly be wrapped by a Mapping object. Furthermore, the implementations provided by MongoDB for those features are close to the metal, meaning that they are optimized, so we want to use them. Sometimes, it is hard to call native features without breaking the Mapping interface.

Examples:

from mongodol import MongoStore
users = MongoStore(host='mongodb://localhost:27017', collection_name='users')
distinct_names = {u['name'] for u in users}  # Hard to call the native distinct feature from mongodb here, since users.__iter__ is called first

from pymongo import MongoClient
distinct_names = (
    MongoClient('mongodb://localhost:27017')  # get client
    ['test']['users']  # get users collection in test db
    .distinct('name')  # get the list of distinct values of the 'name' field
)
list(distinct_names)
# Easier to use the native distinct feature, but we loose the `Mapping` interface since we just delegate the feature to the store instantiation

The second example seems to be an easier way to hook into the native features of the underlying technology, but, by bypassing the Mapping interface, we also lose in normalization and we depend more on the interface of the tech (that we want to be isolated from in the first place).

We also have a hard time normalizing the different levels of a nested store sturcture, because the underlying objects manipulated are often very different in their nature.

clients = S3Dol(...)
buckets = clients['client_name']  # I get an instance of S3ClientDol
users = buckets['users']  # I get an instance of S3BucketDol

Even if those objects are stores, they don't share a common behavior.

Domain-Oriented and Isolated from the Underlying Data Infrastructure

I put them together because they are strongly related with each other. Indeed, the closer the interface is to the domain, the further it is from the underlying technology.
We have a bunch of tools that allow us to wrap the Mapping interface so it becomes domain-oriented. The main tool is definitely wrap_kvs, which enables us to add key and value transformers to a mapping:

# This is NOT domain-oriented
users = MongoStore(host='mongodb://localhost:27017', collection_name='users')
users[user_id] = {
    'username': user.username,
    'password': encrypt(user.password),
    'email': user.email,
}

# When this is domain-oriented

users = wrap_kvs(users, obj_to_data=encode_user, data_to_obj=decode_user)
users[user_id] = user

However, most of the usage we have of the stores are still very close to the underlying technology. A first example is the one we described above, with using native features by passing extra parameters to the store constructor. Another example is that we often have a 1:1 relationship between store instances and targets (or N:1, but rarely 1:N, which would be more domain-oriented)

# We tend to create one store per target technology...
raw_data_store = S3Store(...)
metadata_store = MongoStore(...)
raw_data_store[key] = wf[bytes]
metadata_store[key] = wf[info]

# ...instead of having one store per domain model element
wf_store = WfStore(...)  # Here, we can use the FanoutStore class, along with wrap_kvs, to take care of dispatching the data to multiple targets
wf_store[key] = wf

Evolution

The main purpose of dol is to make the user forget about how and where the data is stored, being able to CRUD data in a domain-oriented way.

Composite tree of stores

The user should be able to do the following, regardless the type and the number of storage target for the store and regardless which level it is applied to:

data = {
    'some': {
        'nested': {
            'data': {
                'structure': 'with_data'
            }
        }
    }
}
store = mk_store(**kwargs)  # with the proper parameters for the store
assert isinstance(store, Store)  # Instance of common Store class
store[k] = data
store = store[k]
assert type(store) == Store  # The sub store is also a Store
assert list(store) == ['some']
store['other'] = data['some']
assert store['other']['nested']['data']['structure'] == 'with_data'
store['other']['nested']['data']['structure'] = {
    'with': {
        'nested': data
    }
}
del store['other']
assert 'other' not in store
assert len(store) == 1

Consistency between concrete dol solutions

We must ensure interface and behavior consistency between the different concrete dol solutions. For instance, in azuredol, if a store is instantiated from a container name but the container does not exist, the container will be created during the store instantiation. On the other hand, in s3dol, the bucket (s3-equivalent to azure storage container) is created when setting the first key-value, not during store instantiation. This is just an example, all our dol solutions differ in term of interface and behavior at some point. We need to fix that, so the user can really forget about what kind of store he's using.

Add your proposals in comments!

[charlie-dufort]

@valentin-feron Thank you very much for putting this together! It's great to have this kind of material that summarizes the ins and outs of such tools.

As a recent "user" (as opposed to "developer") of this tool, I can definitely relate to some of the points you developed above.

Here's a couple of examples:

• Path separator in stores: Slack discussion on the themes of Filesystem-like solutions and Consistency between concrete dol solutions. I'd really like to have your input on this.

• When you write assert isinstance(store, Store) # Instance of common Store class.
  I've had to ask myself several times recently what type hint I should use for Stores, and what kind of operation I should expect a Store to support.
  Until the (abstract?) Store class and type hint is available, here are a few questions I have on the matter:

  • Your document seems to indicate that Mapping is the most appropriate type hint to use. Am I correct?
  • I've seen the type GettableContainer used for config sources. I was thinking it could be appropriate for Stores too, as I guess we could potentially have unsized Stores in the future?
  • I've also seen KvReader as being a common parent class for Stores. In what does it differ from Mapping and GettableContainer? Is it more appropriate to use it as a type hint for Stores?
  • So far I've been using Sequence | Mapping, because I was thinking Sequences could also be another kind of Stores, for which ordering makes more sense than naming. What do you think about that?

I know that's a lot of questions, but I told myself this was the ideal place to put them when I read your post!

[thorwhalen]

The 80% answer is easy. Dependency Inversion. Depend on abstractions! So most of the time you want to use Mapping or MutableMapping (when you need to write/delete on the store), etc.

You don't want to use Store or KvReader or any of the dol classes. You shouldn't depend on those. They only enable us to map to the builtin interfaces.

Then the question of being more or less precise comes in. I sometimes need only to be able to do obj[key] and nothing else. I don't need len, or in etc. So I made my own protocol: Gettable. @andeaseme disagrees with type precision. I disagree with @andeaseme. But the python community and gods seem to be closer to @andeaseme's opinion. So there.

Yes, Sequence could be another kind of store, and there is some support for these kinds of things.
Know that dol gives some support to the Collections interface. In fact, the three main interfaces dol chose to use in its hierarchy is Collection -> Mapping -> MutableMapping.

If you need an append or extend, with a key-value backend, then the appendable.py module is there to help.
If you need something more like a stream (iterable, iterator)... py2store used to have that, but the functionality was moved and developed in creek to keep dol more key-value focused.

There will maybe one day be a package that takes care of all kinds of collections more globally...

[valentin-feron]

@charlie-dufort, I use Store in my example just to show that we have a common interface at every level (btw, it is not the Store class that you can find in dol)

But @thorwhalen told me today that it will be very hard (or even impossible) to get this composite tree since we'll have, at some point, to break the interface for some levels of the tree (depending on the underlying data infrastructure and/or the data itself). We'll talk more about that in our calls.

[thorwhalen]

Thanks for this description.

I few notes:

• There is a third main kind of data we apply dol to: Python objects. Whether it be modules, classes, or a set of functions, there's a lot of "implicitly key-value" interfaces that can be given an explicit, consistent, Mapping interface. I've done this with zipfile and configparser for example, as well as some ad hoc cases. I described this direction in this issue: KV perspectives for standard libs
• The critiques and "evolutions" you describe here only scratch the service. There are some more fundamental limitations of the design, and choices that have to be made regarding it (for example -- how much should we enable/support departures from the simple Mapping interface, and if so, do we do so by deep changes in the inner design, or outside tooling?).
• The directions of improvement you mention seem to be only desires for some specialized tooling for some particular cases. It so happens that all of these have been talked about, and some worked on.
  • Going beyond the mapping core to encompass consistencies that can be found in main categories. This has been described in Structure and consistency of the base persisters and Storage system specific functionality extension (example: file systems) and even implemented here. Early on I reflected on What should the base classes be?. Should this be revised?
  • Revealing less the backend choice. Definitely a wish of mine, but we need to see what it means. This could easily become akin to chasing a mirage. It's about bringing the backend interface to the domain interface. It's a N:N problem, whose complexity we reduce by decomposing into N:1 and 1:N. How much of the backend is revealed is the responsibility of the user. We just make it easier to do. Of course, though, more tools can be built. For example, we worked on, in early years (2019) on making URI-based factories so that store classes wouldn't reveal their backends via their arguments.
  • fanouts and malls -- which you've worked on recently. For the particular question of metadata, there was this py2store issue: Metadata support py2store#58, and even the idea of malls was already found in Malls: Building multi-store objects
  • Recursive stores. I don't think we want the behavior you describe (a composite pattern behavior) to be how all stores work. But we definitely want to capacity to make such stores easily. There's a lot of discussions/issues (and POC, and ad hoc code) on this theme. For example Making conditional_data_trans better, and generally; recursively applying wrappers, kv_walk: docs, tests, tools, and recipes, Dirs, files, and key-dependent value transformation, Nested updates.

[thorwhalen]

I'll try to accumulate a few links to previous work that is worth reading before making any dol re-design choices.

Others

• Proxies and Wrappers (and wrapt design in general)

Ours

• Base operations for file (local or cloud) systems
• Ideas for extensions of dol -- namely when we will re-architecture it
• Fluid interface ideas