The need for a key transformation framework

[thorwhalen]

KeyPath

KeyPath's name is not quite precise enough, and it can lead to misaligned expectations. As the document claims it's;

A key mapper that converts from an iterable key (default tuple) to a string (given a path-separator str)

To be more precise, it converts a non-string iterable to a string one.

Really, KeyPath is meant to deal with situations where we have a nested mapping that we want to access through a "path string", like this:

from dol import KeyPath

d = {'a': {'b': {'c': 1}}}
s = KeyPath()(d)
assert s['a/b/c'] == 1

s = KeyPath('.')(d)
assert s['a.b.c'] == 1

And it works with write:

s['a.b.d'] = 2
assert s['a.b.d'] == 2

Note: but not if the path prefix is not there

from pytest import raises

with raises(KeyError):
    s['a.c.d'] = 2

But it's not KeyPath's fault: KeyPath just maps keys.
It's because d is a dict.
On the other hand if you make it a tree (recursive defaultdict) it works:

from collections import defaultdict
tree = lambda: defaultdict(tree)
dd = tree()
dd['a']['b']['c'] = 1  # to get the tree equivalent of {'a': {'b': {'c': 1}}}
ss = KeyPath('.')(dd)
ss['a.c.d'] = 2
assert ss['a.c.d'] == 2

A more general KeyPath (path transformer)?

KeyPath works well for the intent is was made for, but does not fully address the "path transformation" problem by offering flexibility in mapping one kind of path to another.
It could be extended to be more flexible, using a similar (more recently, and better designed) logic of path_get and path_set.

If we decide to do so though, we have to consider the compromise between non-breaking-changes and better design.

StrTupleDict

StrTupleDict and StrTupleDictWithPrefix are the existing frameworks to do templated path-like key transformation.
They allow you to create mappings any of string, tuple, namedtuple or dict to a string, tuple, namedtuple or dict, based on templates. This in turn can be used to create dol key transforming wrappers.

from dol import StrTupleDict, wrap_kvs

d = {'a/b/c': 1, 'a/b/d': 2, 'a/b/e': 3}

path_trans = StrTupleDict('{first}/{second}/{third}')

s = wrap_kvs(
    d, key_of_id=path_trans.str_to_tuple, id_of_key=path_trans.tuple_to_str
)
s['a', 'b', 'c']

But StrTupleDict comes from ancient history, and carries a lot of design legacy: Both because the context it was designed for was less general than what we want now, and because the author, yours truly, was less mature than he is now.

For example, the templates only allowed you to handle a parameter space of fixed dimensions.
In the example above, our paths lie in 3 dimensions, but if we would want to handle, say, file-paths that covered different levels, like below, we couldn't.

d = {'a/b': 1, 'a/c/d': 2, 'e': 3}

s = wrap_kvs(
    d, key_of_id=path_trans.str_to_tuple, id_of_key=path_trans.tuple_to_str
)
from pytest import raises

with raises(ValueError):
    list(s)

embody and jinja

I'd like to take care of the path transformation problem in dol itself without dependencies. At least refactor the functionality we have to a few (possibly one) consistent choice(s) to solve the problem.

That said, it must be noted, that this problem of transformation based on templates is a subset of what embody is aimed at covering.

One must also note that jinja seems to be standard go-to for templating. The documentation says:

Jinja is a fast, expressive, extensible templating engine. Special placeholders in the template allow writing code similar to Python syntax. Then the template is passed data to render the final document.

As far as I know though, jinja is for strings only, not for any python objects, which embody wants to be when it grows up. That said, it could perhaps be used to do so -- either if through its under-the-hood functions, or through encoding python objects through strings.