questions/ideas for lowlevel/highlevel storage

[snd]

some thoughts i had while working with storage.

@NikVolf, @pepyakin, @fckt what do you think? i'd really appreciate answers to my questions
and feedback on my ideas :)

currently there's a lowlevel storage API:

https://github.com/paritytech/pwasm-std/blob/a16024a03ac3edb68d68957bfe9463633ea45c13/src/storage.rs#L13

it's a 256bit to 256bit key-value store.
that's a good lowlevel interface!

contract developers however will often want to:

• easily persist something under a name
• use strings as keys
• store arrays
• store mappings
• have all the storage options they are used to in solidity
  • with similar or better ergonomics
• more...

currently those things are difficult and error prone:

persisting something under a name:
https://github.com/paritytech/pwasm-token-example/blob/870d91d6a90091fd3e0214ba6cc82498b02fa636/src/token.rs#L61
https://github.com/paritytech/pwasm-token-example/blob/870d91d6a90091fd3e0214ba6cc82498b02fa636/src/token.rs#L84

mappings:
https://github.com/paritytech/pwasm-token-example/blob/870d91d6a90091fd3e0214ba6cc82498b02fa636/src/token.rs#L86
https://github.com/paritytech/pwasm-token-example/blob/870d91d6a90091fd3e0214ba6cc82498b02fa636/src/token.rs#L70

it's easy to get key clashes the current way. those could have very bad consequences.
the current API puts a burden on the programmer to be extremely careful.
i think we should minimize the possibility for such errors in APIs for contract code that has to be extremely reliable later!

wouldn't it be great to have a more highlevel storage API that makes the above mentioned things easy?
that API could live in it's own crate and build on top of the lowlevel one.

for this it probably makes sense to extract storage into its own trait and types.

something like:

trait BasicStorage {
  fn write(key: &H256, value: &[u8; 32]);
  fn read(key: &H256) -> [u8; 32];
}

this way the highlevel API doesn't have to awkwardly depend on the entire Context.
it can just work on anything implementing BasicStorage.

following are a couple of other questions/ideas on storage.
keep in mind that i'm still learning about this.

how is the storage implemented all the way down?

...when WASM gets executed.

would another lowlevel API be possible?
just want to make sure we aren't unnecessarily limiting ourselves
by a specific lowlevel API.

is there a way to delete from storage?

i don't see any way to delete from storage.
shouldn't we add a storage_delete?

seems to me that currently a contracts storage can only ever grow and not shrink.

what happens when we read a key that hasn't previously been written to?

does it return zeroed memory?
does it return uninitialized memory?
that's a source for bugs.

can we detect (in the lowlevel implementation of storage) whether
a key has previously been written to?

maybe storage_read could return an Option. we're using rust not C.

idea: storage struct

what if a contracts storage were modeled by a struct:

#[contract_storage]
struct TokenContract {
    totalSupply: U256,
    ownerHash: H256,
    balances: Mapping<Address, U256>,
}

that could enormously improve ergonomics and prevent many errors!

it would offload a lot of checking from the programmer to the implementation
of #[contract_storage] (or similar) and the rust typesystem.

we can easily map/store such a struct into the lowlevel storage and potentially
make it very safe.

i strongly believe we should do everything reasonably possible to
make rust wasm smart contract development safe since
errors in smart contracts have the potential to be enormously expensive!

[NikVolf]

it's actually out of scope of pwasm-std, but worth discussion anyway

i believe there could be multiple implementations of storage in different crates based on the same low-level-api, user will just stick to one of them

delete operations is actually done with storage_write(key, [0u8;32]), there is no key removal nor key creation in the low-level state db (if you read non-existant key, for example, you get [0u8; 32])

it's all defined in the runtime behaviour (kovan/parity/whatever), not on wasm level

and nothing in wasm is expected to produce unintialized memory or trigger undefined behaviour, just fyi

[snd]

it's actually out of scope of pwasm-std, but worth discussion anyway

thought so. still thought this would be the most appropriate repo for a discussion for now

[snd]

it's all defined in the runtime behaviour (kovan/parity/whatever), not on wasm level

could someone point me to the code where that's implemented?

[snd]

i believe there could be multiple implementations of storage in different crates based on the same low-level-api, user will just stick to one of them

that would be good

[NikVolf]

@snd
https://github.com/paritytech/parity/blob/master/ethcore/wasm/src/runtime.rs#L180
it uses Ext

Ext is implemented for Externalities:
https://github.com/paritytech/parity/blob/master/ethcore/src/externalities.rs

[snd]

@NikVolf thanks! i looked at it a bit and things make a lot more sense

[Pzixel]

Any progess/suggestion on that? I feel it really hard to move a contract with several fields (some of which are variable-sized) on wasm. I'm trying to adopt this code, but it seems to be extremly complicated.

[Pzixel]

Some progress is done here.
Now it support arrays (with nesting!), but interface is really fragile. So it's working, but it worth to hide it under some kind of attribute, as proposed in the first post.

[usd-yamazaki]

I am very interested.
Is there any additional information on this matter?