feat(rust): persist application data in a database

[etorreborre]

This PR replaces the current storage of persistent data in separated files with the storage of data is a SQLite database.

Motivation

There is a large amount of data need to run an Ockam node or application:

• project, user
• identity
• node name
• trust context
• TCP listener
• TCP outlet
• etc...

All this data is inter-related:

• a node has a default identity
• a trust context can be created via a project
• a node has a TCP listener
• a TCP outlet can be created on a given node
• etc...

At the moment, everything is stored in files (and some information just lives in memory, like a started TCP inlet).
Managing those files separately makes it hard to process the relationships between the data and creates some operational issues (there's no way to have atomic transactions to modify data for example), so a database fits better our data model.

Changes

This is a large PR because the access to data, via the CliState struct was not very well encapsulated. As a result it was not possible to just replace one persistence solution with another to leave most of the code unchanged.

The following sections describe the major changes, from low-level crates to high-level ones.

ockam_node

In this crate we introduce a storage/database folder with an SqlxDatabase. sqlx is the Rust library used to handle the database connections and queries. It has been chosen because it supports several "drivers", SQLite, but also Postgres, if we need to migrate to Postgres in the future.

When an SqlxDatabase is started on a given path, a migration file is applied to create or migrate all the SQL tables we need to persist data. The migration files are available in src/storage/database/migration.

Additionally there are traits and conversion functions to allow the data types used in Ockam: Identifier, ChangeHistory, PurposeKey etc... to be mapped to simpler SQLite types when inserting data.

Note that there was no real attempt to make the current schema optimum. But with migrations we should be able to evolve it gracefully.

ockam_vault

In that crate we replaced the persistence of secrets in JSON file with key/values to a table in the database.
Note that the access to secrets is done via an interface, SecretsRepository, and that the implementation currently uses one single database for all data but we could as well persist this information in a separated database file since we never do queries across the table storing secrets and other tables.

ockam_identity

In that crate we have now 3 repositories (each of them backed by a SqlxDatabase):

• a ChangeHistoryRepository to store the ChangeHistories. Note that we don't store VerifiedChanges in order to avoid someone changing the history of an identity by just accessing the database file. When a ChangeHistory is loaded from the database it has to be verified in order to get a full Identity
• an IdentityAttributesRepository to store the attributes associated to an identifier (I reunited the read and the write interface because I don't think that the split is that useful)
• a PurposeKeysRepository to store keys and their associated purpose (secure channel or credentials)

Then most of the changes come from:

• the split of the previous IdentitiesRepository into ChangeHistoryRepository + IdentityAttributesRepository
• some utility functions added for trust contexts, for recreating an identity from a change history, or getting credential data from a Credential
• the removal of the previous storage structs + the initial attempt at using SQLite to store policies and identities

ockam_abac

A new PolicyRepository interface has been added to that crate, with the corresponding PolicySqlxDatabase implementation.
The previous implementations have been removed.

ockam_api

This is the crate with the largest amount of changes. The CliState struct holds all the data that we are interested in via a set of repositories:

• IdentitiesRepository to store named identities, i.e. the fact that we can associate a name to an identity and set a default identity
• CredentialsRepository to store named credentials
• VaultsRepository to store named vaults
• NodesRepository to store data about background nodes (their name, their pid, etc...)
• TrustContextsRepository to store data about trust contexts
• ProjectsRepository to store project data
• SpacesRepository to store space data (note that the association between Space and Project is not exposed at the repository level but is implicit with the space_name field in Project)
• EnrollmentsRepository to keep track of enrolling information for users
• UsersRepository to store user info as retrieved after authentication
• ModelStateRepository to store data for the desktop application: created inlets and outlets

The functions exposed by the repositories cannot be accessed directly. They are hidden behind proper methods on the CliState. For example creating a Node involves several repositories and this logic is hidden behind the CliState::create_node function (or one of its variants).

The main CliState functions are provided in files, domain by domain:

• nodes
• credentials
• projects
• spaces
• identities
• vaults
• nodes
• trust_contexts
• users
• enrollments
• policies
• secure_channels

ockam_command

Most of the functions changed in that crate now call a "facade" function on CliState to access or modify the local state.
In other cases like getting the list of all projects an InMemoryNode is started to first synchronize the local state with the Controller state. This kind of code could be moved to the CliState as well since it has all the necessary information without having to start an InMemoryNode.

ockam_app_lib

The changes here are very similar to the changes to the ockam_command crate. Most of the places where CliState was accessed to get or modify data, I had to modify calls.
One notable addition is the ModelStateRepository to store "outlet statuses" and "incoming services" in the database.