Skip to content
This repository has been archived by the owner on Mar 30, 2018. It is now read-only.

Fabric Next

Jump to bottom
Binh Q Nguyen edited this page Jul 25, 2016 · 34 revisions

Status: Draft (not ready for review; not yet prioritized)

This page contains a list of high-level items proposed for development of the next generation Fabric. It includes both existing feature enhancements and new features.

1. Introduction

The key motivation for the next generation is modularization to achieve plug-ability and concurrency. As outlined in the next consensus architecture and the next ledger structure, we want to enable different implementations to plug in at different points, including consensus, storage, and membership services. We can achieve concurrency at endorsers and consenters based on the transaction endorsement policy.

1.2. Deployment Scenarios

The new architecture offers great flexibility in deployments. As the current generation, the simple 1 peer development environment, where a developer may quickly iterate over edit-compile-debug of the fabric code or a chaincode, is still intact. What change are the varieties of network deployment options, from a simple network of a few peers to a complex hundred of peers with different configurations possible.

Conceptually a peer may perform all the functions (submitting, endorsing, ordering, and committing transactions); this is called full peer, or peer. A partial or light peer may perform a specific task like consensus (ordering transactions) or a combination of submitting, endorsing, and committing transactions. A member of a network generally owes a peer and/or some light peers. Currently we have 2 types of peers: Validating peer (VP) and none-validating peer (NVP). We may roughly think of VP as a full peer and NVP as a light peer with submitting and committing tasks.

1.3. Network Deployment

A simple network could be made up of several full peers, where each peer owed by a member with unique enrollment identity in a permissioned network. Building on this, a member may add more peers to increase availability and to support fail-over; however, these single-member peers must operate as a cluster with 1 vote on endorsement. A cluster of peers could be achieved by a shared ledger and a proxy to dispatch messages to the member peers, or a leader-followers pattern.

1.4. Consensus Service

Consenters may independently operate to form a consensus network, isolating from the other peer functions (submitting, endorsing and committing transactions). This capability allows the members to set up a consensus network such that it can serve multiple blockchains, which only need to deploy light peers (without consenter enabled). The consensus network can be agnostic to the transactions enabling data isolation such that submitters may replace the transaction payload with its hash.

2. Detail Development Items

The following items either come from the proposed next architecture or known enhancements to the existing functions that have been circulated as issues. Each item will be associated with 1 or more issues, but they are all here for the big picture.

2.1. Consensus
  1. Decouple consensus from peer: Start with an implementation of a centralized consensus service that runs on a single consenter (lets call this "SOLO" consensus) with the broadcast/deliver API as described in https://github.com/hyperledger/fabric/wiki/Next-Consensus-Architecture-Proposal. SOLO consensus is to be seen for v2 what NOOPS is for v0.5, i.e., a straightforward consensus service implementation that facilitates the development of components in parallel. SOLO is to be used in development and testing but, obviously, not in production.

After implementing SOLO we can parallelize peer development (Sec 2.1.1) and consensus service development (Sec 2.1.2).

2.1.1 Peer development
  1. Implement basic submitting-endorsing-committing peer communication and protocols following https://github.com/hyperledger/fabric/wiki/Next-Consensus-Architecture-Proposal incl. state transfer. This requires a lot of interaction with Ledger development (Sec 2.2)
  2. Implement basic endorsement policies. Basic endorsement policies should be very static to begin with, e.g., a fixed set of endorsers' signatures is needed to endorse a transaction (e.g., t endorsers out of n).
  3. Have the first version of the peer framework successfully tested against SOLO consensus service.
  4. Explore richer endorsement policy specifications (but not too rich - we do not want another chaincode in the endorsement policy).
  5. Implement peer reconfiguration (addition/removal of peers, both committing peers and endorsers).
  6. Implement peer support for confidential chaincodes (could be done in paralel with some items above)
2.1.2 Consensus service development (after SOLO)
  1. Consensus channels: A blockchain network has 1 consensus channel by default that every peer listens on; this is called public channel. We can establish a separate channel per confidential domain where only permitted peers may subscribe to. Transactions sent to a channel will be ordered respective to each other within the channel, so a batch only contains transactions from the channel, not from any other channels.
  2. Extract PBFT as a stand-alone consensus service from v0.5 codebase
  3. (optionally) plug in Apache Kafka as the (crash-tolerant) consensus service in HL. This would be a nice exercise, and will also give us some rough idea of what is the upper bound on performance of the future consensus service. Moreover, for a certain, limited number of use-cases, even a crash-tolerant consensus service would be sufficient and here Kafka as the HL consensus service may fit nicely.
  4. Implement support for consenter reconfiguration (dynamic membership)
  5. Add support for multiple channels in consensus service (similar to Kafka topics)
  6. Implement novel scalable BFT consensus protocols, replacing PBFT. Non-exhaustive list of candidate protocols/frameworks includes
2.1.3 Other consensus items
  1. Dynamic membership: Allowing peers to come and go and the blockchain network should continue to operate according to the consensus algorithm.
  2. Remove classic and sieves modes, merge batch with core
  3. Implement whitelisting of nodes running consensus (issue #756)
  4. Bound the number of incoming transactions
  5. Prevent censorship of requests
2.2. Ledger

  1. Transaction rw-set: A submitter composes a transaction consisting of [header, payload, rw-set] where rw-set is the set containing the state variables that the transaction reads from (read-set) and the state variables that the transaction writes to (write-set). The rw-set is created via transaction execution simulation (not writing to database). This simulation is also done by the endorsers of the transaction to fulfill the endorsement policy of the transaction.

  2. Enhance API to enable pluggable datastore: Decouple the current API implementation from RocksDB to enable plugging in different datastore.

  3. File-based datastore: This is the default (reference) implementation, where blocks (transaction log) are stored in structured files as marshaled objects such that data replication is a matter of copying files.

  4. World state cache: For better performance, we need to cache the state to help submitters quickly build the rw-set.

  5. Ability to archive/prune "spent" transactions: To reduce the size of the ledger, peers may removed the transactions that are no longer needed (of course that transaction must have been archived) for operation. A transaction is not needed to be in the ledger when its write-set has been consumed. We don't think that it is necessary to remove the block structure since it is small and fixed, but we can retain the transaction ID in the block.

  6. SQL-like queries (point in time, filter): Many requests from developers to provide ability to query states or transactions with some filters.

2.3. Chaincode

Currently we have 2 types of chaincodes: System and user chaincodes. System chaincode is built with the peer code and initialized during peer startup, whereas, user chaincode is built during deploy transaction and runs in a sandbox (default to Docker container).

System chaincode concept allows us to reorganize the peer chaincode services into system chaincodes. For example, chaincode life-cycle (deploy, upgrade, terminate), chaincode naming, key management, etc can be implemented as system chaincodes since these are "trusted" operations which require access to the peer.

  1. Life-cycle management issue 1127: A chaincode begins life at deployment and ends at termination (out of service). Along this timeline, there might be several upgrades either to fix bugs or to amend functions. We have implemented deployment (via Deploy transaction) but have not done (completed) upgrade and terminate. Upgrading of a chaincode is not only involved code changes but also data (and potentially security) migration. We can implement chaincode life-cycle in a system chaincode, named "uber-chaincode", such that, to deploy a chaincode, we would send an Invoke transaction to the uber chaincode; similarly for upgrade and terminate. This means that, our transaction types now would only consist of Invoke and Query. And Deploy transaction is no longer needed.

  2. Naming service issue 1077: Naming service is a system chaincode to map chaincode ID to a user-friendly name, allowing applications to use the name instead of the long ID to address a chaincode.

  3. Calling another chaincode with security: We currently have chaincode calling chaincode locally but without security. Security means both access control and transaction confidentiality, which means whether the callee is visible to the caller. Multiple confidential domains complicate the picture, so the first implementation should focus on chaincode calling chaincode within the same confidential domain. Calling a chaincode in the "public" domain should allow only queries since changes to a chaincode state requires a transaction visible to that chaincode's endorsers.

  4. Access control: The TCert attributes provide a mechanism for the chaincode code to control whom may perform the function by matching the attributes with the intended permissions. For example, uber chaincode may control who can deploy chaincodes by adding access control logic to the Deploy function.

2.4. Membership Services
  1. TCert attributes: We continue the development of attributes as part of TCert, including backend API for external systems to inject attributes and encryption of the attribute values.
  2. Decentralization: Members of a network want ability to enroll their users.
  3. Key rotation: Ability to replace peer key as a system chaincode transaction.
  4. Key revocation: Ability to revoke a key as a system chaincode transaction.
  5. HSM support: Ability to work with HSM to protect keys.
  6. Auditability: Provide auditing APIs
2.5. Protocol
  1. Enhance message structures: Remove unused or duplicated fields; add version number.
  2. Status codes and messages: Standardize the response message status codes and their messages.
  3. Error handling: Standardize the error messages and their handling such as when to bow out vs log and move on vs propagate up the chain.
  4. Extensions: How to extend the message protocol to allow the application to add more fields.
2.6. SDK
  1. Nodejs: Event handling, chaincode deployment API
  2. Java: Similar to Node.js SDK
  3. Go ? Any need for Golang SDK?
  4. Depreciate REST from peer
  5. CLIs to use Nodejs SDK
2.7. Transaction confidentiality

  1. Support mode flexible user-level confidentiality policies, i.e., encrypt transaction contents in a way, such that:

    1. The content content of transactions associated to a chaincode (where content is transaction payload and associated state updates) is decryptable to the chaincode creator, and a subset of users (dynamically) managed by the chaincode-creator.
    2. The content of transactions (e.g., payload) can be decryptable only by the invoker and a set of users specified by the latter.
    3. With the consent of the chaincode creator, chaincode state updates caused by that chaincode' (invoke) transaction,
      can be decryptable by the invoker/set of users specified by the invoker.

  2. Support for basic endorser-level confidentiality policies, i.e., encrypt transaction contents, such that:

    1. Only the set of endorsers that are indicated to be included in the endorserSet of that chaincode (defined by the chaincode creator at deploy time) can decrypt the details of the transaction, execute the associated chaincode, and specify/endorse subsequent state updates.

  3. Support for confidentiality-preserving events. That is, we offer the ability to confidential chaincodes to emit events that only authorized event recipients (users) can read (decrypt).

  4. Basic support for chaincode to chaincode invocation for confidential contracts that belong to the same confidentiality domain (e.g., all endorsers of the chain).

  5. Event security

2.8. Upgrade Fabric

We divide the fabric upgrade into 3 phases to gradually build up the capabilities.

  1. Bug fixes: This is applying patches to the peer code such that no backward compatibility needed. The thought here is that one may just replace the peer code with the new build, and it should resume.
  2. Protocol changes: This kind of updates requires backward compatibility.
  3. Ledger migration? We don't anticipate this kind of updates any time soon.
2.9. Integration with External Applications for Events and Data

The following features are proposed to allow a Hyperledger-based network to interact with external applications via events and APIs.

  1. Plug-able support for external message queues for event subscription. An organization should be able to configure the peer node it owns to pump event messages from the node into a message queue for application integration purposes and (internal) business process management.

  2. Plug-able support for external message queues for async transaction submission. An organization should be able to configure the peer node it owns to take transaction submissions from a message queue in order to better handle peak load (load leveling). This essentially enables asynchronous APIs for transaction submissions.

  3. Support calling external APIs (public or protected) from inside chaincode without introducing non-determinism. Blockchain networks rely on an "oracle" role to accomplish this. In addition, in V2 architecture, the submitting peer can make the API call once during transaction simulation and pass the resulting data along to endorsers which will not need to call the APIs again (which would introduce non-determinism). This provides an additional (simpler) route when the submitting peer are trusted by the endorsers.

Clone this wiki locally