v0.26.0: Address normalization, `agora://` scheme, registry improvements

Major improvements since v0.25.1:

Agora now uses and accepts agora:// as scheme for its RPC interface

Agora now recognizes addresses with the scheme agora, and assume its default
port to be 2826. This will be further used in future releases to replace
the REST API as the default way for nodes to communicate.

DNS name compression is now understood by the registry

The registry wasn't previously able to understand DNS name compression,
rejecting any message that made use of it. This has now been fixed.

DNS registry now serves flash zone correctly

Previously, the registry would serve data for the validator zone for flash too.

CacheDB is now used for banned or whitelisted nodes

Instead of the old binary-serialized file, the ban manager will now store information
about nodes in the cache database.

Agora now normalizes addresses

Agora is now normalizing addresses internally, leading to a more consistent output,
greatly reducing the risks of misconfiguration.

v0.25.1: Fixed DNS responses, correctly handle many missing txs, fix a DoS on empty block

Major improvements since v0.25.0:

DNS: Name registry now returns the question with the answer

The name registry was mistakenly not including the question whenever it was
answering a DNS query, in violation of the specification. This is now fixed.'

DNS: Responses will now include the proper RCode

Previous response could include a valid answer along with a FormErr RCode,
instead of the correct NoError. The RCode is now set correctly.

Rewards: Do not crash on very short block interval

Using block intervals of exactly 5 seconds would crash the nodes at the payment interval,
due to a division by 0. This has been corrected.

Prevent nodes getting stuck when there are too many unknown transactions

When the nodes was trying to fetch unknown transactions, if there were too many of them,
the call will fail, and the node would fail to retry with a lower limit.
The call is now always bounded to a reasonable value.

Do not crash when looking up a merkle path in an empty block

When calling the API to look up a merkle path inside of an empty block,
the node would assert and shut down before verifying the hash.
This out-of-order bug has been fixed and the node now correctly handles this situation.

v0.25.0: Improved RPC interface & URL validation

Major improvements since v0.24.1:

Validators will now always return a frozen UTXO regardless of enrollment status

This affects the GET /public_key and POST /handshake endpoints,
and should allow for a more stable network.

Nodes now support their peers updating UTXOs

This will allow connected clients to re-enroll with a different UTXO.

Name registry will now allow pending enrollment to register

To improve the reactivity of the name registry, registrations are now accepted
as soon as an enrollment exists in the pool, instead of waiting for a block
externalization as was previously the case.

Name registry will now properly clear expired validators

When an enrollment expires, the name registry will now remove the address
from its database instead of indefinitely keeping it.

Block hashing algorithm has changed

The way the hash is derived from the block header has been altered to match
the order of the fields in the structure.

Multiple internal improvements

The Docker image now ships with LDC v1.28.0 instead of v1.26.0,
some noisy log messages have been tamed, and using libconsensus is now easier.

RPC connection are now properly directional and merged

The node will now attempt to merge RPC connections, leading to a single bi-directional
connection per peer instead of a server and a client one.
In addition of reducing the load on the system, this allows to better keep track
of a peer state.

Fix serialization of Set into strings

A previous fix erroneously turned Set into a string representing an array,
instead of an array (of potential strings). This is no longer the case.
Additionally, the empty Set is now correctly serialized.

Agora now handles node addresses consistently internally

The internals of Agora have been revamped to correctly deal with addresses,
and peers having either a REST or an RPC interface.
The change will extend to the configuration file in the near future.

The consensus protocol now enforces Key lock type on freeze transactions

While not previously enforced, it was relied upon in multiple places.

v0.24.1: agora-cli improvements and inclusion in docker image

This patch release adds agora-cli by default to the generated Docker image,
and fixes some outstanding usability issues with it,
mostly with improved argument parsing.

v0.24.0: Updated block signature scheme, registry persistences, block include pre-images

Major improvements since v0.23.1:

Validator can now configure their registration interval

Validators periodically register their address to the name registry.
The period was previously fixed at two minutes.
It can now be configured, and the default has been kept at two minutes.

API methods accepting a Set will now format as a JSON array

Previously, Set in parameters or return value were formatted following Agora's internal representation.
This made it hard and brittle for external tools to interact with those specific endpoints.
Set now always format as a JSON array.

Fixed a registry crash when a query didn't contain a dot

An oversight in the registry led to an OutOfBoundError when a DNS query for a name
such as "foo" (instead of "foo.bar") was performed.
The registry will no longer crash and will correctly reject those.

BlockHeader: Fields missing_validators and random_seed have been replaced with preimages

Previously, the block header included those two fields that were derived from the pre-image.
However, certain consensus rules required the pre-images to be available,
leading to complex situation where some blocks could be fetched but not their corresponding pre-images.
The pre-images are now directly included in the block header.
To save disk space, they can later be compressed by omitting all but the last pre-images
for long-living validators.

Transaction pool will now reject double spend with different unlock_age

Previously, the transaction pool would not detect a double spend that
came with a different unlock_age value.

Name registry now have persistence

The name registry will now persist its state to the cache database,
leading to faster resume time and easier maintainance.

Fixed a bug where duplicated addresses would be returned in GET /node_info

Under some configurations, some addresses could be returned with and without a trailing slash.
This has now been fixed and addresses will always be returned without a trailing slash.

Block signature scheme has been changed

The block signature scheme has been changed to remove a vulnerability which could allow
an attacker to derive a node private's key.
The r value is now longer offset, instead s is set to be the pre-image's value,
and a valid signature requires that R be revealed.

v0.23.1: Bug fix for AXFR, TestNet rewards and some optimizations

The following bug fixes have been done on top of v0.23.0:

TCP interface now reuses connections

Connections are now kept alive instead of being eagerly closed,
leading to less jitter in resources allocation.

Reduced memory footprint of future Enrollment

An optimization has been done in the transaction pool,
which will lead to improved performance and decreased memory usage
when an Enrollment referencing a not-yet-externalized stake is received.

Commons payout: Take block interval into account

The commons payout code was not propely taking the block interval into account.
While the value is fixed for CoinNet, it is different for TestNet.

Fixed a crash triggered by mishandling log messages

A newly introduced log message uncovered a bug that happened under rare
circumstances when log messages were being written to a file,
and said bug has now been fixed.

DNS: Use the proper value when transfering CNAME records

The recent implementation of AXFR uncovered a bug in the CNAME record handling,
which provided the wrong value (either replicated or not fully qualified).

v0.23.0: Overhauled pre-images API, new API for inter-nodes communication, block rewards fixed

Major improvements since v0.22.0:

Some sections of the configuration file are now properly validated

Previously, some nested sections were not adequately validated during config parsing,
which would lead to cryptic errors on startup.
Those sections are now properly validated and the resulting errors will be much more user-friendly.

The metadata.dat storage is obsolete

This file, stored alongside the cache and state DB, was used for network metadata.
It has been superseeded by the cache DB, and can now be completely removed,
all the relevant informations are new stored in their own table.

Flash: Private channels are now gossipped again

Since Agora's Flash path finding is source based, when a private channel
funder wants to receive a payment through their private channel,
the source would have to know about existence of the channel.

The initial implementation of private channels (in v0.22.0) did not gossip them,
leading to routing failures.

Flash: Channel updates are now signed externally

To completely remove the need to communicate one's private key to the node
for the Flash layer to work, the previously-available POST /change_fees
(FlashControlAPI.changeFees()) have been removed and its task now needs
to be performed by the wallet.

Pre-image GET endpoints have been overhauled

The following breaking changes have been made to the endpoints used to get pre-images:

• FullNode.getPreimages (GET /preimages) is now getPreimagesFrom (GET /preimages_from).
  The start_height parameter is still present, however end_height has been removed,
  and the endpoint will now return all known pre-images from that height.
• FullNode.getPreimagesForEnrollkey (GET /preimages_for_enroll_keys) is now getPreimages.
  The parameter and its semantic are unchanged.

Pre-image revelation is now configurable

In order to avoid slashing in the event of a minor downtimes, nodes reveal their pre-images
slightly in advance. Historically this has been fixed to 6 blocks, ensuring nodes can be
offline for up to an hour without being slashed (more if a watchtower is configured).
This value is now configurable under the setting validator.max_preimage_reveal.
It's default value is still set to 6.

Introduced flash node seed for uncollaborative close attempts

When an uncollaborative close is made and detected by a service provider (e.g. wallet operator),
previously-allocated fees might be insufficient.
In order to deal with this situation, a flash node seed has been introduced (flash.seed),
which ensures that closes can always be successful.

DNS registry now supports AXFR

AXFR requests are now supported by the name registry, allowing to transfer whole zones from
an authoritative server to a secondary one.

Nodes will now register themselves to the registry more eagerly

Instead of relying on a timer, nodes will now try to register to the name registry
as soon as they externalize a block with their enrollment.

Removed POST /register_listener endpoint

This HTTP-only endpoint, which wasn't used in production, has been removed,
as it also had no effect.

Block rewards have been corrected

Experimental support for block rewards, added in the previous release,
accidentally delayed rewards by one extra period (1 day) in some cases.
This has been fixed and rewards will now be placed in the right block.

The getBlocksFrom endpoint will now return up to 24 blocks

Up from the previous limit of 20. This limit is in place to prevent DoS attacks,
and may be lifted altogether in the future when sending stream of data without
loading them in memory is implemented.

Nodes will now longer reject Enrollments if they have the stake in their tx pool

Previously, nodes would only accept Enrollment once the matching stake (UTXO) had been externalized.
However, the consensus protocol allows both to be externalized in the same block,
so this was an unnecessary limitation.
Nodes will now correctly check their tx pool for a valid stake before rejecting an enrollment.

Added experimental binary-based communication protocol

Agora currently supports communicating with clients and other nodes alike using a REST interface.
This REST interface can be exposed through HTTP or HTTPS. Additionally, the interface can be bound
to any number of network interface, under any port, to allow for any server-side configuration.

While offering a client interface which is REST-based is convenient, as it allows easy integration
and development from JavaScript, using REST between nodes tend to be wasteful,
as there is a large overhead involved in the HTTP(s) layer and JSON serialization.

From this release, Agora supports a binary interface over TCP, which a client can reference
using the tcp:// scheme in its configuration, and a server can enable using a type: tcp
in its interfaces configuration array.

Ideally, nodes in the network should switch to communicate using this interface only,
and may expose a separate public REST interface.

v0.22.0: Block rewards, improved Flash API & private channels

Major improvements since v0.21.0:

Flash nodes now use separate registry endpoints

Flash nodes should now register themselves via the POST /flash_node endpoint,
and can be accessed via the GET /flash_node endpoint of the registry,
instead of using the /validators endpoint as before,
which is now only used for enrolled validators.

Registering a validator in the registry now uses POST

Before, the registry was expecting a PUT /validators request to register an address,
however that was at odds with the method definition.
Validators now need to use a POST request.

Removed recv_pk parameter from FlashNode.openChannel API

This parameter was redundant with the content of the chan_conf parameter
and shouldn't be provided anymore.

Flash: openChannel now properly validates funding TXs

The openChannel call of the Flash API was previously missing some validation
on the funding transaction: invalid funding transactions will now be properly rejected.

Flash: ThinFlashNode & AgoraFlashNode have been removed

The flash nodes have been unified under a single type, FlashNode,
providing more flexibility by better defining its relationship to Agora.
As part of this, a new parameter, funder_address, is now required for openChannel,
and peer_address is required for FlashControlAPI.openNewChannel.

FlashControlAPI is now a BlockExternalizedHandler

FlashControlAPI now implements the BlockExternalizedHandler API as well,
which allows one to put a Flash node in the event_handler.BlockExternalized section.

The fee check now properly checks the combination of data and transaction fee

Previously, the fee checker mistakenly checked both fee individually,
instead of the combination of both.

The accumulated_fee table has been replaced by block_fees

The old table has been dropped, and the new table is used to store block fees
to be paid in the next block reward between payout period.

FlashControlAPI.openNewChannel: Removed redundant reg_pk

This parameter has been removed, as it is already provided by funding_utxo.address,
and should not be provided anymore.

Block rewards are now included in Coinbase transactions

From this release, block rewards are now properly distributed every payout period.
A payout period is currently set to 144 blocks (1 day).

When the payout block is reached (height % 144 == 0 and not genesis),
a coinbase transaction is added to the block, that contains the data fees,
transaction fees, and confirmation rewards for the period before last.
This means that payout for the range [0; 143] will happen at block 288,
payout for range [144; 287] happens at block 432, etc...

Block rewards are distributed according to the amount set in the whitepaper,
and proportional to the number of blocks a validator has signed during the period.

Registry: TCP queries are now supported

The network registry now also listens to the TCP port (53 by default) in addition of the UDP one,
meaning that it might need to run privileged under some systems.
This will allow zone transfer and large payload to be exchanged.
Note that systemd users might need to bind to the public interface only,
as systemd-resolvd binds to the loopback interface's port 53.

Enrollments no longer include cycle_length

Cycle length was originally included in Enrollment to allow nodes to configure
the duration of their enrollments. However, it had turned into a consensus-controlled
parameters since the inception of Agora, and hence wasn't used anymore.

Registry: Reworked zone configuration

Registry configuration has been simplified:

• The registry.dns section has been folded directly into registry;
• The registry.[dns.]authoritative section has been replaced by registry.validators
  and registry.flash, and an authoritative boolean value has been added to them;
• registry.validators controls the zone for validators (nodes that are enrolled);
• registry.flash controls the zone for Flash nodes (nodes with a channel open);

Node configuration now includes a realm field

In order to support the new function of the registry, the configuration has a new option: node.realm.
This field controls the root domain under which the validators and flash zones can be found.
Available realms at the moment are: testnet.bosagora.io and coinnet.bosagora.io.

Flash: Channels can now be private

The FlashControlAPI.openNewChannel endpoint now accepts a is_private boolean as a parameter.
If set to true, the channel will not be gossipped, and requests to use it for routing will be denied.
This allows users to keep their channels unknown to the rest of the world, further privacy.

v0.21.0: Merged registry into Agora

Major improvements since v0.20.0:

Removed backward compatibility PUT /transaction endpoint

Agora v0.20.0 introduced a POST /transaction endpoint, which is more correct than PUT.
The PUT endpoint was left for v0.20.0 release for backward compatibility, and has now been removed.

Maximum coin supply raised

In preparation for the implementation of block rewards, the maximum coin supply has been raised,
from 500,000,000 to 4,950,000,000.

Registry is now an integral part of Agora

The name registry, previously an independent server (built with dub -c name-registry),
and hosted in its own docker image (bosagora/registry) is now an integral part of Agora.
As a result, the aforementioned configuration has been removed, and the bosagora/registry
Docker Hub repository will no longer be updated (and removed in the near future).
Instead, one should set registry.enabled to true in the configuration file.
All previous registry options are available in the registry section.

Registry will now validate registered validators

Previously, any syntactically valid public key (essentially, any public key a user would generate)
was accepted for registration by the name registry. From this release, the registry will first check
its ledger, and only accept enrolled validators. The registry needs to follow the network as either
a full node or a validator.

Few databases improvements regarding fees

In preparation for block rewards, a few changes have been made to the database schema, requiring a reset.

Registry will now do more validation and return correct record type

A few extra validations have been added to the registry to ensure it complies with the DNS spec more accurately,
which should not have any visible effect on well formed queries.
Additionally, the registry now properly recognize IP addresses and domain name, and will return the correct
record type (CNAME) when a domain name is used for registration, instead of trying to convert it to an IP
address and ignore it on failure as it previously did.

Fixes a spurious formatting error on block reward error

When a full payout period does not contain any transaction, an error is currently triggered.
This error was however swallowed by another error which triggers as a result of the first one.
That second error has been fixed, and the corner case of a payout period without any fee will be
fixed in the upcoming release.

Talos: Dependencies upgrades

Talos dependencies have been overhauled and simplified, resulting in a more secure build.

Added support for zone configuration in the registry

Users can now configure the SOA record of a zone, using the following syntax:

registry:
  enabled: true
  dns:
    authoritative:
      # The zone name
      validators.bosagora.io:
        # Contact information. `dns.validators.bosagora.io` is also accepted,
        # as it is the SOA syntax.
        email: [email protected]
        # The following are optional but available
        refresh:
          minutes: 1
        retry:
          seconds: 30
        expire:
          minutes: 10
        minimum:
          minutes: 1

v0.20.0: Flash validation, major UX improvements, registry role expanding

Major improvements since v0.19.0:

Older CONFIRM envelopes are now properly ignored

A recent security fix removed the ability to gossip signatures via CONFIRM signatures as it was insecure.
However, envelopes where still checked for signatures, which would then fail to validate. This has been fixed.

Minimum fee is now properly enforced

All transactions must now include the minimum fee as defined by the consensus configuration,
not only transactions with a payload as was previously the case.

Pre-image catchup is now more reliable

Pre-image catchup has been improved to be more predictable and eager,
reducing the risk of starting a nomination round with missing pre-images.

API: Many endpoints have been renamed and made more conformant to REST principles

Previously, our API was a mix of naming conventions and HTTP verbs, due to its organic growth.
This release sees the following renames:

• PUT /transaction => POST /transaction (a backward-compatible method exists for this release);
• POST /receive_preimage => POST /preimage;
• POST /enroll_validator => POST /enrollment;
• POST /receive_envelope => POST /envelope;
• PUT /receive_block_signature => POST /block_signature;

Duration are now parsed consistently in the config file

The config file used to mix different styles for duration parsing.
Either the unit was implicit, and documented, or it was part of the field name.
Starting from this release, all fields that deal with duration can specify their value more expressively.

For example, the field that control a node's request timeout used to be specified as:

node:
  # This value is in milliseconds
  retry_delay: 3000

On the other hand, the field controlling how often block catchup is performed was specified as such:

node:
  # This value is in seconds, as the field name shows
  block_catchup_interval_secs: 20

Now, both needs to be specified as such:

node:
  retry_delay:
    msecs: 3000
  block_catchup_interval:
    seconds: 20

The valid units are weeks, days, hours, minutes, seconds, msecs, usecs, hnsecs, nsecs.
Units are additive, so having both seconds: 60 and minutes: 41 has the same effect as minutes: 42.
Fields with a suffix have been renamed to be suffix-less.
This change affects the following config fields:

• node.retry_delay (used to be in msecs);
• node.timeout (used to be in msecs);
• node.block_time_offset_tolerance_secs => node.block_time_offset_tolerance;
• node.network_discovery_interval_secs => node.network_discovery_interval;
• node.block_catchup_interval_secs => node.block_catchup_interval;
• node.relay_tx_interval_secs => node.relay_tx_interval;
• node.relay_tx_cache_exp_secs => node.relay_tx_cache_exp;
• node.preimage_reveal_interval (used to be in seconds);
• node.preimage_catchup_interval (used to be in seconds);
• flash.timeout (used to be in msecs);
• flash.max_retry_time (used to be in msecs);
• banman.ban_duration (used to be in seconds):

Flash: Gossip latest channel update alongside channel open

Previously, there was a possible race condition where the update could be received before the open event,
leading to the update being discarded and parties being out-of-sync.
The latest update are now transmitted alongside the open event to prevent this from being an issue.

Nomination now ignores early envelopes

The nominator would previously accept envelopes as soon as the previous block was externalized,
leading to situation where an attacker with large resources could bias a node on externalizing
a block earlier than it should. There is now a tolerance of 5% for time jitter.

Flash nodes now do extra validation on channel open

A few issues around channel opening were fixed, and nodes will now properly fetch blocks for validation.

The transaction pool can now be configured to exclude txs with much lower fees than average

The transaction pool now keeps track of transaction fee, and can be configured to reject
transactions where the fee is lower than a certain percentage.
This percentage is set to 80 by default, so a transaction that is less than 80% of the fee rate
(not absolute fee) of the average fee rate of the transaction pool will be rejected.
This can be disabled by setting the value to 0.

Name registry now has a configuration file

The name registry's role has evolved to include a full DNS server in addition of an HTTP server,
and most of its options were previously set via CLI arguments.
To make it easier to use, and extend its capabilities even further, it will now read a (required)
configuration file under config.yaml.

The only required field in that new configuration file (if dns.enabled is not set to false)
is dns.authoritative, which defines the zones this server is authoritative for.

The previous CLI arguments are mapped as follows:

• --bind-host (-h) => http.address;
• --bind-port (-p) => http.port;
• --stats-port => http.stats_port (0, or disabled, by default, as before);
• --no-dns => dns.enabled set to false (true by default);
• --verbose => Use logging section, similar to Agora, instead;

Registry now listen to TCP DNS queries in addition to UDP

The first iteration of the DNS support in the registry only listened to DNS queries.
It now also listens to TCP queries, as required by the protocol, on the same port than the UDP listener.

Registry: Authoritative zones are now configurable

When the DNS is enabled, the dns.authoritative array needs to have a least one entry.
The registry will only respond to DNS queries for which it is authoritative.
A future iteration will improve this to allow the DNS to follow an authoritative server
and not be authoritative itself, but for the time being this can be achieved via another DNS server.

Configuration: Restored behavior of root logger

This fixes a minor regression that crop up in v0.19.0: Configuring the logger named root now affects all loggers again.