Add ephemeral block headers to the history network spec

[pipermerriam]

fixes #336

should probably depend on #291

What is wrong

We are going to move towards keeping unproven headers in a separate content type than the proven ones. This is based on the discussion in #336

How was it fixed

We will refer to headers near the front of the chain that are not in a stable HistoricalSummaries as "ephemeral" headers. Their only proving path is through tracing the Header.parent_hash through the chain of headers and anchoring to a trusted header. This trusted header must come from an independent oracle (such as the portal beacon network).

All clients in the network will be assumed to have some quantity of these recent headers. The Ping and Pong custom_data field will also be modified to include a new field that signals how many of these recent headers a client is holding.

Clients will need to perform a data migration on their database to remove the previous unproven headers that are no longer valid after this change. Values in the database that previously used the None proof type should be purged, and the subsequent values should be modifed to correct the Union selector if they have been stored with that as part of the data field.

Clients will need to implement a mechanism to purge ephemeral headers as or after they transition out of the most recent epoch boundary.

Clients will need to implement a syncing mechanism to fetch and fill their database with the appropriate window of recent headers that are still within the current unfinalized epoch.

Clients will need to implement an oracle (such as the portal beacon network) to be continually updated on what the most recent block header is, and to use this as the validation anchor for any headers requested this way.

Our hive and testing infrastructure will need to be updated to reflect these changes.

[kdeme]

We could for now also leave this None in there, allowing this PR to get merged, before clients get ready to migrate data.

And that would allow us to decide on how to deal with it best (future wise).

[kdeme]

Suggested change

	ephemeral_header_payload = List(ByteList[1024], limit=256)
	ephemeral_header_payload = List(ByteList[1024], limit=MAX_EPHEMERAL_HEADER_PAYLOAD)

[pipermerriam]

I'm considering just moving the limit to 16384 (2**16). Very loose rationale is that at the point where we transition across an epoch boundary, it will take some time for the bridge to gossip new headers into the network.

Broader rationale is:

• Allow time for bridges to gossip proven data as epoch boundaries pass by.
• Eliminate awkwardness of having to use uint16 for ephemeral_header_count since it allows for a value up to 16384. This way we don't have a weird "dead range" at the top of that value that is invalid and thus would require some kind of special handling.
• The proving path for headers deep in the ephemeral chain is expensive requiring proving all the way up to head, which adds a natural disincentive for clients to use this old ephemeral data when they don't have to.

Clients can still of course set this value to 8192 or any value they want. This gives us some freedom of movement. Realistically, the real value might be something like 9000 as it gives a little time for bridge gossip but still constrains this value appropriately.

[morph-dev]

Instead of "Gossip", should we say "OFFER/ACCEPT"? We can clarify that this includes gossip as well, but that would be implied.

[morph-dev]

nit: Should we explicitly declare that ancestor_count can't be bigger than MAX_EPHEMERAL_HEADER_PAYLOAD? Or is it enough that it's implied by the content type?

Or is it ok to request more than 256, but answer shouldn't have more than that?

[pipermerriam]

I'll make these things explicitly stated. They are enforced by the encoding types but worth making them stated limits too.

[KolbyML]

This PR changes the custom_payload for history. I propose we deploy this change through #348 which should make upgrading custom_payload easier in the future without the risk of breaking interop and allowing us a window for all implementations to update to the new format before deprecating older versions.