v1.2.4

Substreams improvements

• Performance: prevent reprocessing jobs when there is only a mapper in production mode and everything is already cached
• Performance: prevent "UpdateStats" from running too often and stalling other operations when running with a high parallel jobs count
• Performance: fixed bug in scheduler ramp-up function sometimes waiting before raising the number of workers
• Added the output module's hash to the "incoming request" log

Reader node and Beacon blocks

• The reader-node-bootstrap-url gained the ability to be bootstrapped from a bash script.

  If the bootstrap URL is of the form bash:///<path/to/script>?<parameters>, the bash script at
  <path/to/script> will be executed. The script is going to receive in environment variables the resolved
  reader node variables in the form of READER_NODE_<VARIABLE_NAME>. The fully resolved node arguments
  (from reader-node-arguments) are passed as args to the bash script. The query parameters accepted are:

  • arg=<value> | Pass as extra argument to the script, prepended to the list of resolved node arguments
  • env=<key>%!d(MISSING)<value> | Pass as extra environment variable as <key>=<value> with key being upper-cased (multiple(s) allowed)
  • env_<key>=<value> | Pass as extra environment variable as <key>=<value> with key being upper-cased (multiple(s) allowed)
  • cwd=<path> | Change the working directory to <path> before running the script
  • interpreter=<path> | Use <path> as the interpreter to run the script
  • interpreter_arg=<arg> | Pass <interpreter_arg> as arguments to the interpreter before the script path (multiple(s) allowed)

Note

The bash:/// script support is currently experimental and might change in upcoming releases, the behavior changes will be clearly documented here.

• The reader-node-bootstrap-url gained the ability to be bootstrapped from a pre-made archive file ending with tar.zst or tar.zstd.

• The reader-node-bootstrap-data-url is now added automatically if firecore.Chain#ReaderNodeBootstrapperFactory is non-nil.

  If the bootstrap URL ends with tar.zst or tar.zstd, the archive is read and extracted into the
  reader-node-data-dir location. The archive is expected to contain the full content of the 'reader-node-data-dir'
  and is expanded as is.

• Added Beacon to known list of Block model.

v1.2.3

• Fix marshalling of blocks to JSON in tools like firehose-client and print merged-blocks

v1.2.2

Auth and metering

• Add missing metering events for sf.firehose.v2.Fetch/Block responses.
• Changed default polling interval in 'continuous authentication' from 10s to 60s, added 'interval' query param to URL.

Substreams

• Fixed bug in scheduler ramp-up function sometimes waiting before raising the number of workers
• Fixed load-balancing from tier1 to tier2 when using dns:/// (round-robin policy was not set correctly)
• Added trace_id in grpc authentication calls
• Bumped connect-go library to new "connectrpc.com/connect" location

v1.2.1

Fixed

• Fixed tools firehose-client which was broken because of bad flag handling

Added

• Added --api-key-env-var flag to firehose-clients, which allows you to pass your API Key from an environment variable (HTTP header x-api-key) instead of a JWT (Authorization: bearer), where supported.

v1.2.0

• Poller is now fetching blocks in an optimized way, it will fetch several blocks at once and then process them.

• Poller is now handling skipped blocks, it will fetch the next blocks until it find a none skipped block.

• Poller now has default retry value of infinite.

• Compare tool is now using dynamic protobuf unmarshaler, it will be able to compare any block type.

• Print tool is now using dynamic protobuf unmarshaler, it will be able to print any block type.

• Print tool is encoding bytes in base64 by default, it can be changed to hex or base58 by using parameter bytes-encoding.

• Added 'x-trace-id' header to auth requests when using --common-auth-plugin=grpc

• Fixed Substreams scheduler sometimes taking a long time to spawn more than a single worker.

• Added ACCEPT_SOLANA_LEGACY_BLOCK_FORMAT env var to allow special tweak operations

v1.1.3

• Removed useless chainLatestFinalizeBlock from blockPoller initialization

v1.1.2

• Added Arweave to known list of Block model.

v1.1.1

• Added FORCE_FINALITY_AFTER_BLOCKS environment variable to override block finality information at the reader/poller level. This allows an operator to pretend that finality is still progressing, N blocks behind HEAD, in the case where a beacon chain fails to do so and is intended as a workaround for deprecated chains like Goerli.

v1.1.0

• Updated substreams and dgrpc to latest versions to reduce logging.

• Tools printing Firehose Block model to JSON now have --proto-paths take higher precedence over well-known types and even the chain itself, the order is --proto-paths > chain > well-known (so well-known is lookup last).

• The tools print one-block now works correctly on blocks generated by omni-chain firecore binary.

• The various health endpoint now sets Content-Type: application/json header prior sending back their response to the client.

• The firehose, substreams-tier1 and substream-tier2 health endpoint now respects the common-system-shutdown-signal-delay configuration value meaning that the health endpoint will return false now if SIGINT has been received but we are still in the shutdown unready period defined by the config value. If you use some sort of load balancer, you should make sure they are configured to use the health endpoint and you should common-system-shutdown-signal-delay to something like 15s.

• The firecore.ConsoleReader gained the ability to print stats as it ingest blocks.

• The firecore.ConsoleReader has been made stricter by ensuring Firehose chain exchange protocol is respected.

• Changed reader logger back to reader-node to fit with the app's name which is reader-node.

• Fix -c "" not working properly when no arguments are present when invoking start command.

• Fix tools compare-blocks that would fail on new format.

• Fix substreams to correctly delete .partial files when serving a request that is not on a boundary.

• Add Antelope types to the blockchain's known types.

v1.0.0

This is a major release.

Operators

Important

When upgrading your stack to firehose-core v1.0.0, be sure to upgrade all components simultaneously because the block encapsulation format has changed.
Blocks that are merged using the new merger will not be readable by previous versions.
Blocks that are sent on the wire (ex: through the relayer) with the previous version will not be readable by the new components (ex: firehose)
This will require either a "Stop the world" upgrade, or a very careful execution of the following steps:

1. stop merger
2. start a new reader (making sure it is not picked up by the relayer
3. stop relayer+firehose+substreams (DOWNTIME STARTS HERE)
4. launch new version of relayer+firehose+substreams that connects to the new reader (DOWNTIME ENDS HERE)
5. start new merger
6. replace other components like index-builder

Added

• New binary firecore which can run all firehose components (reader, reader-stdin, merger, relayer, firehose, substreams-tier1|2) in a chain-agnostic way. This is not mandatory (it can still be used as a library) but strongly suggested when possible.

• Current Limitations on Ethereum:

  • The firecore firehose app does not support transforms (filters, header-only --for graph-node compatibility--) so you will want to continue running this app from fireeth
  • The firecore substreams apps do not support eth_calls so you will want to continue running them from fireeth
  • The firecore reader does not support the block format output by the current geth firehose instrumentation, so you will want to continue running it from fireeth

• New BlockPoller library to facilitate the implementation of rpc-poller-based chains, taking care of managing reorgs

• Considering that firehose-core is chain-agnostic, it's not aware of the different of the different block types. To be able to use tools around block decoding/printing,
  there are two ways to provide the type definition:

  1. the 'protoregistry' package contains well-known block type definitions (ethereum, near, solana, bitcoin...), you won't need to provide anything in those cases.
  2. for other types, you can provide additional protobuf files using --proto-path flag

Changed

• Merged blocks storage format has been changed. Current blocks will continue to be decoded, but new merged blocks will not be readable by previous software versions.
• The code from the following repositories have been merged into this repo. They will soon be archived.
  • github.com/streamingfast/node-manager
  • github.com/streamingfast/merger
  • github.com/streamingfast/relayer
  • github.com/streamingfast/firehose
  • github.com/streamingfast/index-builder