v0.10.8

Fixed

• The HTTP adapter would remove a trailing slash on a subdirectory when specifying an extended path, so for instance http://example.com/subdir/ with a param of ?query= extended path would produce the URL http://example.com/subdir?query=, but should now produce: http://example.com/subdir/?query=.

• Matic autoconfig is now enabled for mainnet. Matic nops should remove any custom tweaks they have been running with. In addition, we have better default configs for Optimism, Arbitrum and RSK.

• It is no longer required to set DEFAULT_HTTP_ALLOW_UNRESTRICTED_NETWORK_ACCESS=true to enable local fetches on bridge tasks. Please remove this if you had it set and no longer need it, since it introduces a slight security risk.

• Chainlink can now run with ETH_DISABLED=true without spewing errors everywhere

• Removed prometheus metrics that were no longer valid after recent changes to head tracking:
  head_tracker_heads_in_queue, head_tracker_callback_execution_duration,
  head_tracker_callback_execution_duration_hist, head_tracker_num_heads_dropped

Added

• INSECURE_SKIP_VERIFY configuration variable disables verification of the Chainlink SSL certificates when using the CLI.

• JSON parse tasks (v2) now permit an empty path parameter.

• Eth->eth transfer gas limit is no longer hardcoded at 21000 and can now be adjusted using ETH_GAS_LIMIT_TRANSFER

• HTTP and Bridge tasks (v2 pipeline) now log the request parameters (including the body) upon making the request when LOG_LEVEL=debug.

• Webhook v2 jobs now support two new parameters, externalInitiatorName and externalInitiatorSpec. The v2 version of the following v1 spec:

  {
    "initiators": [
      {
        "type": "external",
        "params": {
          "name": "substrate",
          "body": {
            "endpoint": "substrate",
            "feed_id": 0,
            "account_id": "0x7c522c8273973e7bcf4a5dbfcc745dba4a3ab08c1e410167d7b1bdf9cb924f6c",
            "fluxmonitor": {
              "requestData": {
                "data": { "from": "DOT", "to": "USD" }
              },
              "feeds": [{ "url": "http://adapter1:8080" }],
              "threshold": 0.5,
              "absoluteThreshold": 0,
              "precision": 8,
              "pollTimer": { "period": "30s" },
              "idleTimer": { "duration": "1m" }
            }
          }
        }
      }
    ],
    "tasks": [
      {
        "type": "substrate-adapter1",
        "params": { "multiply": 1e8 }
      }
    ]
  }
  

  is:

  type            = "webhook"
  schemaVersion   = 1
  jobID           = "0EEC7E1D-D0D2-475C-A1A8-72DFB6633F46"
  externalInitiatorName = "substrate"
  externalInitiatorSpec = """
      {
        "endpoint": "substrate",
        "feed_id": 0,
        "account_id": "0x7c522c8273973e7bcf4a5dbfcc745dba4a3ab08c1e410167d7b1bdf9cb924f6c",
        "fluxmonitor": {
          "requestData": {
            "data": { "from": "DOT", "to": "USD" }
          },
          "feeds": [{ "url": "http://adapter1:8080" }],
          "threshold": 0.5,
          "absoluteThreshold": 0,
          "precision": 8,
          "pollTimer": { "period": "30s" },
          "idleTimer": { "duration": "1m" }
        }
      }
  """
  observationSource   = """
      submit [type=bridge name="substrate-adapter1" requestData=<{ "multiply": 1e8 }>]
  """
  

• Task definitions in v2 jobs (those with TOML specs) now support quoting strings with angle brackets (which DOT already permitted). This is particularly useful when defining JSON blobs to post to external adapters. For example:

  my_bridge [type=bridge name="my_bridge" requestData="{\\"hi\\": \\"hello\\"}"]
  

  ... can now be written as:

  my_bridge [type=bridge name="my_bridge" requestData=<{"hi": "hello"}>]
  

  Multiline strings are supported with this syntax as well:

  my_bridge [type=bridge
             name="my_bridge"
             requestData=<{
                 "hi": "hello",
                 "foo": "bar"
             }>]
  

• v2 jobs (those with TOML specs) now support variable interpolation in pipeline definitions. For example:

  fetch1    [type=bridge name="fetch"]
  parse1    [type=jsonparse path="foo,bar"]
  fetch2    [type=bridge name="fetch"]
  parse2    [type=jsonparse path="foo,bar"]
  medianize [type=median]
  submit    [type=bridge name="submit"
             requestData=<{
                            "result": $(medianize),
                            "fetchedData": [ $(parse1), $(parse2) ]
                          }>]
  
  fetch1 -> parse1 -> medianize
  fetch2 -> parse2 -> medianize
  medianize -> submit
  

  This syntax is supported by the following tasks/parameters:

  • bridge
    • requestData
  • http
    • requestData
  • jsonparse
    • data (falls back to the first input if unspecified)
  • median
    • values (falls back to the array of inputs if unspecified)
  • multiply
    • input (falls back to the first input if unspecified)
    • times

• Add ETH_MAX_IN_FLIGHT_TRANSACTIONS configuration option. This defaults to 16 and controls how many unconfirmed transactions may be in-flight at any given moment. This is set conservatively by default, node operators running many jobs on high throughput chains will probably need to increase this above the default to avoid lagging behind. However, before increasing this value, you MUST first ensure your ethereum node is configured not to ever evict local transactions that exceed this number otherwise your node may get permanently stuck. Set to 0 to disable the limit entirely (the old behaviour). Disabling this setting is not recommended.

Relevant settings for geth (and forks e.g. BSC)

[Eth.TxPool]
Locals = ["0xYourNodeAddress1", "0xYourNodeAddress2"]  # Add your node addresses here
NoLocals = false # Disabled by default but might as well make sure
Journal = "transactions.rlp" # Make sure you set a journal file
Rejournal = 3600000000000 # Default 1h, it might make sense to reduce this to e.g. 5m
PriceBump = 10 # Must be set less than or equal to chainlink's ETH_GAS_BUMP_PERCENT
AccountSlots = 16 # Highly recommended to increase this, must be greater than or equal to chainlink's ETH_MAX_IN_FLIGHT_TRANSACTIONS setting
GlobalSlots = 4096 # Increase this as necessary
AccountQueue = 64 # Increase this as necessary
GlobalQueue = 1024 # Increase this as necessary
Lifetime = 10800000000000 # Default 3h, this is probably ok, you might even consider reducing it

Relevant settings for parity/openethereum (and forks e.g. xDai)

NOTE: There is a bug in parity (and xDai) where occasionally local transactions are inexplicably culled. See: openethereum/parity-ethereum#10228

Adjusting the settings below might help.

tx_queue_locals = ["0xYourNodeAddress1", "0xYourNodeAddress2"] # Add your node addresses here
tx_queue_size = 8192 # Increase this as necessary
tx_queue_per_sender = 16 # Highly recommended to increase this, must be greater than or equal to chainlink's ETH_MAX_IN_FLIGHT_TRANSACTIONS setting
tx_queue_mem_limit = 4 # In MB. Highly recommended to increase this or set to 0
tx_queue_no_early_reject = true # Recommended to set this
tx_queue_no_unfamiliar_locals = false # This is disabled by default but might as well make sure

• Keeper jobs now support prometheus metrics, they are considered a pipeline with a single keeper task type. Example:

pipeline_run_errors{job_id="1",job_name="example keeper spec"} 1
pipeline_run_total_time_to_completion{job_id="1",job_name="example keeper spec"} 8.470456e+06
pipeline_task_execution_time{job_id="1",job_name="example keeper spec",task_type="keeper"} 8.470456e+06
pipeline_tasks_total_finished{job_id="1",job_name="example keeper spec",status="completed",task_type="keeper"} 1

Fixed

• It is no longer required to set DEFAULT_HTTP_ALLOW_UNRESTRICTED_NETWORK_ACCESS=true to enable local fetches on bridge tasks. Please remove this if you had it set and no longer need it, since it introduces a slight security risk.
• Chainlink can now run with ETH_DISABLED=true without spewing errors everywhere

Changed

• The v2 (TOML) bridge task's includeInputAtKey parameter is being deprecated in favor of variable interpolation. Please migrate your jobs to the new syntax as soon as possible.

• Chainlink no longers writes/reads eth key files to disk

• Add sensible default configuration settings for Fantom

• Rename ETH_MAX_UNCONFIRMED_TRANSACTIONS to ETH_MAX_QUEUED_TRANSACTIONS. It still performs the same function but the name was misleading and would have caused confusion with the new ETH_MAX_IN_FLIGHT_TRANSACTIONS.

v0.10.7

Fixed

• If a CLI command is issued after the session has expired, and an api credentials file is found, auto login should now work.

• GasUpdater now works on RSK and xDai

• Offchain reporting jobs that have had a latest round requested can now be deleted from the UI without error

Added

• Add ETH_GAS_LIMIT_MULTIPLIER configuration option, the gas limit is multiplied by this value before transmission. So a value of 1.1 will add 10% to the on chain gas limit when a transaction is submitted.

• Add ETH_MIN_GAS_PRICE_WEI configuration option. This defaults to 1Gwei on mainnet. Chainlink will never send a transaction at a price lower than this value.

• Add chainlink node db migrate for running database migrations. It's
  recommended to use this and set MIGRATE_DATABASE=false if you want to run
  the migrations separately outside of application startup.

Changed

• Chainlink now automatically cleans up old eth_txes to reduce database size. By default, any eth_txes older than a week are pruned on a regular basis. It is recommended to use the default value, however the default can be overridden by setting the ETH_TX_REAPER_THRESHOLD env var e.g. ETH_TX_REAPER_THRESHOLD=24h. Reaper can be disabled entirely by setting ETH_TX_REAPER_THRESHOLD=0. The reaper will run on startup and again every hour (interval is configurable using ETH_TX_REAPER_INTERVAL).

• Heads corresponding to new blocks are now delivered in a sampled way, which is to improve
  node performance on fast chains. The frequency is by default 1 second, and can be changed
  by setting ETH_HEAD_TRACKER_SAMPLING_INTERVAL env var e.g. ETH_HEAD_TRACKER_SAMPLING_INTERVAL=5s.

• Database backups: default directory is now a subdirectory 'backup' of chainlink root dir, and can be changed
  to any chosed directory by setting a new configuration value: DATABASE_BACKUP_DIR

v0.10.6

Added

• Add MockOracle.sol for testing contracts

• New CLI command to convert v1 flux monitor jobs (JSON) to
  v2 flux monitor jobs (TOML). Running it will archive the v1
  job and create a new v2 job. Example:

// Get v1 job ID:
chainlink job_specs list
// Migrate it to v2:
chainlink jobs migrate fe279ed9c36f4eef9dc1bdb7bef21264

// To undo the migration:
1. Archive the v2 job in the UI
2. Unarchive the v1 job manually in the db:
update job_specs set deleted_at = null where id = 'fe279ed9-c36f-4eef-9dc1-bdb7bef21264'

• Improved support for Optimism chain. Added a new boolean OPTIMISM_GAS_FEES configuration variable which makes a call to estimate gas before all transactions, suitable for use with Optimism's L2 chain. When this option is used ETH_GAS_LIMIT_DEFAULT is ignored.

• Chainlink now supports routing certain calls to the eth node over HTTP instead of websocket, when available. This has a number of advantages - HTTP is more robust and simpler than websockets, reducing complexity and allowing us to make large queries without running the risk of hitting websocket send limits. The HTTP url should point to the same node as the ETH_URL and can be specified with an env var like so: ETH_HTTP_URL=https://my.ethereumnode.example/endpoint.

Adding an HTTP endpoint is particularly recommended for BSC, which is hitting websocket limitations on certain queries due to its large block size.

v0.10.5

Added

• Add MockOracle.sol for testing contracts
• Cron jobs can now be created for the v2 job pipeline:

type            = "cron"
schemaVersion   = 1
schedule        = "*/10 * * * *"
observationSource   = """
ds          [type=http method=GET url="http://example.com"];
ds_parse    [type=jsonparse path="data"];
ds -> ds_parse;
"""

Changed

• Default for JOB_PIPELINE_REAPER_THRESHOLD has been reduced from 1 week to 1 day to save database space. This variable controls how long past job run history for OCR is kept. To keep the old behaviour, you can set JOB_PIPELINE_REAPER_THRESHOLD=168h
• Removed support for the env var JOB_PIPELINE_PARALLELISM.
• OCR jobs no longer show TaskRuns in success cases. This reduces
  DB load and significantly improves the performance of archiving OCR jobs.
• Archiving OCR jobs should be 5-10x faster.

Fixed

• Added GAS_UPDATER_BATCH_SIZE option to workaround websocket: read limit exceeded issues on BSC

• Basic support for Optimism chain: node no longer gets stuck with 'nonce too low' error if connection is lost

v0.10.4

[0.10.4] - 2021-04-05

Added

• VRF Jobs now support an optional coordinatorAddress field that, when present, will tell the node to check the fulfillment status of any VRF request before attempting the fulfillment transaction. This will assist in the effort to run multiple nodes with one VRF key.

• Experimental: Add DATABASE_BACKUP_MODE, DATABASE_BACKUP_FREQUENCY and DATABASE_BACKUP_URL configuration variables

It's now possible to configure database backups: on node start and separately, to be run at given frequency.

DATABASE_BACKUP_MODE enables the initial backup on node start (with one of the values: none, lite, full where lite excludes
potentially large tables related to job runs, among others). Additionally, if DATABASE_BACKUP_FREQUENCY variable is set to a duration of
at least '1m', it enables periodic backups.

DATABASE_BACKUP_URL can be optionally set to point to e.g. a database replica, in order to avoid excessive load on the main one.

Example settings:

DATABASE_BACKUP_MODE="full" and DATABASE_BACKUP_FREQUENCY not set, will run a full back only at the start of the node.
DATABASE_BACKUP_MODE="lite" and DATABASE_BACKUP_FREQUENCY="1h" will lead to a partial backup on node start and then again a partial backup every one hour.

• Added periodic resending of eth transactions. This means that we no longer rely exclusively on gas bumping to resend unconfirmed transactions that got "lost" for whatever reason. This has two advantages:
  1. Chainlink no longer relies on gas bumping settings to ensure our transactions always end up in the mempool
  2. Chainlink will continue to resend existing transactions even in the event that heads are delayed. This is especially useful on chains like Arbitrum which have very long wait times between heads.

Periodic resending can be controlled using the ETH_TX_RESEND_AFTER_THRESHOLD env var (default 30s). Unconfirmed transactions will be resent periodically at this interval. It is recommended to leave this at the default setting, but it can be set to any valid duration or to 0 to disable periodic resending.

• Logging can now be configured in the Operator UI.

Fixed

• Chainlink node now automatically sets the correct nonce on startup if you are restoring from a previous backup (manual setnextnonce is no longer necessary).

• Flux monitor jobs should now work correctly with outlier-detection and market-closure external adapters.

• Performance improvements to OCR job adds. Removed the pipeline_task_specs table
  and added a new column dot_id to the pipeline_task_runs table which links a pipeline_task_run
  to a dotID in the pipeline_spec.dot_dag_source.

• Fixed bug where node will occasionally submit an invalid OCR transmission which reverts with "address not authorized to sign".

• Fixed bug where a node will sometimes double submit on runlog jobs causing reverted transactions on-chain

v0.10.3

Added

• Add ADMIN_CREDENTIALS_FILE configuration variable

This variable defaults to $ROOT/apicredentials and when defined / the
file exists, any command using the CLI that requires authentication will use it
to automatically log in.

• Add ETH_MAX_UNCONFIRMED_TRANSACTIONS configuration variable

Chainlink node now has a maximum number of unconfirmed transactions that
may be in flight at any one time (per key).

If this limit is reached, further attempts to send transactions will fail
and the relevant job will be marked as failed.

Jobs will continue to fail until at least one transaction is confirmed
and the queue size is reduced. This is introduced as a sanity limit to
prevent unbounded sending of transactions e.g. in the case that the eth
node is failing to broadcast to the network.

The default is set to 500 which considered high enough that it should
never be reached under normal operation. This limit can be changed
by setting the ETH_MAX_UNCONFIRMED_TRANSACTIONS environment variable.

• Support requestNewRound in libocr

requestNewRound enables dedicated requesters to request a fresh report to
be sent to the contract right away regardless of heartbeat or deviation.

• New prometheus metric:

Name: "head_tracker_eth_connection_errors",
Help: "The total number of eth node connection errors",

• Gas bumping can now be disabled by setting ETH_GAS_BUMP_THRESHOLD=0

• Support for arbitrum

Fixed

• Improved handling of the case where we exceed the configured TX fee cap in
  geth.

Node will now fatally error jobs if the total transaction costs exceeds the
configured cap (default 1 Eth). Also, it will no longer continue to bump gas on
transactions that started hitting this limit and instead continue to resubmit
at the highest price that worked.

Node operators should check their geth nodes and remove this cap if configured,
you can do this by running your geth node with --rpc.gascap=0 --rpc.txfeecap=0 or setting these values in your config toml.

• Make head backfill asynchronous. This should eliminate some harmless but
  annoying errors related to backfilling heads, logged on startup and
  occasionally during normal operation on fast chains like Kovan.

• Improvements to the GasUpdater

Various efficiency and correctness improvements have been made to the
GasUpdater. It places less load on the ethereum node and now features re-org
detection.

Most notably, GasUpdater no longer takes a 24 block delay to "warm up" on
application start and instead loads all relevant block history immediately.
This means that the application gas price will always be updated correctly
after reboot before the first transaction is ever sent, eliminating the previous
scenario where the node could send underpriced or overpriced transactions for a
period after a reboot, until the gas updater caught up.

Changed

• Bump ORM_MAX_OPEN_CONNS default from 10 to 20
• Bump ORM_MAX_IDLE_CONNS default from 5 to 10

Each Chainlink node will now use a maximum of 23 database connections (up from previous max of 13). Make sure your postgres database is tuned accordingly, especially if you are running multiple Chainlink nodes on a single database. If you find yourself hitting connection limits, you can consider reducing ORM_MAX_OPEN_CONNS but this may result in degraded performance.

• The global env var JOB_PIPELINE_MAX_TASK_DURATION is no longer supported
  for OCR jobs.

v0.10.2

Fixed

• Fix a potential hang in headtracker under high database load
• Fix a performance regression introduced in 0.10.0 in the v1 job runner

v0.10.1

Fixed

• Prevent autosaving Task Spec on when Task Runs are saved to lower database load.

v0.10.0

Fixed

• Fix a case where archiving jobs could try to delete it from the external initiator even if the job was not an EI job.
• Fixed a fluxmonitor job bug where submitting a value outside the acceptable range would stall the job permanently. Now a job spec error will be thrown if the polled answer is outside the acceptable range and no ethtx will be submitted. As additional protection, we also now check the receipts of the ethtx's and if they were reverted, we mark the ethtx task as failed.
• Improved performance of the transaction manager by fetching receipts in batches. This should help prevent the node from getting stuck when processing large numbers of OCR jobs.
• Squashed migrations into a single 1_initial migration. If you were running a version older than 0.9.10, you need to upgrade to 0.9.10 first before upgrading to the next version so that the migrations are run.

Added

• A new Operator UI feature that visualize JSON and TOML job spec tasks on a 'New Job' page.

v0.9.10

Fixed

• Fixed a UI bug with fluxmonitor jobs where initiator params were bunched up.
• Improved performance of OCR jobs to reduce database load. OCR jobs now run with unlimited parallelism and are not affected by JOB_PIPELINE_PARALLELISM.

Added

• A new env var JOB_PIPELINE_MAX_RUN_DURATION has been added which controls maximum duration of the total run.