rpc: improve latency by not blocking worker threads polling IO notifi…

…cations (#3242)

* rpc: limit the number of blocking threads in tokio runtime

By default tokio allows up to 512 blocking threas. We don't want that
many threads, as they'd slow down other validator threads.

* rpc: make getMultipleAccounts async

Make the function async and use tokio::task::spawn_blocking() to execute
CPU-bound code in background. This prevents stalling the worker threads
polling IO notifications and serving other non CPU-bound rpc methods.

* rpc: make getAccount async

* rpc: run get_filtered_program_accounts with task::spawn_blocking

get_filtered_program_accounts can be used to retrieve _a list_ of
accounts that match some filters. This is CPU bound and can block the
calling thread for a significant amount of time when copying many/large
accounts.

* rpc: use our custom runtime to spawn blocking tasks

Pass the custom runtime to JsonRpcRequestProcessor and use it to spawn
blocking tasks from rpc methods.

* Make `get_blocks()` and `get_block()` yieldy

When these methods reach out to Blockstore, yield the thread

* Make `get_supply()` yieldy

When this method reaches out to accounts_db (through a call to `calculate_non_circulating_supply()`), yield the thread.

* Make `get_first_available_block()` yieldy

When this method reaches out to blockstore, yield the thread

* Make `get_transaction()` yieldy

When this method reaches out to blockstore, yield the thread

* Make `get_token_supply()` yieldy

When this method reaches out to methods on bank that do reads, yield the thread

* Make the choice of `cpus / 4` as the default for `rpc_blocking_threads`

* Encode blocks async

* Revert "Make `get_first_available_block()` yieldy"

This blockstore method doesn't actually do expensive reads.

This reverts commit 3bbc57f.

* Revert "Make `get_blocks()` and `get_block()` yieldy"

Kept the `spawn_blocking` around:

* Call to `get_rooted_block`
* Call to `get_complete_block`

This reverts commit 710f9c6.

* Revert "Make `get_token_supply()` yieldy"

* Reverted the change to `interest_bearing_config`
* Reverted moving `bank.get_account(&mint)` to the background pool

This reverts commit 02f5c94.

* Share spawned call to `calculate_non_circulating_supply` between `get_supply` and `get_largest_accounts`

* Create a shim for `get_filtered_indexed_accounts` that sends the work to the background thread internally

* Send call to `get_largest_accounts` to the background pool

---------

Co-authored-by: Steven Luscher <[email protected]>