fix: update miner mempool iterator query to consider both nonces and fee rates #5541

rafaelcr · 2024-12-07T20:54:59Z

Problem

The current query used by Stacks miners to iterate over pending mempool transactions sorts them only by fee_rate in descending order. This approach creates the following problems:

Since it does not consider transaction nonces, non-mineable transactions with high fees but old (or very future) nonces go to the top of the list and get visited every time, thus wasting block construction time budget.
Transactions with null fee rates are sent to the bottom of the list even if their nonces should be evaluated next, forcing the algorithm to query them separately with some probability of inclusion.

Solution

This PR changes the transaction selection query to a new one that prioritizes those that can be confirmed as fast as possible (lowest valid nonces) with the highest fees as possible. This means that even if it doesn't select the transactions with the highest global fees first, it will select those that can be mined faster therefore optimizing block building time and allowing the miner to fit in more transactions in total within the allotted time.

The new query also mixes in transactions with null fee rates in results by simulating a fee rate for the purposes of ordering only.

CLAassistant · 2024-12-07T20:55:05Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

CLAassistant · 2024-12-07T20:55:06Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

…nto fix/mempool-query

stackslib/src/core/mempool.rs

obycode · 2024-12-16T17:36:33Z

stackslib/src/chainstate/stacks/tests/block_construction.rs

+                vec![
+                    (address_0.clone(), 2),
+                    (address_0.clone(), 3),
+                    (address_0.clone(), 4),
+                    (address_0.clone(), 5),
+                    (address_0.clone(), 6),
+                    (address_1.clone(), 7), // Higher fee for address 1
+                    (address_0.clone(), 7),
+                    (address_1.clone(), 8),
+                    (address_0.clone(), 8),
+                    (address_2.clone(), 9), // Higher fee for address 2
+                    (address_1.clone(), 9),
+                    (address_0.clone(), 9),
+                ],


It looks like the ideal ordering would be:

vec![ (address_2.clone(), 9), // Higher fee for address 2 (address_1.clone(), 7), // Higher fee for address 1 (address_1.clone(), 8), (address_1.clone(), 9), (address_0.clone(), 2), (address_0.clone(), 3), (address_0.clone(), 4), (address_0.clone(), 5), (address_0.clone(), 6), (address_0.clone(), 7), (address_0.clone(), 8), (address_0.clone(), 9), ],

We don't want to penalize ready transactions with higher fees just for having a higher nonce. We want to select all of the transactions with the ready nonce, then order that set by fee rate.

Great catch, will make that change

Great test!

Done @obycode , I had a bug in the WHERE condition for the LEFT JOIN

Nice. I see how this latest version is basically grouped in sets of nonces that are ready for each round:

Round 1: (addr 2, nonce 9), (addr 1, nonce 7), (addr 0, nonce 2)

Round 2: (addr 1, nonce 8), (addr 0, nonce 3)

Round 3: (addr 1, nonce 9), addr 0, nonce 4)

Round 4: (addr 0, nonce 5)

...

I think this is fine, but is there any good way to handle the chained nonces better? That could be part of a separate PR after this one if there is not a clear solution.

It would be great if we could get a good benchmark setup to compare how changes affect the transaction selection timing.

If we push that to a separate PR, can you add a comment in this test specifying the optimal ordering I noted earlier, so we can easily come back to this later.

I see what you mean, let me do some tests here and come back to you with an answer

…nto fix/mempool-query

obycode · 2024-12-16T21:29:07Z

stackslib/src/core/mempool.rs

+                LEFT JOIN nonces ON mempool.origin_address = nonces.address
+                WHERE nonces.address IS NULL OR mempool.origin_nonce >= nonces.nonce


Can you also take the sponsor nonce into account similarly?

jcnelson · 2024-12-16T21:32:14Z

stackslib/src/chainstate/stacks/tests/block_construction.rs

+    let address_1 = origin_addresses[1].to_string();
+    let address_2 = origin_addresses[2].to_string();
+
+    let test_name = "mempool_walk_test_nonce_filtered_and_ranked";


Nit: you can use function_name!()

jcnelson · 2024-12-16T21:32:38Z

stackslib/src/chainstate/stacks/tests/block_construction.rs

+    let address_2 = origin_addresses[2].to_string();
+
+    let test_name = "mempool_walk_test_nonce_filtered_and_ranked";
+    let mut peer_config = TestPeerConfig::new(test_name, 2002, 2003);


Please use TestPeerConfig::new(test_name, 0, 0), so the OS will allocate you ports.

jcnelson · 2024-12-16T21:33:56Z

stackslib/src/core/mempool.rs

+        // Iterate pending mempool transactions using a heuristic that maximizes miner fee profitability and minimizes CPU time
+        // wasted on already-mined or not-yet-mineable transactions. This heuristic takes the following steps:
+        //
+        // 1. Filters out transactions that have nonces smaller than the origin address' next expected nonce as stated in the


As @obycode has pointed out, you will also need to consider sponsor nonces.

jcnelson · 2024-12-16T21:35:55Z

stackslib/src/core/mempool.rs

+                LEFT JOIN nonces ON mempool.origin_address = nonces.address
+                WHERE nonces.address IS NULL OR mempool.origin_nonce >= nonces.nonce
+            ),
+            address_nonce_ranked AS (


You're going to need two of these subqueries -- one for origin addresses and nonces, and one for sponsor addresses and nonces.

jcnelson

Thanks for taking this on @rafaelcr!

This is something that I think is going to need somewhat extensive real-world testing before we make it the default behavior. Do you think you could add this new mempool walk logic as an opt-in alternative, which the node operator can opt to use? This would give miners a chance to compare/contrast the new behavior with the old behavior in an easy-to-rollback manner. A subsequent PR could make the new behavior the default.

What do you think?

rafaelcr · 2024-12-16T21:41:59Z

Great idea @jcnelson and thanks for the feedback! I'll get on it.

jcnelson · 2024-12-16T21:49:30Z

stackslib/src/chainstate/stacks/tests/block_construction.rs

+    let mut tx_events = Vec::new();
+
+    // Submit nonces 0 through 9 for each of the 3 senders.
+    for nonce in 0..10 {


This is only testing unsponsored transactions. You are going to want to expand this to test sponsored transactions.

In particular, you'll want to test the case where a fixed number of sponsors send a wide variety of sponsored transactions (much like how e.g. a BNS registrar would), and where both the sponsors and sponsored accounts also send their own unsponsored transactions. In addition to testing that the transaction frontier is ordered from highest to lowest by fee, you'd test that the set of subsequence of sponsored transactions present in considered_txs are exactly the ones with both the next nonces for its origin and sponsor accounts. In other words, a heterogenous mempool with unsponsored and sponsored transactions where origin and sponsor accounts can each have in-flight unsponsored and sponsored transactions still produces the desired priority list of mineable transactions.

Nice. Will do. That sounds like a great test.

jcnelson · 2024-12-16T21:49:55Z

stackslib/src/core/mempool.rs

-            WHERE fee_rate IS NOT NULL
-            ORDER BY fee_rate DESC
+            FROM address_nonce_ranked
+            ORDER BY rank ASC, sort_fee_rate DESC


Before merging, can you manually confirm that this query does not do any table scans via EXPLAIN QUERY PLAN?

Sure. I've been monitoring that and will add any indexes necessary to avoid table scans.

rafaelcr added 2 commits December 7, 2024 11:51

fix: new query

b0705d0

chore: remove dev comment

a0600b6

rafaelcr added 5 commits December 11, 2024 10:10

style: lint fixes

3719188

fix: add simulated fee rates for null

2cd116f

fix: indexes

685924c

fix: remove tx_fee column

e1dac9d

Merge branch 'fix/mempool-query' of github.com:rafaelcr/stacks-core i…

c587c53

…nto fix/mempool-query

rafaelcr commented Dec 13, 2024

View reviewed changes

stackslib/src/core/mempool.rs Show resolved Hide resolved

rafaelcr added 2 commits December 14, 2024 18:04

test: correct tx order

dd9729c

Merge branch 'develop' into fix/mempool-query

2406d25

rafaelcr changed the title ~~(draft) fix: update miner mempool transaction iterator query to consider fee rate and nonce~~ fix: update miner mempool transaction iterator query to consider fee rate and nonce Dec 15, 2024

rafaelcr marked this pull request as ready for review December 15, 2024 00:09

rafaelcr requested a review from a team as a code owner December 15, 2024 00:09

rafaelcr changed the title ~~fix: update miner mempool transaction iterator query to consider fee rate and nonce~~ fix: update miner mempool iterator query to consider both nonces and fee rates Dec 15, 2024

obycode reviewed Dec 16, 2024

View reviewed changes

obycode added this to the 3.1.0.0.3 milestone Dec 16, 2024

rafaelcr added 3 commits December 16, 2024 12:39

fix: nonce ordering

7702f67

Merge branch 'fix/mempool-query' of github.com:rafaelcr/stacks-core i…

89d1078

…nto fix/mempool-query

fix: remove now unneccessary index

d0d1f8d

obycode self-requested a review December 16, 2024 20:32

obycode reviewed Dec 16, 2024

View reviewed changes

jcnelson reviewed Dec 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: update miner mempool iterator query to consider both nonces and fee rates #5541

fix: update miner mempool iterator query to consider both nonces and fee rates #5541

rafaelcr commented Dec 7, 2024 •

edited

Loading

CLAassistant commented Dec 7, 2024

CLAassistant commented Dec 7, 2024

obycode Dec 16, 2024

rafaelcr Dec 16, 2024

obycode Dec 16, 2024

rafaelcr Dec 16, 2024

obycode Dec 16, 2024

obycode Dec 16, 2024

rafaelcr Dec 16, 2024

obycode Dec 16, 2024

jcnelson Dec 16, 2024

jcnelson Dec 16, 2024

jcnelson Dec 16, 2024

jcnelson Dec 16, 2024

jcnelson left a comment

rafaelcr commented Dec 16, 2024

jcnelson Dec 16, 2024

rafaelcr Dec 16, 2024

jcnelson Dec 16, 2024

rafaelcr Dec 16, 2024

		LEFT JOIN nonces ON mempool.origin_address = nonces.address
		WHERE nonces.address IS NULL OR mempool.origin_nonce >= nonces.nonce

fix: update miner mempool iterator query to consider both nonces and fee rates #5541

Are you sure you want to change the base?

fix: update miner mempool iterator query to consider both nonces and fee rates #5541

Conversation

rafaelcr commented Dec 7, 2024 • edited Loading

Problem

Solution

CLAassistant commented Dec 7, 2024

CLAassistant commented Dec 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcnelson left a comment

Choose a reason for hiding this comment

rafaelcr commented Dec 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rafaelcr commented Dec 7, 2024 •

edited

Loading