Introduce support for tablets #137

sylwiaszunejko · 2023-07-20T11:53:00Z

This PR introduces changes to the driver that are necessary for shard-awareness and token-awareness to work effectively with the tablets.

Now if driver send the request to the wrong node/shard it will get the correct tablet information from Scylla in custom_payload. It uses this information to obtain target replicas and shard numbers for tables managed by tablet replication.

I also added parsing TABLETS_ROUTING_V1 extension. In order for Scylla to send the tablet info, the driver must tell the database during connection handshake that it is able to interpret it. It was implemented similarly to the already-present LWT optimization extension negotiation.

The metadata for tablets are provided to the user.

To test this change I created both integration and unit tests.

Fixes: #155

avikivity · 2023-07-20T14:53:41Z

Please add an experimental setting and require the user to enable it. Tablets are still experimental in ScyllaDB and this allows us to change the tablets table format, but if drivers start to rely on it, then we cannot.

sylwiaszunejko · 2023-07-24T13:05:03Z

Please add an experimental setting and require the user to enable it. Tablets are still experimental in ScyllaDB and this allows us to change the tablets table format, but if drivers start to rely on it, then we cannot.

Added an experimental setting and corrected .github/workflows/main.yml to make Checks passed.

avikivity · 2023-07-24T13:35:24Z

Unfortunately we'll need to add such a flag to scylla-bench and plumb it down, but that's the price of compatibility.

policies.go

cluster.go

host_source.go

avelanarius · 2023-07-27T09:36:41Z

host_source.go

+func findTabletForToken(tablets []*TabletInfo, token token, l int) *TabletInfo {
+	r := l + tablets[l].TabletCount() - 1
+	for l < r {
+		m := (l + r) / 2


Won't this l+r overflow is l, r are big?

It could happen, I changed it a bit to prevent that

Isn't m := l + (r-l)/2 still incorrect? For:

l, r - signed 64-bit ints r = 2^63 - 5 l = -(2^63-5) r - l = 2^63 - 5 - (-(2^63-5)) r - l = 2^63 - 5 + 2^63 - 5 r - l = 2^64 - 10 > range of 64-bit signed int

(some other context: https://stackoverflow.com/questions/19106350/explanation-of-the-safe-average-of-two-numbers)

It's interesting that the tests didn't catch the problem, so either they are not sufficient enough to catch the bug or this code is actually correct for some reason.

yeah it is incorrect it only solves the problem for two positive or two negative variables, I think that the tests are not sufficient enough to catch this bug

host_source.go

tablet_test.go

policies.go

scylla.go

avelanarius · 2023-07-28T10:22:06Z

tablet_integration_test.go

+			}
+		}
+
+		assertEqual(t, "queriedHosts", 1, queriedHosts)


Please add a (temporary?) print here of all hostAddresses (before strings.Contains filter) so you can debug the failing CI.

bhalevy · 2023-12-22T10:02:37Z

I opened #155 as an umbrella issue

bhalevy · 2023-12-28T14:38:23Z

@sylwiaszunejko when do you plan to get back to this PR?

sylwiaszunejko · 2023-12-28T15:28:46Z

@sylwiaszunejko when do you plan to get back to this PR?

@bhalevy It is ready for review, but @avelanarius is on PTO, and it has to wait until he is back.

sylwiaszunejko · 2024-01-04T08:26:33Z

@avelanarius I adapted the code to recent changes in the core. Now all tablet info are sent under one key in custom_payload and driver has to inform Scylla that it is able to interpret tablet info during connection handshake.

avikivity · 2024-01-04T10:49:59Z

Please add an experimental setting and require the user to enable it. Tablets are still experimental in ScyllaDB and this allows us to change the tablets table format, but if drivers start to rely on it, then we cannot.

Added an experimental setting and corrected .github/workflows/main.yml to make Checks passed.

With the negotiation thing, this is no longer needed.

Also, the cover letter is outdated.

sylwiaszunejko · 2024-01-04T10:57:17Z

Please add an experimental setting and require the user to enable it. Tablets are still experimental in ScyllaDB and this allows us to change the tablets table format, but if drivers start to rely on it, then we cannot.

Added an experimental setting and corrected .github/workflows/main.yml to make Checks passed.

With the negotiation thing, this is no longer needed.

Also, the cover letter is outdated.

Do I understand correctly that I should delete experimental setting and replace it with checking if ScyllaDB supports TABLETS_ROUTING_V1?

avikivity · 2024-01-04T10:58:35Z

Please add an experimental setting and require the user to enable it. Tablets are still experimental in ScyllaDB and this allows us to change the tablets table format, but if drivers start to rely on it, then we cannot.

Added an experimental setting and corrected .github/workflows/main.yml to make Checks passed.

With the negotiation thing, this is no longer needed.
Also, the cover letter is outdated.

Do I understand correctly that I should delete experimental setting and replace it with checking if ScyllaDB supports TABLETS_ROUTING_V1?

Yes; you must check for it anyway, otherwise the server won't send routing info.

In order for Scylla to send the tablet info, the driver must tell the database during connection handshake that it is able to interpret it. It was implemented similarly to the already-present LWT optimisation extension negotiation.

Add mechanism to parse system.tablets periodically. In TokenAwareHostPolicy check if keyspace uses tablets if so try to use them to find replicas. Make shard awareness work when using tablets.

tablet_integration_test.go: Integration tests checking if TokenAwareHostPolicy and shard awareness works correctly when using tablets, and if adding new table changes tablets table. tablet_test.go: Unit tests checking if searching in tablets list works correctly.

sylwiaszunejko · 2024-01-08T12:55:08Z

experimental setting removed

piodul · 2024-02-02T10:02:43Z

scylla.go

-	idx := p.shardOf(mmt)
+	idx := -1
+
+	p.conns[0].mu.Lock()


@sylwiaszunejko This code seems to assume that p.conns[0] is never nil. I don't think this is always true. AFAIK p.conns is indexed by the shard of the connection. It may temporarily happen that there is no connection to shard 0 for some time (e.g. the first connection to the node landed on a different shard than 0).

I think this might be the cause of scylladb/scylla-bench#134. You can try reproducing it by writing with a high concurrency and shutting down a node or starting one. The node should have more than one shard set up (the more, the easier to reproduce).

You are probably right about the root cause of this issue, in this part of the code I needed to take any connection just to check if tabletsRoutingV1 is true, but taking p.conns[0] was definitely not the right approach to do that. I will try to confirm that it is true and fix it. Thank you for your feedback

@piodul I created PR to fix this comment: #158

sylwiaszunejko force-pushed the poll_system_tablets branch from 1d791a9 to d0d5b66 Compare July 20, 2023 12:00

sylwiaszunejko requested a review from avelanarius July 21, 2023 07:56

sylwiaszunejko force-pushed the poll_system_tablets branch 4 times, most recently from d8bf274 to b217867 Compare July 24, 2023 12:44

sylwiaszunejko force-pushed the poll_system_tablets branch 6 times, most recently from 4fa94d0 to 9bb1f52 Compare July 25, 2023 07:58

wprzytula reviewed Jul 26, 2023

View reviewed changes

policies.go Outdated Show resolved Hide resolved