diff --git a/changelog/21.0/21.0.0/changelog.md b/changelog/21.0/21.0.0/changelog.md new file mode 100644 index 00000000000..c74c1b2a1f1 --- /dev/null +++ b/changelog/21.0/21.0.0/changelog.md @@ -0,0 +1,2 @@ +# Changelog of Vitess v21.0.0 + diff --git a/changelog/21.0/21.0.0/release_notes.md b/changelog/21.0/21.0.0/release_notes.md new file mode 100644 index 00000000000..fbd9964bdef --- /dev/null +++ b/changelog/21.0/21.0.0/release_notes.md @@ -0,0 +1,222 @@ +# Release of Vitess v21.0.0 +## Summary + +### Table of Contents + +- **[Major Changes](#major-changes)** + - **[Deprecations and Deletions](#deprecations-and-deletions)** + - [Deprecated VTTablet Flags](#vttablet-flags) + - [Deletion of deprecated metrics](#metric-deletion) + - [Deprecated Metrics](#deprecations-metrics) + - **[Traffic Mirroring](#traffic-mirroring)** + - **[New VTGate Shutdown Behavior](#new-vtgate-shutdown-behavior)** + - **[Tablet Throttler: Multi-Metric support](#tablet-throttler)** + - **[Allow Cross Cell Promotion in PRS](#allow-cross-cell)** + - **[Support for recursive CTEs](#recursive-cte)** + - **[VTGate Tablet Balancer](#tablet-balancer)** + - **[Query Timeout Override](#query-timeout)** + - **[New Backup Engine](#new-backup-engine)** + - **[Dynamic VReplication Configuration](#dynamic-vreplication-configuration)** + - **[Reference Table Materialization](#reference-table-materialization)** + - **[New VEXPLAIN Modes: TRACE and KEYS](#new-vexplain-modes)** + - **[Errant GTID Detection on VTTablets](#errant-gtid-vttablet)** + - **[Automatically Replace MySQL auto_increment Clauses with Vitess Sequences](#auto-replace-mysql-autoinc-with-seq)** + +## Major Changes + +### Deprecations and Deletions + +#### Deprecated VTTablet Flags + +- `queryserver-enable-settings-pool` flag, added in `v15`, has been on by default since `v17`. + It is now deprecated and will be removed in a future release. + +#### Deletion of deprecated metrics + +The following VTOrc metrics were deprecated in `v20`. They have now been deleted. + +| Metric Name | +|:--------------------------------------------:| +| `analysis.change.write` | +| `audit.write` | +| `discoveries.attempt` | +| `discoveries.fail` | +| `discoveries.instance_poll_seconds_exceeded` | +| `discoveries.queue_length` | +| `discoveries.recent_count` | +| `instance.read` | +| `instance.read_topology` | +| `emergency_reparent_counts` | +| `planned_reparent_counts` | +| `reparent_shard_operation_timings` | + +#### Deprecated Metrics + +The following metrics are now deprecated and will be deleted in a future release, please use their replacements. + +| Component | Metric Name | Replaced By | +|------------|:---------------------:|:-------------------------------:| +| `vttablet` | `QueryCacheLength` | `QueryEnginePlanCacheLength` | +| `vttablet` | `QueryCacheSize` | `QueryEnginePlanCacheSize` | +| `vttablet` | `QueryCacheCapacity` | `QueryEnginePlanCacheCapacity` | +| `vttablet` | `QueryCacheEvictions` | `QueryEnginePlanCacheEvictions` | +| `vttablet` | `QueryCacheHits` | `QueryEnginePlanCacheHits` | +| `vttablet` | `QueryCacheMisses` | `QueryEnginePlanCacheMisses` | + +### Traffic Mirroring + +Traffic mirroring is intended to help reduce some of the uncertainty inherent to `MoveTables SwitchTraffic`. When +traffic mirroring is enabled, VTGate will mirror a percentage of traffic from one keyspace to another. + +Mirror rules may be enabled through `vtctldclient` with `MoveTables MirrorTraffic`. For example: + +```bash +$ vtctldclient --server :15999 MoveTables --target-keyspace customer --workflow commerce2customer MirrorTraffic --percent 5.0 +``` + +Mirror rules can be inspected with `GetMirrorRules`. + +### New VTGate Shutdown Behavior + +We added a new option to VTGate to disallow new connections while VTGate is shutting down, +while allowing existing connections to finish their work until they manually disconnect or until +the `--onterm_timeout` is reached, without getting a `Server shutdown in progress` error. + +This new behavior can be enabled by specifying the new `--mysql-server-drain-onterm` flag to VTGate. + +You can find more information about this option in the [RFC](https://github.com/vitessio/vitess/issues/15971). + +### Tablet Throttler: Multi-Metric support + +Up until `v20`, the tablet throttler would only monitor and use a single metric. That would be replication lag, by +default, or could be the result of a custom query. In this release, we introduce a major redesign so that the throttler +monitors and uses multiple metrics at the same time, including the above two. + +The default behavior now is to monitor all metrics, but only use `lag` (if the custom query is undefined) or the `custom` +metric (if the custom query is defined). This is backwards-compatible with `v20`. A `v20` `PRIMARY` is compatible with +a `v21` `REPLICA`, and a `v21` `PRIMARY` is compatible with a `v20` `REPLICA`. + +However, it is now possible to assign any combination of one or more metrics for a given app. The throttler +would then accept or reject the app's requests based on the health of _all_ assigned metrics. We have provided a pre-defined +list of metrics: + +- `lag`: replication lag based on heartbeat injection. +- `threads_running`: concurrent active threads on the MySQL server. +- `loadavg`: per core load average measured on the tablet instance/pod. +- `custom`: the result of a custom query executed on the MySQL server. + +Each metric has a default threshold which can be overridden by the `UpdateThrottlerConfig` command. + +The throttler also supports the catch-all `"all"` app name, and it is thus possible to assign metrics to **all** apps. +Explicit app to metric assignments will override the catch-all configuration. + +Metrics are assigned a default _scope_, which could be `self` (isolated to the tablet) or `shard` (max, aka **worst** +value among shard tablets). It is further possible to require a different scope for each metric. + +### Allow Cross Cell Promotion in PRS + +Up until now if the users wanted to promote a replica in a different cell from the current primary +using `PlannedReparentShard`, they had to specify the new primary with the `--new-primary` flag. + +We have now added a new flag `--allow-cross-cell-promotion` that lets `PlannedReparentShard` choose a primary in a +different cell even if no new primary is provided explicitly. + +### Experimental support for recursive CTEs + +We have added experimental support for recursive CTEs in Vitess. We are marking it as experimental because it is not yet +fully tested and may have some limitations. We are looking for feedback from the community to improve this feature. + +### VTGate Tablet Balancer + +When a VTGate routes a query and has multiple available tablets for a given shard / tablet type (e.g. REPLICA), the +current default behavior routes the query with local cell affinity and round robin policy. The VTGate Tablet Balancer +provides an alternate mechanism that routes queries to maintain an even distribution of query load to each tablet, while +preferentially routing to tablets in the same cell as the VTGate. + +The tablet balancer is enabled by a new flag `--enable-balancer` and configured by `--balancer-vtgate-cells` +and `--balancer-keyspaces`. + +See the [RFC ](https://github.com/vitessio/vitess/issues/12241) for more details on the design and configuration of this feature. + +### Query Timeout Override + +VTGate sends an authoritative query timeout to VTTablet when the `QUERY_TIMEOUT_MS` comment directive, +`query_timeout` session system variable, or `query-timeout` flag is set. +The order of precedence is: comment directive > session variable > VTGate flag. +VTTablet overrides its default query timeout with the value received from VTGate. +All timeouts are specified in milliseconds. + +When a query is executed inside a transaction, there is an additional nuance. The actual timeout used will be the smaller +of the transaction timeout and the query timeout. + +A query can also be set to have no timeout by using the `QUERY_TIMEOUT_MS` comment directive with a value of `0`. + +Example usage: +`select /*vt+ QUERY_TIMEOUT_MS=30 */ col from tbl` + +### New Backup Engine (EXPERIMENTAL) + +We are introducing a new backup engine for logical backups in order to support use cases that require something other +than physical backups. This feature is experimental and is based on [MySQL Shell](https://dev.mysql.com/doc/mysql-shell/8.0/en/). + +The new engine is enabled by using `--backup_engine_implementation=mysqlshell`. There are other options that are required, +so please read the [documentation](https://vitess.io/docs/21.0/user-guides/operating-vitess/backup-and-restore/creating-a-backup/) to learn which options are required and how to configure them. + +### Dynamic VReplication Configuration + +Previously, many of the configuration options for VReplication Workflows had to be provided using VTTablet flags. This +meant that any change to VReplication configuration required restarting VTTablets. We now allow these to be overridden +while creating a workflow or dynamically after the workflow is already in progress. + +### Reference Table Materialization + +There is a new option in [`Materialize` workflows](https://vitess.io/docs/reference/vreplication/materialize/) to keep a synced copy of [reference or lookup tables](https://vitess.io/docs/reference/vreplication/reference_tables/) +(countries, states, zip codes, etc) from an unsharded keyspace, which holds the source of truth for the reference +table, to all shards in a sharded keyspace. + +### New VEXPLAIN Modes: TRACE and KEYS + +#### VEXPLAIN TRACE + +The new `TRACE` mode for `VEXPLAIN` provides a detailed execution trace of queries, showing how they're processed through various +operators and interactions with tablets. This mode is particularly useful for: + +- Identifying performance bottlenecks +- Understanding query execution patterns +- Optimizing complex queries +- Debugging unexpected query behavior + +`TRACE` mode runs the query and logs all interactions, returning a JSON representation of the query execution plan with additional +statistics like number of calls, average rows processed, and number of shards queried. + +#### VEXPLAIN KEYS + +The `KEYS` mode for `VEXPLAIN` offers a concise summary of query structure, highlighting columns used in joins, filters, and +grouping operations. This information is crucial for: + +- Identifying potential sharding key candidates +- Optimizing query performance +- Analyzing query patterns to inform database design decisions + +`KEYS` mode analyzes the query structure without executing it, providing JSON output that includes grouping columns, join columns, +filter columns (potential candidates for indexes, primary keys, or sharding keys), and the statement type. + +These new `VEXPLAIN` modes enhance Vitess's query analysis capabilities, allowing for more informed decisions about sharding +strategies and query optimization. + +### Errant GTID Detection on VTTablets + +VTTablets now run an errant GTID detection logic before they join the replication stream. So, if a replica has an errant GTID, it will +not start replicating from the primary. This protects us from running into situations which are very difficult to recover from. + +For users running with the vitess-operator on Kubernetes, this change means that replica tablets with errant GTIDs will have broken +replication and will report as unready. Users will need to manually replace and clean up these errant replica tablets. + +### Automatically Replace MySQL auto_increment Clauses with Vitess Sequences + +In https://github.com/vitessio/vitess/pull/16860 we added support for replacing MySQL `auto_increment` clauses with [Vitess Sequences](https://vitess.io/docs/reference/features/vitess-sequences/), performing all of the setup and initialization +work automatically during the [`MoveTables`](https://vitess.io/docs/reference/vreplication/movetables/) workflow. As part of that work we have deprecated the +[`--remove-sharded-auto-increment` boolean flag](https://vitess.io/docs/20.0/reference/programs/vtctldclient/vtctldclient_movetables/vtctldclient_movetables_create/) and you should begin using the new +[`--sharded-auto-increment-handling` flag](https://vitess.io/docs/21.0/reference/programs/vtctldclient/vtctldclient_movetables/vtctldclient_movetables_create/) instead. Please see the new +[`MoveTables` Auto Increment Handling](https://vitess.io/docs/21.0/reference/vreplication/movetables/#auto-increment-handling) documentation for additional details. + diff --git a/changelog/21.0/README.md b/changelog/21.0/README.md index bade1b597f8..a77e98bcaba 100644 --- a/changelog/21.0/README.md +++ b/changelog/21.0/README.md @@ -1,2 +1,4 @@ ## v21.0 * **[21.0.0](21.0.0)** + * [Changelog](21.0.0/changelog.md) + * [Release Notes](21.0.0/release_notes.md)