Skip to content

Commit

Permalink
Merge pull request apache#34009 from strongduanmu/dev-12112
Browse files Browse the repository at this point in the history 
Add load-table-metadata-batch-size props to concurrent load table metadata
  • Loading branch information
@iamhucong
iamhucong authored Dec 11, 2024
2 parents 4b47c41 + bbc75db commit 278c6cc
Show file tree
Hide file tree
Showing 11 changed files with 66 additions and 32 deletions.
1 change: 1 addition & 0 deletions RELEASE-NOTES.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
1. Sharding: Support GroupConcat function for aggregating multiple shards in MySQL, OpenGauss, Doris - [#33808](https://github.com/apache/shardingsphere/pull/33808)
1. Proxy Native: Support Seata AT integration under Proxy Native in GraalVM Native Image - [#33889](https://github.com/apache/shardingsphere/pull/33889)
1. Agent: Simplify the use of Agent's Docker Image - [#33356](https://github.com/apache/shardingsphere/pull/33356)
1. Metadata: Add load-table-metadata-batch-size props to concurrent load table metadata - [#34009](https://github.com/apache/shardingsphere/pull/34009)

### Bug Fixes

Expand Down
1 change: 1 addition & 0 deletions docs/document/content/user-manual/common-config/props.cn.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ Apache ShardingSphere 提供属性配置的方式配置系统级配置。
| kernel-executor-size (?) | int | 用于设置任务处理线程池的大小<br />每个 ShardingSphereDataSource 使用一个独立的线程池，同一个 JVM 的不同数据源不共享线程池 | infinite |
| max-connections-size-per-query (?) | int | 一次查询请求在每个数据库实例中所能使用的最大连接数 | 1 |
| check-table-metadata-enabled (?) | boolean | 在程序启动和更新时，是否检查分片元数据的结构一致性 | false |
| load-table-metadata-batch-size (?) | int | 在程序启动或刷新元数据时，单个批次加载表元数据的数量 | 1000 |

## 操作步骤

Expand Down
1 change: 1 addition & 0 deletions docs/document/content/user-manual/common-config/props.en.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ Apache ShardingSphere provides the way of property configuration to configure sy
| kernel-executor-size (?) | int | The max thread size of worker group to execute SQL. One ShardingSphereDataSource will use a independent thread pool, it does not share thread pool even different data source in same JVM | infinite |
| max-connections-size-per-query (?) | int | Max opened connection size for each query | 1 |
| check-table-metadata-enabled (?) | boolean | Whether validate table meta data consistency when application startup or updated | false |
| load-table-metadata-batch-size (?) | int | The number of table metadata loaded at a time when application startup or refreshes table metadata | 1000 |

## Procedure

Expand Down
37 changes: 19 additions & 18 deletions docs/document/content/user-manual/shardingsphere-proxy/yaml-config/props.cn.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,25 @@ Apache ShardingSphere 提供了丰富的系统配置属性，用户可通过 `gl

## 参数解释

| *名称* | *数据类型* | *说明* | *默认值* | *动态生效* |
|-------------------------------------------|-----------|----------------------------------------------------------------------------------------------------------------------------------------|-----------------|--------|
| system-log-level (?) | String | 系统日志输出级别，支持 DEBUG、INFO、WARN 和 ERROR，默认级别是 INFO。 | INFO | 是 |
| sql-show (?) | boolean | 是否在日志中打印 SQL。 <br /> 打印 SQL 可以帮助开发者快速定位系统问题。日志内容包含：逻辑 SQL，真实 SQL 和 SQL 解析结果。<br /> 如果开启配置，日志将使用 Topic `ShardingSphere-SQL`，日志级别是 INFO。 | false | 是 |
| sql-simple (?) | boolean | 是否在日志中打印简单风格的 SQL。 | false | 是 |
| kernel-executor-size (?) | int | 用于设置任务处理线程池的大小。每个 ShardingSphereDataSource 使用一个独立的线程池，同一个 JVM 的不同数据源不共享线程池。 | infinite | 否 |
| max-connections-size-per-query (?) | int | 一次查询请求在每个数据库实例中所能使用的最大连接数。 | 1 | 是 |
| check-table-metadata-enabled (?) | boolean | 在程序启动和更新时，是否检查分片元数据的结构一致性。 | false | 是 |
| proxy-frontend-flush-threshold (?) | int | 在 ShardingSphere-Proxy 中设置传输数据条数的 IO 刷新阈值。 | 128 | 是 |
| proxy-backend-query-fetch-size (?) | int | Proxy 后端与数据库交互的每次获取数据行数（使用游标的情况下）。数值增大可能会增加 ShardingSphere Proxy 的内存使用。默认值为 -1，代表设置为 JDBC 驱动的最小值。 | -1 | 是 |
| proxy-frontend-executor-size (?) | int | Proxy 前端 Netty 线程池线程数量，默认值 0 代表使用 Netty 默认值。 | 0 | 否 |
| proxy-frontend-max-connections (?) | int | 允许连接 Proxy 的最大客户端数量，默认值 0 代表不限制。 | 0 | 是 |
| proxy-default-port (?) | String | Proxy 通过配置文件指定默认端口。 | 3307 | 否 |
| proxy-netty-backlog (?) | int | Proxy 通过配置文件指定默认netty back_log参数。 | 1024 | 否 |
| proxy-frontend-database-protocol-type (?) | String | Proxy 前端协议类型，支持 MySQL，PostgreSQL 和 openGauss | \"\" | 否 |
| proxy-frontend-ssl-enabled (?) | boolean | Proxy 前端启用 SSL/TLS。 | false | 否 |
| proxy-frontend-ssl-version (?) | String | 要启用的 SSL/TLS 协议。空白以使用默认值。 | TLSv1.2,TLSv1.3 | 否 |
| proxy-frontend-ssl-cipher (?) | String | 按偏好顺序启用的密码套件。用逗号分隔的多密码套件。空白以使用默认值。 | \"\" | 否 |
| *名称* | *数据类型* | *说明* | *默认值* | *动态生效* |
|-------------------------------------------|---------|----------------------------------------------------------------------------------------------------------------------------------------|-----------------|--------|
| system-log-level (?) | String | 系统日志输出级别，支持 DEBUG、INFO、WARN 和 ERROR，默认级别是 INFO。 | INFO | 是 |
| sql-show (?) | boolean | 是否在日志中打印 SQL。 <br /> 打印 SQL 可以帮助开发者快速定位系统问题。日志内容包含：逻辑 SQL，真实 SQL 和 SQL 解析结果。<br /> 如果开启配置，日志将使用 Topic `ShardingSphere-SQL`，日志级别是 INFO。 | false | 是 |
| sql-simple (?) | boolean | 是否在日志中打印简单风格的 SQL。 | false | 是 |
| kernel-executor-size (?) | int | 用于设置任务处理线程池的大小。每个 ShardingSphereDataSource 使用一个独立的线程池，同一个 JVM 的不同数据源不共享线程池。 | infinite | 否 |
| max-connections-size-per-query (?) | int | 一次查询请求在每个数据库实例中所能使用的最大连接数。 | 1 | 是 |
| check-table-metadata-enabled (?) | boolean | 在程序启动和更新时，是否检查分片元数据的结构一致性。 | false | 是 |
| load-table-metadata-batch-size (?) | int | 在程序启动或刷新元数据时，单个批次加载表元数据的数量。 | 1000 | 是 |
| proxy-frontend-flush-threshold (?) | int | 在 ShardingSphere-Proxy 中设置传输数据条数的 IO 刷新阈值。 | 128 | 是 |
| proxy-backend-query-fetch-size (?) | int | Proxy 后端与数据库交互的每次获取数据行数（使用游标的情况下）。数值增大可能会增加 ShardingSphere Proxy 的内存使用。默认值为 -1，代表设置为 JDBC 驱动的最小值。 | -1 | 是 |
| proxy-frontend-executor-size (?) | int | Proxy 前端 Netty 线程池线程数量，默认值 0 代表使用 Netty 默认值。 | 0 | 否 |
| proxy-frontend-max-connections (?) | int | 允许连接 Proxy 的最大客户端数量，默认值 0 代表不限制。 | 0 | 是 |
| proxy-default-port (?) | String | Proxy 通过配置文件指定默认端口。 | 3307 | 否 |
| proxy-netty-backlog (?) | int | Proxy 通过配置文件指定默认netty back_log参数。 | 1024 | 否 |
| proxy-frontend-database-protocol-type (?) | String | Proxy 前端协议类型，支持 MySQL，PostgreSQL 和 openGauss | \"\" | 否 |
| proxy-frontend-ssl-enabled (?) | boolean | Proxy 前端启用 SSL/TLS。 | false | 否 |
| proxy-frontend-ssl-version (?) | String | 要启用的 SSL/TLS 协议。空白以使用默认值。 | TLSv1.2,TLSv1.3 | 否 |
| proxy-frontend-ssl-cipher (?) | String | 按偏好顺序启用的密码套件。用逗号分隔的多密码套件。空白以使用默认值。 | \"\" | 否 |

属性配置可以通过 [DistSQL#RAL](/cn/user-manual/shardingsphere-proxy/distsql/syntax/ral/) 在线修改。
其中支持动态修改的属性立即生效，不支持动态修改的属性在重启后生效。
Expand Down
1 change: 1 addition & 0 deletions docs/document/content/user-manual/shardingsphere-proxy/yaml-config/props.en.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ Apache ShardingSphere provides a wealth of system configuration properties, whic
| kernel-executor-size (?) | int | Set the size of the thread pool for task processing. Each ShardingSphereDataSource uses an independent thread pool, and different data sources on the same JVM do not share thread pools. | infinite | False |
| max-connections-size-per-query (?) | int | The maximum number of connections that a query request can use in each database instance. | 1 | True |
| check-table-metadata-enabled (?) | boolean | Whether shard metadata is checked for structural consistency when the program is started and updated. | false | True |
| load-table-metadata-batch-size (?) | int | The number of table metadata loaded at a time when application startup or refreshes table metadata. | 1000 | True |
| proxy-frontend-flush-threshold (?) | int | Set the I/O refresh threshold for the number of transmitted data items in ShardingSphere-Proxy. | 128 | True |
| proxy-backend-query-fetch-size (?) | int | The number of rows of data obtained when the backend Proxy interacts with databases (using a cursor). A larger number may increase the occupied memory of ShardingSphere-Proxy. The default value of -1 indicates the minimum value for JDBC driver. | -1 | True |
| proxy-frontend-executor-size (?) | int | The number of threads in the Netty thread pool of front-end Proxy. | 0 | False |
Expand Down
5 changes: 5 additions & 0 deletions .../src/main/java/org/apache/shardingsphere/infra/config/props/ConfigurationPropertyKey.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@ public enum ConfigurationPropertyKey implements TypedPropertyKey {
*/
CHECK_TABLE_METADATA_ENABLED("check-table-metadata-enabled", String.valueOf(Boolean.FALSE), boolean.class, false),

/**
* Load table metadata batch size.
*/
LOAD_TABLE_METADATA_BATCH_SIZE("load-table-metadata-batch-size", String.valueOf(1000), int.class, false),

/**
* Frontend database protocol for ShardingSphere-Proxy.
*/
Expand Down
4 changes: 1 addition & 3 deletions ...rg/apache/shardingsphere/infra/metadata/database/schema/builder/GenericSchemaBuilder.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@

import lombok.AccessLevel;
import lombok.NoArgsConstructor;
import org.apache.shardingsphere.infra.config.props.ConfigurationPropertyKey;
import org.apache.shardingsphere.infra.database.core.metadata.data.loader.MetaDataLoader;
import org.apache.shardingsphere.infra.database.core.metadata.data.loader.MetaDataLoaderMaterial;
import org.apache.shardingsphere.infra.database.core.metadata.data.model.SchemaMetaData;
Expand Down Expand Up @@ -87,8 +86,7 @@ private static Collection<String> getAllTableNames(final Collection<ShardingSphe
}

private static Map<String, SchemaMetaData> loadSchemas(final Collection<String> tableNames, final GenericSchemaBuilderMaterial material) throws SQLException {
boolean checkMetaDataEnable = material.getProps().getValue(ConfigurationPropertyKey.CHECK_TABLE_METADATA_ENABLED);
Collection<MetaDataLoaderMaterial> materials = SchemaMetaDataUtils.getMetaDataLoaderMaterials(tableNames, material, checkMetaDataEnable);
Collection<MetaDataLoaderMaterial> materials = SchemaMetaDataUtils.getMetaDataLoaderMaterials(tableNames, material);
return materials.isEmpty() ? Collections.emptyMap() : MetaDataLoader.load(materials);
}

Expand Down
Loading

0 comments on commit 278c6cc

Please sign in to comment.