Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clickhouse: new database engine (experimental) #1094

Closed
wants to merge 11 commits into from
Closed

Clickhouse: new database engine (experimental) #1094

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
5ee90ab
Clickhouse interface
May 2, 2022
626f89e
Add configuration support for Clickhouse
Apr 19, 2022
9efbc69
Configure Clickhouse backend for testing
May 2, 2022
eee3390
Fix unit test to support Clickhouse
May 2, 2022
1a1bd76
Specific Clickhouse queries
Nov 30, 2023
3569d0f
Check job state before ALTER UPDATE
Nov 30, 2023
101f6b3
Cast the batch_id to undef (NULL)
Nov 30, 2023
94c91a6
Manually check for user uniqueness in add_api_user
May 16, 2022
7eb4418
Insert batch in one pass
May 4, 2022
981ceb0
Insert batch data in bundle
May 17, 2022
b58a5d1
Remove left-over note
Apr 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions MANIFEST
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ lib/Zonemaster/Backend.pm
lib/Zonemaster/Backend/Config.pm
lib/Zonemaster/Backend/Config/DCPlugin.pm
lib/Zonemaster/Backend/DB.pm
lib/Zonemaster/Backend/DB/Clickhouse.pm
lib/Zonemaster/Backend/DB/MySQL.pm
lib/Zonemaster/Backend/DB/PostgreSQL.pm
lib/Zonemaster/Backend/DB/SQLite.pm
Expand Down
86 changes: 82 additions & 4 deletions lib/Zonemaster/Backend/Config.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ sub parse {

# Validate section names
{
my %sections = map { $_ => 1 } ( 'DB', 'MYSQL', 'POSTGRESQL', 'SQLITE', 'LANGUAGE', 'PUBLIC PROFILES', 'PRIVATE PROFILES', 'ZONEMASTER', 'METRICS', 'RPCAPI' );
my %sections = map { $_ => 1 } ( 'DB', 'MYSQL', 'POSTGRESQL', 'SQLITE', 'CLICKHOUSE', 'LANGUAGE', 'PUBLIC PROFILES', 'PRIVATE PROFILES', 'ZONEMASTER', 'METRICS', 'RPCAPI' );
for my $section ( $ini->Sections ) {
if ( !exists $sections{$section} ) {
die "config: unrecognized section: $section\n";
Expand All @@ -210,6 +210,7 @@ sub parse {
$obj->_set_DB_polling_interval( '0.5' );
$obj->_set_MYSQL_port( '3306' );
$obj->_set_POSTGRESQL_port( '5432' );
$obj->_set_CLICKHOUSE_port( '9004' );
$obj->_set_ZONEMASTER_max_zonemaster_execution_time( '600' );
$obj->_set_ZONEMASTER_number_of_processes_for_frontend_testing( '20' );
$obj->_set_ZONEMASTER_number_of_processes_for_batch_testing( '20' );
Expand Down Expand Up @@ -277,6 +278,21 @@ sub parse {
if ( defined( my $value = $get_and_clear->( 'SQLITE', 'database_file' ) ) ) {
$obj->_set_SQLITE_database_file( $value );
}
if ( defined( my $value = $get_and_clear->( 'CLICKHOUSE', 'host' ) ) ) {
$obj->_set_CLICKHOUSE_host( $value );
}
if ( defined( my $value = $get_and_clear->( 'CLICKHOUSE', 'port' ) ) ) {
$obj->{_CLICKHOUSE_port} = $value;
}
if ( defined( my $value = $get_and_clear->( 'CLICKHOUSE', 'user' ) ) ) {
$obj->_set_CLICKHOUSE_user( $value );
}
if ( defined( my $value = $get_and_clear->( 'CLICKHOUSE', 'password' ) ) ) {
$obj->_set_CLICKHOUSE_password( $value );
}
if ( defined( my $value = $get_and_clear->( 'CLICKHOUSE', 'database' ) ) ) {
$obj->_set_CLICKHOUSE_database( $value );
}
if ( defined( my $value = $get_and_clear->( 'ZONEMASTER', 'max_zonemaster_execution_time' ) ) ) {
$obj->_set_ZONEMASTER_max_zonemaster_execution_time( $value );
}
Expand Down Expand Up @@ -367,6 +383,19 @@ sub parse {
die "config: missing required property SQLITE.database_file (required when DB.engine = SQLite)\n"
if !defined $obj->SQLITE_database_file;
}
elsif ( $obj->DB_engine eq 'Clickhouse' ) {
die "config: missing required property CLICKHOUSE.host (required when DB.engine = Clickhouse)\n"
if !defined $obj->CLICKHOUSE_host;

die "config: missing required property CLICKHOUSE.user (required when DB.engine = Clickhouse)\n"
if !defined $obj->CLICKHOUSE_user;

die "config: missing required property CLICKHOUSE.password (required when DB.engine = Clickhouse)\n"
if !defined $obj->CLICKHOUSE_password;

die "config: missing required property CLICKHOUSE.database (required when DB.engine = Clickhouse)\n"
if !defined $obj->CLICKHOUSE_database;
}

# Check unknown property names
{
Expand Down Expand Up @@ -397,15 +426,15 @@ Returns a normalized string based on the supported databases.

=head3 EXCEPTION

Dies if the value is not one of SQLite, PostgreSQL or MySQL.
Dies if the value is not one of SQLite, PostgreSQL, MySQL or Clickhouse.

=cut

sub check_db {
my ( $self, $db ) = @_;

$db = untaint_engine_type( $db ) #
// die "Unknown database '$db', should be one of SQLite, MySQL or PostgreSQL\n";
// die "Unknown database '$db', should be one of SQLite, MySQL, PostgreSQL or Clickhouse\n";

return _normalize_engine_type( $db );
}
Expand All @@ -415,7 +444,7 @@ sub check_db {

Get the value of L<DB.engine|https://github.com/zonemaster/zonemaster/blob/master/docs/public/configuration/backend.md#engine>.

Returns one of C<"SQLite">, C<"PostgreSQL"> or C<"MySQL">.
Returns one of C<"SQLite">, C<"PostgreSQL">, C<"MySQL"> or C<"Clickhouse">.

=cut

Expand Down Expand Up @@ -520,6 +549,44 @@ Get the value of L<SQLITE.database_file|https://github.com/zonemaster/zonemaster
Returns a string.


=head2 CLICKHOUSE_database

Get the value of L<CLICKHOUSE.database|https://github.com/zonemaster/zonemaster-backend/blob/master/docs/Configuration.md#database-2>.

Connection to the Clickhouse server is done using the MySQL interface.

Returns a string.


=head2 CLICKHOUSE_host

Get the value of L<CLICKHOUSE.host|https://github.com/zonemaster/zonemaster-backend/blob/master/docs/Configuration.md#host-2>.

Returns a string.


=head2 CLICKHOUSE_port

Returns the L<CLICKHOUSE.port|https://github.com/zonemaster/zonemaster-backend/blob/master/docs/Configuration.md#port-2>
property from the loaded config.

Returns a number.


=head2 CLICKHOUSE_password

Get the value of L<CLICKHOUSE.password|https://github.com/zonemaster/zonemaster-backend/blob/master/docs/Configuration.md#password-2>.

Returns a string.


=head2 CLICKHOUSE_user

Get the value of L<CLICKHOUSE.user|https://github.com/zonemaster/zonemaster-backend/blob/master/docs/Configuration.md#user-2>.

Returns a string.


=head2 LANGUAGE_locale

Get the value of L<LANGUAGE.locale|https://github.com/zonemaster/zonemaster/blob/master/docs/public/configuration/backend.md#locale>.
Expand Down Expand Up @@ -657,6 +724,11 @@ sub POSTGRESQL_user { return $_[0]->{_POSTGR
sub POSTGRESQL_password { return $_[0]->{_POSTGRESQL_password}; }
sub POSTGRESQL_database { return $_[0]->{_POSTGRESQL_database}; }
sub SQLITE_database_file { return $_[0]->{_SQLITE_database_file}; }
sub CLICKHOUSE_host { return $_[0]->{_CLICKHOUSE_host}; }
sub CLICKHOUSE_port { return $_[0]->{_CLICKHOUSE_port}; }
sub CLICKHOUSE_user { return $_[0]->{_CLICKHOUSE_user}; }
sub CLICKHOUSE_password { return $_[0]->{_CLICKHOUSE_password}; }
sub CLICKHOUSE_database { return $_[0]->{_CLICKHOUSE_database}; }
sub LANGUAGE_locale { return %{ $_[0]->{_LANGUAGE_locale} }; }
sub PUBLIC_PROFILES { return %{ $_[0]->{_public_profiles} }; }
sub PRIVATE_PROFILES { return %{ $_[0]->{_private_profiles} }; }
Expand Down Expand Up @@ -686,6 +758,11 @@ UNITCHECK {
_create_setter( '_set_POSTGRESQL_password', '_POSTGRESQL_password', \&untaint_password );
_create_setter( '_set_POSTGRESQL_database', '_POSTGRESQL_database', \&untaint_postgresql_ident );
_create_setter( '_set_SQLITE_database_file', '_SQLITE_database_file', \&untaint_abs_path );
_create_setter( '_set_CLICKHOUSE_host', '_CLICKHOUSE_host', \&untaint_host );
_create_setter( '_set_CLICKHOUSE_port', '_CLICKHOUSE_port', \&untaint_strictly_positive_int );
_create_setter( '_set_CLICKHOUSE_user', '_CLICKHOUSE_user', \&untaint_postgresql_ident );
_create_setter( '_set_CLICKHOUSE_password', '_CLICKHOUSE_password', \&untaint_password );
_create_setter( '_set_CLICKHOUSE_database', '_CLICKHOUSE_database', \&untaint_postgresql_ident );
_create_setter( '_set_ZONEMASTER_max_zonemaster_execution_time', '_ZONEMASTER_max_zonemaster_execution_time', \&untaint_strictly_positive_int );
_create_setter( '_set_ZONEMASTER_lock_on_queue', '_ZONEMASTER_lock_on_queue', \&untaint_non_negative_int );
_create_setter( '_set_ZONEMASTER_number_of_processes_for_frontend_testing', '_ZONEMASTER_number_of_processes_for_frontend_testing', \&untaint_strictly_positive_int );
Expand Down Expand Up @@ -920,6 +997,7 @@ sub _normalize_engine_type {
mysql => 'MySQL',
postgresql => 'PostgreSQL',
sqlite => 'SQLite',
clickhouse => 'Clickhouse',
};

return $db_module_names->{ lc $value };
Expand Down
Loading