Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PG17 compatibility: Fix Test Failure in dml_recursive #7727

Closed
wants to merge 14 commits into from

Conversation

m3hm3t
Copy link
Contributor

@m3hm3t m3hm3t commented Nov 11, 2024

PostgreSQL 17 includes an enhancement that allows the optimizer to transform correlated IN subqueries into more efficient join operations.
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=9f1337639

UPDATE
	second_distributed_table
SET
	dept = foo.tenant_id::int / 4
FROM
(
	SELECT baz.tenant_id FROM
	(
		SELECT
			second_distributed_table.dept, second_distributed_table.tenant_id
		FROM
			second_distributed_table, distributed_table as d1
		WHERE
			d1.tenant_id = second_distributed_table.tenant_id
		AND
			second_distributed_table.dept IN (3,4)
			AND
			second_distributed_table.tenant_id IN
			(
					SELECT s2.tenant_id
					FROM second_distributed_table as s2
					GROUP BY d1.tenant_id, s2.tenant_id
			)
	) as baz
	) as foo WHERE second_distributed_table.tenant_id = foo.tenant_id
RETURNING *;
-ERROR:  complex joins are only supported when all distributed tables are co-located and joined on their distribution columns
+ tenant_id | dept |          info          | tenant_id 
+-----------+------+------------------------+-----------
+ 14        |    3 | {"f1": 14, "f2": 196}  | 14
+ 23        |    5 | {"f1": 23, "f2": 529}  | 23
+ 24        |    6 | {"f1": 24, "f2": 576}  | 24
+ 3         |    0 | {"f1": 3, "f2": 9}     | 3
+ 33        |    8 | {"f1": 33, "f2": 1089} | 33
+ 34        |    8 | {"f1": 34, "f2": 1156} | 34
+ 4         |    1 | {"f1": 4, "f2": 16}    | 4
+ 43        |   10 | {"f1": 43, "f2": 1849} | 43
+ 44        |   11 | {"f1": 44, "f2": 1936} | 44
+ 53        |   13 | {"f1": 53, "f2": 2809} | 53
+ 54        |   13 | {"f1": 54, "f2": 2916} | 54
+ 63        |   15 | {"f1": 63, "f2": 3969} | 63
+ 64        |   16 | {"f1": 64, "f2": 4096} | 64
+ 73        |   18 | {"f1": 73, "f2": 5329} | 73
+ 74        |   18 | {"f1": 74, "f2": 5476} | 74
+ 83        |   20 | {"f1": 83, "f2": 6889} | 83
+ 84        |   21 | {"f1": 84, "f2": 7056} | 84
+ 93        |   23 | {"f1": 93, "f2": 8649} | 93
+ 94        |   23 | {"f1": 94, "f2": 8836} | 94
+(19 rows)
+

naisila and others added 12 commits November 11, 2024 11:57
(cherry picked from commit ae3ed7d)
(cherry picked from commit 76f60a7)
(cherry picked from commit df9c7b4)
In PG17, the outer loop in acquire_sample_rows() changed
from
while (BlockSampler_HasMore(&bs))
to
while (table_scan_analyze_next_block(scan, stream))

Relevant PG commit:
041b96802efa33d2bc9456f2ad946976b92b5ae1
postgres/postgres@041b968

It is expected that the scan_analyze_next_block function will
check if there are any blocks left. So we add that check in
columnar_scan_analyze_next_block

(cherry picked from commit 7eb0ad5)
PG 17 added support for DEFAULT in ALTER TABLE .. SET ACCESS METHOD

Relevant PG commit:
d61a6cad6418f643a5773352038d0dfe5d3535b8
postgres/postgres@d61a6ca

In that case, name in AlterTableCmd would be null.
Add a null check here to avoid crash.

(cherry picked from commit 71b9974)
…tests

Fix pg15 pg16 multi_mx_create_table multi_schema_support

Relevant PG commit:
postgres/postgres@f696c0c
f696c0cd5f299f1b51e214efc55a22a782cc175d

(cherry picked from commit 17a2ed0)
Relevant PG commit:
f69319f2f1fb16eda4b535bcccec90dff3a6795e
postgres/postgres@f69319f

(cherry picked from commit 6c12b10)
This PR addresses a regression test failure in the multi-mx feature of
Citus with the new PostgreSQL 17 version. The regression was identified
during the execution of multi-node tests, specifically targeting
compatibility issues introduced with PostgreSQL 17.

---------

Co-authored-by: Mehmet YILMAZ <[email protected]>
(cherry picked from commit 70cf729)
This reverts commit e4040dd.

Reverting for now as this commit is fixing more than one thing
at once at multi_extension.out file

Its a harmless revert for testing purposes
Copy link

codecov bot commented Nov 11, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (naisila/pg17_support@c0a5f5c). Learn more about missing BASE report.

Additional details and impacted files
@@                   Coverage Diff                   @@
##             naisila/pg17_support    #7727   +/-   ##
=======================================================
  Coverage                        ?   89.61%           
=======================================================
  Files                           ?      274           
  Lines                           ?    59689           
  Branches                        ?     7446           
=======================================================
  Hits                            ?    53492           
  Misses                          ?     4067           
  Partials                        ?     2130           

@m3hm3t m3hm3t self-assigned this Nov 11, 2024
@m3hm3t m3hm3t marked this pull request as ready for review November 11, 2024 13:09
Comment on lines +301 to +323
tenant_id | dept | info | tenant_id
---------------------------------------------------------------------
14 | 3 | {"f1": 14, "f2": 196} | 14
23 | 5 | {"f1": 23, "f2": 529} | 23
24 | 6 | {"f1": 24, "f2": 576} | 24
3 | 0 | {"f1": 3, "f2": 9} | 3
33 | 8 | {"f1": 33, "f2": 1089} | 33
34 | 8 | {"f1": 34, "f2": 1156} | 34
4 | 1 | {"f1": 4, "f2": 16} | 4
43 | 10 | {"f1": 43, "f2": 1849} | 43
44 | 11 | {"f1": 44, "f2": 1936} | 44
53 | 13 | {"f1": 53, "f2": 2809} | 53
54 | 13 | {"f1": 54, "f2": 2916} | 54
63 | 15 | {"f1": 63, "f2": 3969} | 63
64 | 16 | {"f1": 64, "f2": 4096} | 64
73 | 18 | {"f1": 73, "f2": 5329} | 73
74 | 18 | {"f1": 74, "f2": 5476} | 74
83 | 20 | {"f1": 83, "f2": 6889} | 83
84 | 21 | {"f1": 84, "f2": 7056} | 84
93 | 23 | {"f1": 93, "f2": 8649} | 93
94 | 23 | {"f1": 94, "f2": 8836} | 94
(19 rows)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to take a closer look into the new query plan to see how that optimization allowed Citus to run the query now, you can ping me tomorrow for us to quickly debug this together.

@naisila naisila changed the title Fix Test Failure in dml_recursive in PG17 PG17 compatibility: Fix Test Failure in dml_recursive Nov 12, 2024
naisila
naisila previously approved these changes Nov 12, 2024
Copy link
Member

@naisila naisila left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fix, thanks @m3hm3t and @onurctirtir
Always appreciate a fix that avoids a new output file :)
Let's merge directly to release-13.0 and then rebase naisila/pg17_support branch

EDIT: sorry, wrong PR

@naisila naisila dismissed their stale review November 12, 2024 11:01

accidentally approved the wrong PR

-- dml_recursive_0.out for PG16 and before
-- dml_recursive.out for PG17
-- related commit
-- PostgreSQL 17 includes an enhancement that allows the optimizer to transform correlated IN subqueries into more efficient join operations.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's hold off a bit more before merging this, until we are sure that nothing needs to change in the Citus codebase.

@naisila
Copy link
Member

naisila commented Nov 17, 2024

@m3hm3t given that we have a way to avoid an alternative output #7745, I suggest closing this PR, if you also agree.

@m3hm3t m3hm3t closed this Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants