Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save prepared statements to be able to use them with connetion pooling #281

Open
wants to merge 46 commits into
base: REL_15_STABLE_neon
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
f341594
Neon rebased to REL_15_STABLE (up to 0a9045c9ff)
lubennikovaav Jul 22, 2022
290ba8c
fix regression tests
lubennikovaav Aug 10, 2022
259ee69
rebase to the latest origin and resolve conflicts
lubennikovaav Aug 16, 2022
0e44ead
Remove contrib neon and neon_test_utils.
lubennikovaav Aug 24, 2022
7cca665
Prevent access to uninitalized shaerd memory in InstallXLogFileSegmen…
lubennikovaav Aug 24, 2022
8dccc09
Remove Dockerfile, it's now in the neon repo
lubennikovaav Aug 29, 2022
de43d1f
Move backpressure throttling implementation to neon extension (#203)
knizhnik Sep 2, 2022
a661533
Merge last written cache lsn with new main branch (#201)
knizhnik Sep 4, 2022
216c6d1
Local prefetch implementation for Postgres 15
hlinnaka Sep 13, 2022
681dc96
Set last written LSN for the created relation (#212)
hlinnaka Sep 16, 2022
3a9fec1
Use normal install program to install server headers.
hlinnaka Sep 12, 2022
125fe69
Update expected output for sysviews test because of changed default v…
knizhnik Sep 12, 2022
51fda47
Undo diasming VM check warning in vacuumlazy.c (#214)
knizhnik Sep 20, 2022
13fb90c
Set Neon-specific FMGR_ABI_EXTRA
lubennikovaav Sep 21, 2022
c6a30fb
Don't use newline in PG_VERSION file.
lubennikovaav Sep 21, 2022
00c68fe
Unset ArchiveRecoveryRequested for Neon code path.
lubennikovaav Sep 21, 2022
748e9a4
Fix memory leak in ApplyRecord
knizhnik Sep 21, 2022
3af5dc6
Rebase to Stamp 15.0
lubennikovaav Oct 5, 2022
6e4d997
Pin pages with speculative insert tuples to prevent their reconstruct…
knizhnik Oct 20, 2022
448bafa
Fix shared memory initialization for last written LSN cache (#226)
knizhnik Oct 24, 2022
b1ef4ab
Fix upper boundary caculation in the chunks loop in SetLastWrittenLSN…
knizhnik Oct 28, 2022
a81e2b6
Stamp XLP_FIRST_IS_CONTRECORD only if we start writing with page offset.
arssher Oct 28, 2022
d509e1b
Move walredo process code under pgxn in the main 'neon' repository.
hlinnaka Oct 31, 2022
d53d568
Misc cleanup, mostly to reduce unnecessary differences with upstream.
hlinnaka Oct 31, 2022
99947e8
Optimize prefetch patterns in both heap seqscan and vacuum scans. (#228)
MMeent Nov 7, 2022
d71a77c
Fix prefetch issues in parallel scans and vacuum's cleanup scan (#235)
MMeent Nov 11, 2022
cb8971d
Fix expected results for regression tests (#238)
bayandin Nov 15, 2022
a3b5c0d
Use prefetch in pg_prewarm extension (#237)
knizhnik Nov 17, 2022
59220f1
PG15: Prefetch cleanup (#241)
MMeent Nov 23, 2022
285cd13
Drop unlogged table in regress test to avoid noise in tests
lubennikovaav Nov 21, 2022
46c44e8
Do not produce open file error for unlogged relations (#240)
knizhnik Nov 24, 2022
6ac694c
Maintain last written LSN for each page to enable prefetch on vacuum,…
knizhnik Nov 24, 2022
d3188f2
Prefetch cleanup: (#246)
MMeent Dec 5, 2022
c772b48
Set lsn fix v15 (#252)
knizhnik Dec 8, 2022
f538627
Show prefetch statistic in EXPLAIN (#249)
knizhnik Jan 12, 2023
ba841dd
Implement efficient prefetch for parallel bitmap heap scan (#258)
knizhnik Feb 14, 2023
ad5e789
Unlogged index fix v15 (#262)
knizhnik Feb 22, 2023
9a20933
Fix bitmap scan prefetch (#261)
knizhnik Feb 23, 2023
919851e
Revert handling of UNLOGGED tables on compute side v15.
arssher Feb 24, 2023
1af2632
Allow external main functions to skip config load and make last
kelvich Feb 23, 2023
257aaef
Remove walredo-related hacks from InternalIpcMemoryCreate()
kelvich Mar 6, 2023
f8a650e
Adjust prefetch target for parallel bitmap scan (#274)
knizhnik Mar 27, 2023
03505eb
Copy iterator result in BitmapHeapNext (#276)
knizhnik Apr 7, 2023
4ad87b0
Prefetch for index and index-only scans (#271)
knizhnik Apr 7, 2023
aee72b7
[PG15] Feature/replicas (#279)
MMeent Apr 13, 2023
52ef300
Save prepared statements to be able to use them with connetion pooling
knizhnik Apr 27, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
.git
.vscode
.circleci
tmp_install
compute_build
86 changes: 86 additions & 0 deletions configure
Original file line number Diff line number Diff line change
Expand Up @@ -717,6 +717,7 @@ with_libxml
with_uuid
with_readline
with_systemd
with_libseccomp
with_selinux
with_ldap
with_krb_srvnam
Expand Down Expand Up @@ -864,6 +865,7 @@ with_bsd_auth
with_ldap
with_bonjour
with_selinux
with_libseccomp
with_systemd
with_readline
with_libedit_preferred
Expand Down Expand Up @@ -1573,6 +1575,7 @@ Optional Packages:
--with-ldap build with LDAP support
--with-bonjour build with Bonjour support
--with-selinux build with SELinux support
--with-libseccomp build with libseccomp support
--with-systemd build with systemd support
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
Expand Down Expand Up @@ -8631,6 +8634,39 @@ fi
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_selinux" >&5
$as_echo "$with_selinux" >&6; }

#
# libseccomp
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with libseccomp support" >&5
$as_echo_n "checking whether to build with libseccomp support... " >&6; }



# Check whether --with-libseccomp was given.
if test "${with_libseccomp+set}" = set; then :
withval=$with_libseccomp;
case $withval in
yes)
:
;;
no)
:
;;
*)
as_fn_error $? "no argument expected for --with-libseccomp option" "$LINENO" 5
;;
esac

else
with_libseccomp=no

fi



{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_libseccomp" >&5
$as_echo "$with_libseccomp" >&6; }

#
# Systemd
#
Expand Down Expand Up @@ -14575,6 +14611,56 @@ else
fi


fi

if test "$with_libseccomp" = yes ; then
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for seccomp_init in -lseccomp" >&5
$as_echo_n "checking for seccomp_init in -lseccomp... " >&6; }
if ${ac_cv_lib_seccomp_seccomp_init+:} false; then :
$as_echo_n "(cached) " >&6
else
ac_check_lib_save_LIBS=$LIBS
LIBS="-lseccomp $LIBS"
cat confdefs.h - <<_ACEOF >conftest.$ac_ext
/* end confdefs.h. */

/* Override any GCC internal prototype to avoid an error.
Use char because int might match the return type of a GCC
builtin and then its argument prototype would still apply. */
#ifdef __cplusplus
extern "C"
#endif
char seccomp_init ();
int
main ()
{
return seccomp_init ();
;
return 0;
}
_ACEOF
if ac_fn_c_try_link "$LINENO"; then :
ac_cv_lib_seccomp_seccomp_init=yes
else
ac_cv_lib_seccomp_seccomp_init=no
fi
rm -f core conftest.err conftest.$ac_objext \
conftest$ac_exeext conftest.$ac_ext
LIBS=$ac_check_lib_save_LIBS
fi
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_seccomp_seccomp_init" >&5
$as_echo "$ac_cv_lib_seccomp_seccomp_init" >&6; }
if test "x$ac_cv_lib_seccomp_seccomp_init" = xyes; then :
cat >>confdefs.h <<_ACEOF
#define HAVE_LIBSECCOMP 1
_ACEOF

LIBS="-lseccomp $LIBS"

else
as_fn_error $? "library 'libseccomp' is required for Seccomp BPF support" "$LINENO" 5
fi

fi

# for contrib/uuid-ossp
Expand Down
13 changes: 13 additions & 0 deletions configure.ac
Original file line number Diff line number Diff line change
Expand Up @@ -927,6 +927,14 @@ PGAC_ARG_BOOL(with, selinux, no, [build with SELinux support])
AC_SUBST(with_selinux)
AC_MSG_RESULT([$with_selinux])

#
# libseccomp
#
AC_MSG_CHECKING([whether to build with libseccomp support])
PGAC_ARG_BOOL(with, libseccomp, no, [build with libseccomp support])
AC_SUBST(with_libseccomp)
AC_MSG_RESULT([$with_libseccomp])

#
# Systemd
#
Expand Down Expand Up @@ -1611,6 +1619,11 @@ dnl If you want to use Apple's own Bonjour code on another platform,
dnl just add -ldns_sd to LIBS manually.
fi

if test "$with_libseccomp" = yes ; then
AC_CHECK_LIB(seccomp, seccomp_init, [],
[AC_MSG_ERROR([library 'libseccomp' is required for Seccomp BPF support])])
fi

# for contrib/uuid-ossp
if test "$with_uuid" = bsd ; then
AC_CHECK_HEADERS(uuid.h,
Expand Down
35 changes: 35 additions & 0 deletions contrib/pg_prewarm/autoprewarm.c
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
#include "utils/rel.h"
#include "utils/relfilenodemap.h"
#include "utils/resowner.h"
#include "utils/spccache.h"

#define AUTOPREWARM_FILE "autoprewarm.blocks"

Expand Down Expand Up @@ -449,10 +450,12 @@ void
autoprewarm_database_main(Datum main_arg)
{
int pos;
int io_concurrency;
BlockInfoRecord *block_info;
Relation rel = NULL;
BlockNumber nblocks = 0;
BlockInfoRecord *old_blk = NULL;
BlockInfoRecord *prefetch_blk = NULL;
dsm_segment *seg;

/* Establish signal handlers; once that's done, unblock signals. */
Expand Down Expand Up @@ -499,6 +502,7 @@ autoprewarm_database_main(Datum main_arg)
{
relation_close(rel, AccessShareLock);
rel = NULL;
io_concurrency = -1;
CommitTransactionCommand();
}

Expand All @@ -518,6 +522,8 @@ autoprewarm_database_main(Datum main_arg)

if (!rel)
CommitTransactionCommand();
else
io_concurrency = get_tablespace_maintenance_io_concurrency(rel->rd_rel->reltablespace);
}
if (!rel)
{
Expand Down Expand Up @@ -550,6 +556,35 @@ autoprewarm_database_main(Datum main_arg)
continue;
}

/* if prefetching is enabled for this relation */
if (io_concurrency > 0)
{
/* make prefetch_blk catch up */
if (blk > prefetch_blk)
{
prefetch_blk = blk;
}

/* now, prefetch all following blocks */
while (prefetch_blk <= &block_info[apw_state->prewarm_stop_idx])
{
/* unless they're of a different relfilenode */
if (prefetch_blk->filenode != blk->filenode ||
prefetch_blk->forknum != blk->forknum ||
prefetch_blk->blocknum >= nblocks)
break;

/* or unless they are more than io_concurrency blocks ahead */
if (blk + io_concurrency <= prefetch_blk)
break;

PrefetchBuffer(rel, prefetch_blk->forknum, prefetch_blk->blocknum);

/* continue with the next block */
prefetch_blk++;
}
}

/* Prewarm buffer. */
buf = ReadBufferExtended(rel, blk->forknum, blk->blocknum, RBM_NORMAL,
NULL);
Expand Down
18 changes: 16 additions & 2 deletions contrib/pg_prewarm/pg_prewarm.c
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,14 @@
#include "access/relation.h"
#include "fmgr.h"
#include "miscadmin.h"
#include "optimizer/cost.h"
#include "storage/bufmgr.h"
#include "storage/smgr.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/spccache.h"

PG_MODULE_MAGIC;

Expand Down Expand Up @@ -183,14 +185,26 @@ pg_prewarm(PG_FUNCTION_ARGS)
}
else if (ptype == PREWARM_BUFFER)
{
BlockNumber prefetch_block = first_block;
Oid nspOid;
int io_concurrency;

nspOid = rel->rd_rel->reltablespace;
io_concurrency = get_tablespace_maintenance_io_concurrency(nspOid);

/*
* In buffer mode, we actually pull the data into shared_buffers.
*/
for (block = first_block; block <= last_block; ++block)
{
Buffer buf;

Buffer buf;
BlockNumber prefetch_stop = block + Min(last_block - block + 1,
io_concurrency);
CHECK_FOR_INTERRUPTS();
while (prefetch_block < prefetch_stop)
{
PrefetchBuffer(rel, forkNumber, prefetch_block++);
}
buf = ReadBufferExtended(rel, forkNumber, block, RBM_NORMAL, NULL);
ReleaseBuffer(buf);
++blocks_done;
Expand Down
1 change: 1 addition & 0 deletions contrib/pg_prewarm/pg_prewarm.control
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ comment = 'prewarm relation data'
default_version = '1.2'
module_pathname = '$libdir/pg_prewarm'
relocatable = true
trusted = true
1 change: 1 addition & 0 deletions src/Makefile.global.in
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,7 @@ with_tcl = @with_tcl@
with_ssl = @with_ssl@
with_readline = @with_readline@
with_selinux = @with_selinux@
with_libseccomp = @with_libseccomp@
with_systemd = @with_systemd@
with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
Expand Down
3 changes: 2 additions & 1 deletion src/backend/access/brin/brin_xlog.c
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ brin_xlog_insert_update(XLogReaderState *record,
}

/* need this page's blkno to store in revmap */
regpgno = BufferGetBlockNumber(buffer);
//ZENITH XXX Don't use BufferGetBlockNumber because wal-redo doesn't pin buffer.
XLogRecGetBlockTag(record, 0, NULL, NULL, &regpgno);

/* insert the index item into the page */
if (action == BLK_NEEDS_REDO)
Expand Down
8 changes: 8 additions & 0 deletions src/backend/access/gin/gininsert.c
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,8 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
elog(ERROR, "index \"%s\" already contains data",
RelationGetRelationName(index));

smgr_start_unlogged_build(index->rd_smgr);

initGinState(&buildstate.ginstate, index);
buildstate.indtuples = 0;
memset(&buildstate.buildStats, 0, sizeof(GinStatsData));
Expand Down Expand Up @@ -408,6 +410,8 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
buildstate.buildStats.nTotalPages = RelationGetNumberOfBlocks(index);
ginUpdateStats(index, &buildstate.buildStats, true);

smgr_finish_unlogged_build_phase_1(index->rd_smgr);

/*
* We didn't write WAL records as we built the index, so if WAL-logging is
* required, write all pages to the WAL now.
Expand All @@ -417,8 +421,12 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
log_newpage_range(index, MAIN_FORKNUM,
0, RelationGetNumberOfBlocks(index),
true);
SetLastWrittenLSNForBlockRange(XactLastRecEnd, index->rd_smgr->smgr_rnode.node, MAIN_FORKNUM, 0, RelationGetNumberOfBlocks(index));
SetLastWrittenLSNForRelation(XactLastRecEnd, index->rd_smgr->smgr_rnode.node, MAIN_FORKNUM);
}

smgr_end_unlogged_build(index->rd_smgr);

/*
* Return statistics
*/
Expand Down
19 changes: 13 additions & 6 deletions src/backend/access/gin/ginxlog.c
Original file line number Diff line number Diff line change
Expand Up @@ -407,6 +407,7 @@ ginRedoSplit(XLogReaderState *record)
rootbuf;
bool isLeaf = (data->flags & GIN_INSERT_ISLEAF) != 0;
bool isRoot = (data->flags & GIN_SPLIT_ROOT) != 0;
XLogRedoAction action;

/*
* First clear incomplete-split flag on child page if this finishes a
Expand All @@ -415,21 +416,27 @@ ginRedoSplit(XLogReaderState *record)
if (!isLeaf)
ginRedoClearIncompleteSplit(record, 3);

if (XLogReadBufferForRedo(record, 0, &lbuffer) != BLK_RESTORED)
action = XLogReadBufferForRedo(record, 0, &lbuffer);
if (action != BLK_RESTORED && action != BLK_DONE)
elog(ERROR, "GIN split record did not contain a full-page image of left page");

if (XLogReadBufferForRedo(record, 1, &rbuffer) != BLK_RESTORED)
action = XLogReadBufferForRedo(record, 1, &rbuffer);
if (action != BLK_RESTORED && action != BLK_DONE)
elog(ERROR, "GIN split record did not contain a full-page image of right page");

if (isRoot)
{
if (XLogReadBufferForRedo(record, 2, &rootbuf) != BLK_RESTORED)
action = XLogReadBufferForRedo(record, 2, &rootbuf);
if (action != BLK_RESTORED && action != BLK_DONE)
elog(ERROR, "GIN split record did not contain a full-page image of root page");
UnlockReleaseBuffer(rootbuf);
if (rootbuf != InvalidBuffer)
UnlockReleaseBuffer(rootbuf);
}

UnlockReleaseBuffer(rbuffer);
UnlockReleaseBuffer(lbuffer);
if (rbuffer != InvalidBuffer)
UnlockReleaseBuffer(rbuffer);
if (lbuffer != InvalidBuffer)
UnlockReleaseBuffer(lbuffer);
}

/*
Expand Down
Loading