Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persist pgstat file to preserve statistic between sessions #357

Open
wants to merge 66 commits into
base: REL_15_STABLE_neon
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 65 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
b2f37fc
Neon rebased to REL_15_STABLE (up to 0a9045c9ff)
lubennikovaav Jul 22, 2022
58cfeeb
fix regression tests
lubennikovaav Aug 10, 2022
65dd0f7
rebase to the latest origin and resolve conflicts
lubennikovaav Aug 16, 2022
b2dbd9d
Remove contrib neon and neon_test_utils.
lubennikovaav Aug 24, 2022
283fcdf
Prevent access to uninitalized shaerd memory in InstallXLogFileSegmen…
lubennikovaav Aug 24, 2022
205d59d
Remove Dockerfile, it's now in the neon repo
lubennikovaav Aug 29, 2022
246bae1
Move backpressure throttling implementation to neon extension (#203)
knizhnik Sep 2, 2022
ad18f68
Merge last written cache lsn with new main branch (#201)
knizhnik Sep 4, 2022
9b168ed
Local prefetch implementation for Postgres 15
hlinnaka Sep 13, 2022
5237824
Set last written LSN for the created relation (#212)
hlinnaka Sep 16, 2022
871400c
Use normal install program to install server headers.
hlinnaka Sep 12, 2022
945737b
Update expected output for sysviews test because of changed default v…
knizhnik Sep 12, 2022
3c976a2
Undo diasming VM check warning in vacuumlazy.c (#214)
knizhnik Sep 20, 2022
67d1c0a
Set Neon-specific FMGR_ABI_EXTRA
lubennikovaav Sep 21, 2022
c85bc09
Don't use newline in PG_VERSION file.
lubennikovaav Sep 21, 2022
d363e94
Unset ArchiveRecoveryRequested for Neon code path.
lubennikovaav Sep 21, 2022
eefa704
Fix memory leak in ApplyRecord
knizhnik Sep 21, 2022
ed3602e
Rebase to Stamp 15.0
lubennikovaav Oct 5, 2022
f50d2de
Pin pages with speculative insert tuples to prevent their reconstruct…
knizhnik Oct 20, 2022
2df27d7
Fix shared memory initialization for last written LSN cache (#226)
knizhnik Oct 24, 2022
7c26185
Fix upper boundary caculation in the chunks loop in SetLastWrittenLSN…
knizhnik Oct 28, 2022
ea88e63
Stamp XLP_FIRST_IS_CONTRECORD only if we start writing with page offset.
arssher Oct 28, 2022
ae333c0
Move walredo process code under pgxn in the main 'neon' repository.
hlinnaka Oct 31, 2022
7ab6932
Misc cleanup, mostly to reduce unnecessary differences with upstream.
hlinnaka Oct 31, 2022
7e5a1f0
Optimize prefetch patterns in both heap seqscan and vacuum scans. (#228)
MMeent Nov 7, 2022
9544eaa
Fix prefetch issues in parallel scans and vacuum's cleanup scan (#235)
MMeent Nov 11, 2022
3a06992
Fix expected results for regression tests (#238)
bayandin Nov 15, 2022
4d4f707
Use prefetch in pg_prewarm extension (#237)
knizhnik Nov 17, 2022
d2453bd
PG15: Prefetch cleanup (#241)
MMeent Nov 23, 2022
ebf2fec
Drop unlogged table in regress test to avoid noise in tests
lubennikovaav Nov 21, 2022
0d5c7be
Do not produce open file error for unlogged relations (#240)
knizhnik Nov 24, 2022
9bcaa14
Maintain last written LSN for each page to enable prefetch on vacuum,…
knizhnik Nov 24, 2022
3ace5a3
Prefetch cleanup: (#246)
MMeent Dec 5, 2022
20d665d
Set lsn fix v15 (#252)
knizhnik Dec 8, 2022
8c1ae26
Show prefetch statistic in EXPLAIN (#249)
knizhnik Jan 12, 2023
723f85f
Implement efficient prefetch for parallel bitmap heap scan (#258)
knizhnik Feb 14, 2023
1621daf
Unlogged index fix v15 (#262)
knizhnik Feb 22, 2023
1c049dc
Fix bitmap scan prefetch (#261)
knizhnik Feb 23, 2023
d1e3fb3
Revert handling of UNLOGGED tables on compute side v15.
arssher Feb 24, 2023
a9a9b43
Allow external main functions to skip config load and make last
kelvich Feb 23, 2023
c2b4266
Remove walredo-related hacks from InternalIpcMemoryCreate()
kelvich Mar 6, 2023
8fea298
Adjust prefetch target for parallel bitmap scan (#274)
knizhnik Mar 27, 2023
59f7ecb
Copy iterator result in BitmapHeapNext (#276)
knizhnik Apr 7, 2023
34608d6
Prefetch for index and index-only scans (#271)
knizhnik Apr 7, 2023
96fca0d
[PG15] Feature/replicas (#279)
MMeent Apr 13, 2023
e3e5075
Fix entering hot standby mode for Neon
lubennikovaav May 5, 2023
875fc76
Do not allow users with CREATEROLE privelege to manage system user
kelvich Jul 7, 2023
58b7244
Fix regression tests after the patch with CREATEROLE restrictions
kelvich Jul 15, 2023
b3b692c
Add startup logs (#293)
bojanserafimov Jul 17, 2023
ac82176
Make it possible to grant self created roles (#298)
knizhnik Jul 27, 2023
c6c5d17
Update expected file for create_role test (#301)
knizhnik Jul 28, 2023
429ecaf
Define NEON_SMGR in smgr.h to make it possible for extensions to use …
knizhnik Jul 29, 2023
9e1dd46
Request extension files and libraries from compute_ctl
lubennikovaav Jun 13, 2023
1d89f6d
Upadte last written LSN after walloging all createdb stuff (#307)
knizhnik Sep 25, 2023
1afb4f6
[refer #111] Persist logical replication files in WAL (#306)
knizhnik Oct 18, 2023
9ca7417
Fix elog format error in wallog_mapping_file (#316)
knizhnik Oct 19, 2023
ac718a9
Update WAL buffers when restoring WAL at compute needed for LR (#324)
knizhnik Nov 3, 2023
98d8d6a
Prevent output callbacks from hearing about neon-file messages (#329)
Dec 14, 2023
b9591d3
strncmp vs strcmp
save-buffer Dec 15, 2023
b0297cd
Remove excessive walsender reply logging.
arssher Oct 19, 2023
940d29b
Allow publications FOR ALL TABLES to neon_superuser
save-buffer Jan 8, 2024
8d18a8d
Switch GetCurrentRoleId to GetUserId
save-buffer Jan 24, 2024
0ff55c5
Support creating subscriptions as neon_superuser
Jan 25, 2024
6ee78a3
On demand downloading of SLRU segments (#332)
knizhnik Jan 31, 2024
771e5f5
Persist pgstat file to preserve statistic between sessions
Feb 8, 2024
6afa5f4
Persistent pgstat file only if it is successfully created
Feb 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
.git
.vscode
.circleci
tmp_install
compute_build
86 changes: 86 additions & 0 deletions configure
Original file line number Diff line number Diff line change
Expand Up @@ -717,6 +717,7 @@ with_libxml
with_uuid
with_readline
with_systemd
with_libseccomp
with_selinux
with_ldap
with_krb_srvnam
Expand Down Expand Up @@ -864,6 +865,7 @@ with_bsd_auth
with_ldap
with_bonjour
with_selinux
with_libseccomp
with_systemd
with_readline
with_libedit_preferred
Expand Down Expand Up @@ -1573,6 +1575,7 @@ Optional Packages:
--with-ldap build with LDAP support
--with-bonjour build with Bonjour support
--with-selinux build with SELinux support
--with-libseccomp build with libseccomp support
--with-systemd build with systemd support
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
Expand Down Expand Up @@ -8633,6 +8636,39 @@ fi
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_selinux" >&5
$as_echo "$with_selinux" >&6; }

#
# libseccomp
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with libseccomp support" >&5
$as_echo_n "checking whether to build with libseccomp support... " >&6; }



# Check whether --with-libseccomp was given.
if test "${with_libseccomp+set}" = set; then :
withval=$with_libseccomp;
case $withval in
yes)
:
;;
no)
:
;;
*)
as_fn_error $? "no argument expected for --with-libseccomp option" "$LINENO" 5
;;
esac

else
with_libseccomp=no

fi



{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_libseccomp" >&5
$as_echo "$with_libseccomp" >&6; }

#
# Systemd
#
Expand Down Expand Up @@ -14589,6 +14625,56 @@ else
fi


fi

if test "$with_libseccomp" = yes ; then
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for seccomp_init in -lseccomp" >&5
$as_echo_n "checking for seccomp_init in -lseccomp... " >&6; }
if ${ac_cv_lib_seccomp_seccomp_init+:} false; then :
$as_echo_n "(cached) " >&6
else
ac_check_lib_save_LIBS=$LIBS
LIBS="-lseccomp $LIBS"
cat confdefs.h - <<_ACEOF >conftest.$ac_ext
/* end confdefs.h. */

/* Override any GCC internal prototype to avoid an error.
Use char because int might match the return type of a GCC
builtin and then its argument prototype would still apply. */
#ifdef __cplusplus
extern "C"
#endif
char seccomp_init ();
int
main ()
{
return seccomp_init ();
;
return 0;
}
_ACEOF
if ac_fn_c_try_link "$LINENO"; then :
ac_cv_lib_seccomp_seccomp_init=yes
else
ac_cv_lib_seccomp_seccomp_init=no
fi
rm -f core conftest.err conftest.$ac_objext \
conftest$ac_exeext conftest.$ac_ext
LIBS=$ac_check_lib_save_LIBS
fi
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_seccomp_seccomp_init" >&5
$as_echo "$ac_cv_lib_seccomp_seccomp_init" >&6; }
if test "x$ac_cv_lib_seccomp_seccomp_init" = xyes; then :
cat >>confdefs.h <<_ACEOF
#define HAVE_LIBSECCOMP 1
_ACEOF

LIBS="-lseccomp $LIBS"

else
as_fn_error $? "library 'libseccomp' is required for Seccomp BPF support" "$LINENO" 5
fi

fi

# for contrib/uuid-ossp
Expand Down
13 changes: 13 additions & 0 deletions configure.ac
Original file line number Diff line number Diff line change
Expand Up @@ -927,6 +927,14 @@ PGAC_ARG_BOOL(with, selinux, no, [build with SELinux support])
AC_SUBST(with_selinux)
AC_MSG_RESULT([$with_selinux])

#
# libseccomp
#
AC_MSG_CHECKING([whether to build with libseccomp support])
PGAC_ARG_BOOL(with, libseccomp, no, [build with libseccomp support])
AC_SUBST(with_libseccomp)
AC_MSG_RESULT([$with_libseccomp])

#
# Systemd
#
Expand Down Expand Up @@ -1613,6 +1621,11 @@ dnl If you want to use Apple's own Bonjour code on another platform,
dnl just add -ldns_sd to LIBS manually.
fi

if test "$with_libseccomp" = yes ; then
AC_CHECK_LIB(seccomp, seccomp_init, [],
[AC_MSG_ERROR([library 'libseccomp' is required for Seccomp BPF support])])
fi

# for contrib/uuid-ossp
if test "$with_uuid" = bsd ; then
AC_CHECK_HEADERS(uuid.h,
Expand Down
35 changes: 35 additions & 0 deletions contrib/pg_prewarm/autoprewarm.c
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
#include "utils/rel.h"
#include "utils/relfilenodemap.h"
#include "utils/resowner.h"
#include "utils/spccache.h"

#define AUTOPREWARM_FILE "autoprewarm.blocks"

Expand Down Expand Up @@ -449,10 +450,12 @@ void
autoprewarm_database_main(Datum main_arg)
{
int pos;
int io_concurrency;
BlockInfoRecord *block_info;
Relation rel = NULL;
BlockNumber nblocks = 0;
BlockInfoRecord *old_blk = NULL;
BlockInfoRecord *prefetch_blk = NULL;
dsm_segment *seg;

/* Establish signal handlers; once that's done, unblock signals. */
Expand Down Expand Up @@ -499,6 +502,7 @@ autoprewarm_database_main(Datum main_arg)
{
relation_close(rel, AccessShareLock);
rel = NULL;
io_concurrency = -1;
CommitTransactionCommand();
}

Expand All @@ -518,6 +522,8 @@ autoprewarm_database_main(Datum main_arg)

if (!rel)
CommitTransactionCommand();
else
io_concurrency = get_tablespace_maintenance_io_concurrency(rel->rd_rel->reltablespace);
}
if (!rel)
{
Expand Down Expand Up @@ -550,6 +556,35 @@ autoprewarm_database_main(Datum main_arg)
continue;
}

/* if prefetching is enabled for this relation */
if (io_concurrency > 0)
{
/* make prefetch_blk catch up */
if (blk > prefetch_blk)
{
prefetch_blk = blk;
}

/* now, prefetch all following blocks */
while (prefetch_blk <= &block_info[apw_state->prewarm_stop_idx])
{
/* unless they're of a different relfilenode */
if (prefetch_blk->filenode != blk->filenode ||
prefetch_blk->forknum != blk->forknum ||
prefetch_blk->blocknum >= nblocks)
break;

/* or unless they are more than io_concurrency blocks ahead */
if (blk + io_concurrency <= prefetch_blk)
break;

PrefetchBuffer(rel, prefetch_blk->forknum, prefetch_blk->blocknum);

/* continue with the next block */
prefetch_blk++;
}
}

/* Prewarm buffer. */
buf = ReadBufferExtended(rel, blk->forknum, blk->blocknum, RBM_NORMAL,
NULL);
Expand Down
18 changes: 16 additions & 2 deletions contrib/pg_prewarm/pg_prewarm.c
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,14 @@
#include "access/relation.h"
#include "fmgr.h"
#include "miscadmin.h"
#include "optimizer/cost.h"
#include "storage/bufmgr.h"
#include "storage/smgr.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/spccache.h"

PG_MODULE_MAGIC;

Expand Down Expand Up @@ -183,14 +185,26 @@ pg_prewarm(PG_FUNCTION_ARGS)
}
else if (ptype == PREWARM_BUFFER)
{
BlockNumber prefetch_block = first_block;
Oid nspOid;
int io_concurrency;

nspOid = rel->rd_rel->reltablespace;
io_concurrency = get_tablespace_maintenance_io_concurrency(nspOid);

/*
* In buffer mode, we actually pull the data into shared_buffers.
*/
for (block = first_block; block <= last_block; ++block)
{
Buffer buf;

Buffer buf;
BlockNumber prefetch_stop = block + Min(last_block - block + 1,
io_concurrency);
CHECK_FOR_INTERRUPTS();
while (prefetch_block < prefetch_stop)
{
PrefetchBuffer(rel, forkNumber, prefetch_block++);
}
buf = ReadBufferExtended(rel, forkNumber, block, RBM_NORMAL, NULL);
ReleaseBuffer(buf);
++blocks_done;
Expand Down
1 change: 1 addition & 0 deletions contrib/pg_prewarm/pg_prewarm.control
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ comment = 'prewarm relation data'
default_version = '1.2'
module_pathname = '$libdir/pg_prewarm'
relocatable = true
trusted = true
1 change: 1 addition & 0 deletions src/Makefile.global.in
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,7 @@ with_tcl = @with_tcl@
with_ssl = @with_ssl@
with_readline = @with_readline@
with_selinux = @with_selinux@
with_libseccomp = @with_libseccomp@
with_systemd = @with_systemd@
with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
Expand Down
3 changes: 2 additions & 1 deletion src/backend/access/brin/brin_xlog.c
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ brin_xlog_insert_update(XLogReaderState *record,
}

/* need this page's blkno to store in revmap */
regpgno = BufferGetBlockNumber(buffer);
//ZENITH XXX Don't use BufferGetBlockNumber because wal-redo doesn't pin buffer.
XLogRecGetBlockTag(record, 0, NULL, NULL, &regpgno);

/* insert the index item into the page */
if (action == BLK_NEEDS_REDO)
Expand Down
8 changes: 8 additions & 0 deletions src/backend/access/gin/gininsert.c
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,8 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
elog(ERROR, "index \"%s\" already contains data",
RelationGetRelationName(index));

smgr_start_unlogged_build(index->rd_smgr);

initGinState(&buildstate.ginstate, index);
buildstate.indtuples = 0;
memset(&buildstate.buildStats, 0, sizeof(GinStatsData));
Expand Down Expand Up @@ -408,6 +410,8 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
buildstate.buildStats.nTotalPages = RelationGetNumberOfBlocks(index);
ginUpdateStats(index, &buildstate.buildStats, true);

smgr_finish_unlogged_build_phase_1(index->rd_smgr);

/*
* We didn't write WAL records as we built the index, so if WAL-logging is
* required, write all pages to the WAL now.
Expand All @@ -417,8 +421,12 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
log_newpage_range(index, MAIN_FORKNUM,
0, RelationGetNumberOfBlocks(index),
true);
SetLastWrittenLSNForBlockRange(XactLastRecEnd, index->rd_smgr->smgr_rnode.node, MAIN_FORKNUM, 0, RelationGetNumberOfBlocks(index));
SetLastWrittenLSNForRelation(XactLastRecEnd, index->rd_smgr->smgr_rnode.node, MAIN_FORKNUM);
}

smgr_end_unlogged_build(index->rd_smgr);

/*
* Return statistics
*/
Expand Down
19 changes: 13 additions & 6 deletions src/backend/access/gin/ginxlog.c
Original file line number Diff line number Diff line change
Expand Up @@ -407,6 +407,7 @@ ginRedoSplit(XLogReaderState *record)
rootbuf;
bool isLeaf = (data->flags & GIN_INSERT_ISLEAF) != 0;
bool isRoot = (data->flags & GIN_SPLIT_ROOT) != 0;
XLogRedoAction action;

/*
* First clear incomplete-split flag on child page if this finishes a
Expand All @@ -415,21 +416,27 @@ ginRedoSplit(XLogReaderState *record)
if (!isLeaf)
ginRedoClearIncompleteSplit(record, 3);

if (XLogReadBufferForRedo(record, 0, &lbuffer) != BLK_RESTORED)
action = XLogReadBufferForRedo(record, 0, &lbuffer);
if (action != BLK_RESTORED && action != BLK_DONE)
elog(ERROR, "GIN split record did not contain a full-page image of left page");

if (XLogReadBufferForRedo(record, 1, &rbuffer) != BLK_RESTORED)
action = XLogReadBufferForRedo(record, 1, &rbuffer);
if (action != BLK_RESTORED && action != BLK_DONE)
elog(ERROR, "GIN split record did not contain a full-page image of right page");

if (isRoot)
{
if (XLogReadBufferForRedo(record, 2, &rootbuf) != BLK_RESTORED)
action = XLogReadBufferForRedo(record, 2, &rootbuf);
if (action != BLK_RESTORED && action != BLK_DONE)
elog(ERROR, "GIN split record did not contain a full-page image of root page");
UnlockReleaseBuffer(rootbuf);
if (rootbuf != InvalidBuffer)
UnlockReleaseBuffer(rootbuf);
}

UnlockReleaseBuffer(rbuffer);
UnlockReleaseBuffer(lbuffer);
if (rbuffer != InvalidBuffer)
UnlockReleaseBuffer(rbuffer);
if (lbuffer != InvalidBuffer)
UnlockReleaseBuffer(lbuffer);
}

/*
Expand Down
Loading