Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libSQL WAL API #1788

Merged
merged 8 commits into from
Nov 11, 2024
Merged

libSQL WAL API #1788

merged 8 commits into from
Nov 11, 2024

Conversation

penberg
Copy link
Collaborator

@penberg penberg commented Oct 11, 2024

The libSQL has the following WAL API functions, which are useful for
syncing the WAL between databases:

  • libsql_wal_disable_checkpoint -- Disable checkpointing, including on database close.
  • libsql_wal_frame_count -- Get the number of frames in the WAL.
  • libsql_wal_get_frame -- Get a frame from the WAL.
  • libsql_wal_insert_begin -- Begin WAL insertion.
  • libsql_wal_insert_frame -- Insert a frame into the WAL.
  • libsql_wal_insert_end -- End WAL insertion.

The purpose of this API is to allow syncing the frames of one database WAL onto another. Example usage looks as follows (omitting error handling):

static void sync_db(sqlite3 *db_primary, sqlite3 *db_backup){
  unsigned int max_frame;

  libsql_wal_frame_count(db_primary, &max_frame);
  libsql_wal_begin_commit(db_backup);
  for(int i=1; i<=max_frame; i++){
    char frame[4096+24];
    libsql_wal_get_frame(db_primary, i, frame, sizeof(frame));
    libsql_wal_insert_frame(db_backup, i, frame, sizeof(frame));
  }
  libsql_wal_end_commit(db_backup);
}

Possible future work:

  • Do we need to make the WAL salt the same across synced databases?
  • How can we detect that frames belong to a database that it is synced to?

Fixes #200

@penberg penberg changed the title WAL API (wip) libSQL WAL API Oct 18, 2024
@penberg penberg force-pushed the wal-api branch 4 times, most recently from ab75b74 to 53a804e Compare October 24, 2024 09:56
@penberg penberg marked this pull request as ready for review October 24, 2024 09:57
SQLite unconditionally checkpoints when the last connection is closed to
a database. Let's add a libSQL extension API to give callers control
over that.
@penberg penberg force-pushed the wal-api branch 3 times, most recently from c22b22b to 929c8e9 Compare October 29, 2024 11:22
This adds a new xReadFrameRaw() function to the virtual WAL API, which
upper layers can use to fetch the full frame, including the page number,
that is useful for appending frames to a WAL.
This patch adds a libsql_wal_get_frame() API to read the raw WAL frame
from a database.

You can use the API as follows (omitting error handling) to obtain all
the WAL frames in a database:

```c
static int sync_db(sqlite3 *db_primary, sqlite3 *db_backup){
  unsigned int max_frame;

  libsql_wal_frame_count(db_primary, &max_frame);

  for(int i=1; i<=max_frame; i++){
    char frame[4096+24];

    libsql_wal_get_frame(db_primary, i, frame, sizeof(frame));
  }
}
```
libsql-sqlite3/src/pager.c Outdated Show resolved Hide resolved
libsql-sqlite3/src/pager.c Show resolved Hide resolved
```c
static void sync_db(sqlite3 *db_primary, sqlite3 *db_backup){
  unsigned int max_frame;

  libsql_wal_frame_count(db_primary, &max_frame);
  libsql_wal_begin_commit(db_backup);
  for(int i=1; i<=max_frame; i++){
    char frame[4096+24];
    libsql_wal_get_frame(db_primary, i, frame, sizeof(frame));
    libsql_wal_insert_frame(db_backup, i, frame, sizeof(frame));
  }
  libsql_wal_end_commit(db_backup);
}
```
@penberg penberg added this pull request to the merge queue Nov 11, 2024
Merged via the queue into main with commit 246667c Nov 11, 2024
19 checks passed
@penberg penberg deleted the wal-api branch November 11, 2024 12:23
@justjake
Copy link

Is the sqlite3_mutex_enter seen here disabled outside of SQLITE_CONFIG_SERIALIZED? Or is this mutex only disabled with SQLITE_CONFIG_SINGLETHREAD? Is grabbing a mutex around each read typical in the rest of sqlite? For a potential bulk-copy need like WAL I would think amortizing the mutex by batching multiple read calls per mutex enter would be helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

libsql_inject_frames
3 participants