sqlite: support db.loadExtension

[himself65]

Closes #53898

This PR will implement loadExtension API to align with other SQLite3 JS library like better-sqlite3, node-sqlite3, jsr:@db/sqlite, bun:sqlite

Example Code:

import { load } from 'sqlite-vec'
import { DatabaseSync } from 'node:sqlite'

const db = new DatabaseSync(':memory:', {
  allowExtension: true
})
load(db)  // db.loadExtension() support
const { sqlite_version, vec_version } = db.prepare(
  'select sqlite_version() as sqlite_version, vec_version() as vec_version;'
).get()
console.log(`SQLite version: ${sqlite_version}, vector version ${vec_version}`)

const items = [
  [1, [0.1, 0.1, 0.1, 0.1]],
  [2, [0.2, 0.2, 0.2, 0.2]],
  [3, [0.3, 0.3, 0.3, 0.3]],
  [4, [0.4, 0.4, 0.4, 0.4]],
  [5, [0.5, 0.5, 0.5, 0.5]]
]
const query = [0.3, 0.3, 0.3, 0.3]

db.exec('CREATE VIRTUAL TABLE vec_items USING vec0(embedding float[4])')

const insertStmt = db.prepare(
  'INSERT INTO vec_items(rowid, embedding) VALUES (?, ?)'
)

for (const [id, vector] of items) {
  const rowId = BigInt(id)
  const embedding = new Uint8Array(Float32Array.from(vector).buffer)
  insertStmt.run(rowId, embedding)
}

const rows = db.prepare(
  `
  SELECT
    rowid,
    distance
  FROM vec_items
  WHERE embedding MATCH ?
  ORDER BY distance
  LIMIT 3
`
).all(new Uint8Array(Float32Array.from(query).buffer))

console.log(rows)

> sqlite-vec /Users/himself65/Code/node/out/Debug/node --experimental-sqlite ./index.mjs
(node:72169) ExperimentalWarning: SQLite is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
SQLite version: 3.46.0, vector version v0.1.1
[
  { rowid: 3, distance: 0 },
  { rowid: 4, distance: 0.19999998807907104 },
  { rowid: 2, distance: 0.20000001788139343 }
]

[himself65]

Found another small issue: nodejs/core-validate-commit#122

[RedYetiDev]

Found another small issue: nodejs/core-validate-commit#122

Duplicate of nodejs/core-validate-commit#121

[himself65]

I will re start this PR

[himself65]

I have no idea how to write a test for load extension now.

[himself65]

/cc @nodejs/sqlite (no such group?)

anyway I cc @nodejs/tsc since this is a breaking change

[codecov]

Codecov Report

Attention: Patch coverage is 69.44444% with 22 lines in your changes missing coverage. Please review.

Project coverage is 88.52%. Comparing base (3c2da4b) to head (08280ba).
Report is 54 commits behind head on main.

Files with missing lines	Patch %	Lines
src/node_sqlite.cc	69.44%	8 Missing and 14 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #53900      +/-   ##
==========================================
+ Coverage   87.99%   88.52%   +0.53%     
==========================================
  Files         656      657       +1     
  Lines      188999   189929     +930     
  Branches    35981    36472     +491     
==========================================
+ Hits       166301   168133    +1832     
+ Misses      15865    14992     -873     
+ Partials     6833     6804      -29     

Files with missing lines	Coverage Δ
src/node_errors.h	85.00% <ø> (ø)
src/node_sqlite.h	70.00% <ø> (ø)
src/node_sqlite.cc	80.25% <69.44%> (-0.93%)	⬇️

... and 108 files with indirect coverage changes

[benjamingr]

cc @cjihrig

[cjihrig]

since this is a breaking change

I took a quick look at the code. I'm not sure what makes this a breaking change though.

I do think the TSC needs to decide if they want to support loading arbitrary native code in this manner. It wouldn't be the first way that Node supports loading native code, so maybe it's fine.

I have no idea how to write a test for load extension now.

Is it possible to create a very trivial extension that can be accessed as a test fixture?

[tniessen]

I do think the TSC needs to decide if they want to support loading arbitrary native code in this manner.

It definitely must be disabled when the experimental permission model is enabled, just like native add-ons etc.

[himself65]

For the native test, I'm not sure if we should build an SQLite binding from scratch or if we should just copy some community dll/so/lib to fixtures.

[targos]

We also need a test that tries to load an unexpected file (that is not a sqlite extension). It should throw, not crash the process.

[jakecastelli]

I'm not sure if we should build an SQLite binding from scratch or if we should just copy some community dll/so/lib to fixtures.

IMO we should build from scratch to avoid any potential attack and make update tests fixture easier if we ever need to update them.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63903/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63905/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63911/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63916/

[mcollina]

lgtm

[cjihrig]

FYI - I think test/parallel/test-permission-sqlite-load-extension.js is legitimately failing on Windows. I'm not sure if it's related to the use of common.escapePOSIXShell in that test or what.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63919/

[himself65]

not sure how to fix it right now. setting up local env on my local windows pc

[himself65]

ok the error is legitimately falling, testing on my windows locally, I think because wrong usage of the cp api

[himself65]

OK, I found because \n on windows will truncate the script code, remove it

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63985/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63993/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63994/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63995/

[himself65]

will merge this today

[himself65]

Landed in 5c2f599