| layout | title | permalink | redirect_to |
|---|---|---|---|
default |
About |
/about |
Wildly successful, and broadly useful, SQLite has solidified its place in modern technology stacks, embedded in nearly any computing device you can think of. Its open source nature and public domain availability make it a popular choice for modification to meet specific use cases.
It seems to us that there are two obvious avenues to modify SQLite: forking the code to add features directly to it, or running it on top of a modified OS. History suggests that neither of these work well. The way we see it, this is a result of one major limitation of the software: SQLite is open source but does not accept contributions, so community improvements cannot be widely enjoyed.
We want to see a world where everyone can benefit from all of the great ideas and hard work that the SQLite community contributes back to the codebase. Community contributions work well, because we’ve done it before. If this was possible, what do you think SQLite could become?
SQLite is gaining ground in edge use cases, since it is fast, embeddable, and matches well the read-mostly, low-to-medium volume of data use cases that often arise at the edge. But there is still the problem of how to make the data available in all nodes.
That is challenging to do without support from the core database. Without proper hooks, existing solutions either build a different database around SQLite (like dqlite, rqlite, ChiselStore), or have to replicate at the filesystem layer (LiteFS).
What if we could do it natively?
Recently, Linux has gained a new supposedly magical interface called [io_uring]. It leverages Asynchronous I/O and has been slowly but surely gaining adoption everywhere. Other databases like Postgres have already adopted it for asynchronous I/O, but this hasn’t made its way to SQLite. Supposedly because SQLite interfaces are synchronous. But with asynchronous runtimes and interfaces gaining popularity in both languages and the Kernel, is it time to add a new, async interface to SQLite, that plays well with io_uring?
Another innovation in the Linux Kernel is eBPF. That is a domain-specific VM that allows programs to execute in the kernel. Although still mainly used for tracing, there’s research about pushing complex data functions like B-Tree lookups entirely in the kernel, as close as possible to NVMe devices. What if SQLite could take advantage of that, and also be unbeatable in workloads that don’t fit in memory?
SQLite does support user-defined functions. But there are two big problems with how they are approached: first, functions are written in C, which is increasingly becoming a tall ask for most developers with safety in mind. WASM is growing in popularity, allowing developers to write functions in their preferred language and be safely executed.
Second, once a function proves itself generally useful, the “not Open Contribution” policy makes it difficult for it to be included in the standard distribution of SQLite.
We’ve decided that now is the time to take SQLite to the places where the community wants to be. Are you interested in building the future of SQLite? We’re kicking off our efforts on [GitHub], and you can find us on Discord.
There is nothing to be gained in reinventing greatness, so we will simply build upon that with a fork of core SQLite. Unlike SQLite, this fork will be both fully open source (MIT) and open to community contributions.
We are committed to preserving compatibility with everything that was previously written for SQLite. All of your favorite tools and libraries will continue to work as-is.
SQLite is a very well-tested piece of software. We admire that, and commit to preserving the existing test suite while expanding it for the new code we add.
We intend to add Rust to implement new capabilities, but not exclusively so. The existing C codebase already serves as a great foundation for interoperability with other languages and systems, and we don’t intend to change that.
We are strong believers in open source that is also open to community contributions. If and when SQLite changes its policy to accept contributions, we will gladly merge our work back into the core product and continue in that space.
We take our code of conduct seriously, and unlike SQLite, we do not substitute it with an unclear alternative. We strive to foster a community that values diversity, equity, and inclusion. We encourage others to speak up if they feel uncomfortable.