Make a libpq package

[nh2]

Right now nixpkgs only has postgresql, which contains much more than libpq.

By offering a libpq package like other distros, the closure size of many packages could be reduced significantly.

Even more so because postgresql currently depends on systemd. And that also means that no application or library using libpq can be built against musl, as systemd relies on glibc-only features and thus does not build with musl.

Tasks:

• Make postgresql -> systemd dependency configurable
• Add libpq package containing only libpq
• Change Haskell packages so that they depend on libpq instead of all of postgresql
• Change other packages that can benefit from this (help wanted for the other programming languages!)

[nh2]

CC @matthewbauer @peti for Haskell.

CC @FRidh for Python, would Python benefit from this?

[matthewbauer]

I thought the postgresql.lib output would do this for us?

[7c6f434c]

@matthewbauer yes for runtime closure size, no for systemd

[thoughtpolice]

FWIW, regarding closure size, I considered splitting this up before when I did significant Postgres work last year but ultimately went against it because multi-output .lib is supported for all Postgres expressions -- which does reduce the size a bit and worked well enough for dependent expressions.

That said something containing "only" libpq.so (for example, via an override to create a new expression/derivation) would be much smaller, certainly, but it's not clear that's the best use of work or time -- right now .lib for postgresql.lib (9.6) is barely 6mb and that's mostly due to it containing most of the default server extensions that are shipped with the build system (libpq.so is barely 200KB). Moving these out would probably reduce this quite a lot -- for example, they could just be part of .bin instead.

As for dependencies, well, they're pretty slim too:

a@link> nix path-info -rS ./result-lib
/nix/store/57i8nqp2vljg5h61gvk4p1ar9vzhxayj-libossp-uuid-1.6.2             29565976
/nix/store/5p8qcz6i13ymgf80kimhx2g9pb5r5nia-bash-4.4-p23                   29463504
/nix/store/lhngda6cn951z68v5rlh4dni6bb5sjw1-zlib-1.2.11                    28279616
/nix/store/s5sgkvhy2wdgn7fmsvq47srxjhfp9ld9-libxml2-2.9.9                  29985080
/nix/store/w7d4sfgxjpalphg3h9bink47yzhf4f96-postgresql-9.6.12-lib          41456208
/nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27                     28147008
/nix/store/yglqxz2as4ygai26d2hbfxqpf0shdh4l-openssl-1.0.2r                 31880056

But why do we need them?

a@link> nix path-info -r ./result-lib | grep -v postgresql | while read l; do nix why-depends ./result-lib "$l"; done

/nix/store/w7d4sfgxjpalphg3h9bink47yzhf4f96-postgresql-9.6.12-lib
╚═══lib/uuid-ossp.so: …LIBC_2.2.5.GLIBC_2.4./nix/store/57i8nqp2vljg5h61gvk4p1ar9vzhxayj-libossp-uuid-1.6.2/lib:/nix/sto…
    => /nix/store/57i8nqp2vljg5h61gvk4p1ar9vzhxayj-libossp-uuid-1.6.2
/nix/store/w7d4sfgxjpalphg3h9bink47yzhf4f96-postgresql-9.6.12-lib
╚═══lib/uuid-ossp.so: …LIBC_2.2.5.GLIBC_2.4./nix/store/57i8nqp2vljg5h61gvk4p1ar9vzhxayj-libossp-uuid-1.6.2/lib:/nix/sto…
    => /nix/store/57i8nqp2vljg5h61gvk4p1ar9vzhxayj-libossp-uuid-1.6.2
    ╚═══bin/uuid-config: …#!/nix/store/5p8qcz6i13ymgf80kimhx2g9pb5r5nia-bash-4.4-p23/bin/sh.##.##  OSSP…
        => /nix/store/5p8qcz6i13ymgf80kimhx2g9pb5r5nia-bash-4.4-p23
/nix/store/w7d4sfgxjpalphg3h9bink47yzhf4f96-postgresql-9.6.12-lib
╚═══lib/pgcrypto.so: …LIBC_2.4.GLIBC_2.2.5./nix/store/lhngda6cn951z68v5rlh4dni6bb5sjw1-zlib-1.2.11/lib:/nix/store/yglq…
    => /nix/store/lhngda6cn951z68v5rlh4dni6bb5sjw1-zlib-1.2.11
/nix/store/w7d4sfgxjpalphg3h9bink47yzhf4f96-postgresql-9.6.12-lib
╚═══lib/pgxml.so: …2.2.5.LIBXML2_2.4.30./nix/store/s5sgkvhy2wdgn7fmsvq47srxjhfp9ld9-libxml2-2.9.9/lib:/nix/store/xq…
    => /nix/store/s5sgkvhy2wdgn7fmsvq47srxjhfp9ld9-libxml2-2.9.9
/nix/store/w7d4sfgxjpalphg3h9bink47yzhf4f96-postgresql-9.6.12-lib
╚═══lib/_int.so: …LIBC_2.4.GLIBC_2.2.5./nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib.XXXXXXXXXXXXXXXX…
    => /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27
/nix/store/w7d4sfgxjpalphg3h9bink47yzhf4f96-postgresql-9.6.12-lib
╚═══lib/libpq.so.5.9: …LIBC_2.3.4.GLIBC_2.3./nix/store/yglqxz2as4ygai26d2hbfxqpf0shdh4l-openssl-1.0.2r/lib:/nix/store/x…
    => /nix/store/yglqxz2as4ygai26d2hbfxqpf0shdh4l-openssl-1.0.2r

bash, libuuid, zlib, and libxml are needed for auxiliary extensions. So we could actually achieve significant reductions here by just moving the server extensions out of the .lib output, which would yield nothing except glibc and openssl, I believe -- but these dependencies should be considered "practically free" since you always need them for anything at all. As a result you're looking at a realistic closure size of < 1mb with that change.

Therefore, I think a much better use of time would be to just slim down postgresql.lib and call it a day. And if that doesn't fix the itch, we can always alias libpq = postgresql.lib somewhere in the top-level and say "we have libpq"

[7c6f434c]

@nh2 Does overriding systemd argument to null resolve the first point?

[nh2]

@nh2 Does overriding systemd argument to null resolve the first point?

I don't know, but I'd rather make this an explicit flag as done in #61581 than using null.

[nh2]

@thoughtpolice Thanks for your analysis, that is good to know.

I think a much better use of time would be to just slim down postgresql.lib and call it a day.

That is not enough for musl support though because with the current setup where we build all of postgresql with --with-systemd, and then throw away everything that's not lib, it it fails to compile already before we can throw away anything.

So in my proposal, a libpq package would be built with enableSystemd = false.

It seems a good idea to me to give that "postgres without systemd build dependency" thing a name, and I found libpq the easiest solution because it solves both problems (buldability and client apps not depending on any postgresql server bits) in one go.

[thoughtpolice]

I don't follow the logic here. You want a systemd-less version of postgresql so you can use Musl for your stdenv and built packages. libpq actually does not need systemd, only the .bin outputs do, but they do not work with Musl. So if this was turned off or libpq was recompiled separately, you could avoid it. Therefore, libpq would override enableSystemd = false at all times and everything would link against it, but:

• This implies another rebuild of postgresql for the separate derivation, which is annoying and tiresome
• It requires everything depending on postgresql to be refactored to libpq in order to take any effect. Because everything currently implicitly relies on .lib to provide libpq, you must change it -- you can't simply override the postgresql.lib output to point to a completely different expression -- unless you want to rebuild every version of postgresql twice just so every postgresqlXX.lib output can be a completely different, recompiled piece of code.
• In fact you probably do want to recompile every server version twice anyway with this scheme, because people will probably want a libpq that is consistent with the version of postgresql they depend on. Something smells very fishy about having something like buildInputs = [ libpq postgresql_10 ] where libpq = postgresql96.lib or whatever. I'm sure libpq has also had several API/feature changes between versions, probably making this mandatory.
• And it doesn't really matter anyway because, in order to use Musl as the stdenv, you have to re-compile everything with musl anyway. So what did the libpq derivation get you, exactly? You might as well just override postgresql to enableSystemd = false and be done with it in this same case.

In other words: why should upstream glibc users, Hydra, Borg, etc always compile a second postgresql derivation with this feature disabled, and a substantial amount of changes to all postgres-dependent extensions? When the musl use case requires you recompiling it anyway.

Put another way -- what use cases does this particular scheme make easier/possible that aren't currently possible, and why is this the way it must be done?

Keep in mind that third parties outside use postgresql.lib as a buildInput as well, so this effectively creates a schism between upstream and any third party uses (now you have to override them awkwardly if you want upstream/a third party to behave the same, etc). Unless you just completely remove postgresql.lib and break downstreams, I guess. (It would actually be worse than that, I think -- and would result in confusing build-time errors, because mkDerivation doesn't consider the lack of a .lib to be a problem, it just assumes the files must be in .out instead, AKA it's not multi-output -- but those files won't be there!)

---

I'm not convinced this is a good plan yet. IMO it would be strictly better to simply disable systemd support unilaterally if stdenv.hostPlatforms.isMusl == true and simply put that in the default generic expression, and also clean up .lib as I explained earlier. Then:

• Everyone gets a smaller .lib closure, no matter what
• No major refactorings are needed across the codebase or to make postgres.lib dependent expressions depend on a new expression, or involve excessive recompilations. Everyone keeps using postgresql.lib
• Musl users automatically get a version that does not depend on systemd at all, transparently to them.

I may be misunderstanding something, but this seems somewhat fishy as an approach, to me. Also, the whole phrasing of "introduce libpq so I can use musl" seems like a classic X-is-Y problem... The actual problem is not that libpq must be separate -- it's that postgresql currently has dependent inputs that are incompatible for you. That's a much narrower problem to solve with a fix that is substantially easier, and much less invasive.

[nh2]

IMO it would be strictly better to simply disable systemd support unilaterally if stdenv.hostPlatforms.isMusl == true and simply put that in the default generic expression, and also clean up .lib as I explained earlier.

@thoughtpolice What you propose sounds like a good plan to me.

One question though:

Are there situations where some programs may want to use only the postgresql server libraries?

E.g. should there be postgresql.lib (being essentially libpq, as you suggest), and e.g. postgresql.server-lib (for programs that use server libs but don't need the binaries)? Or is that an uncommon use case that can be handled later, if somebody needs it in the future?

[thoughtpolice]

I am not sure about that in the case of first-party things shipped with PostgreSQL. I do not think there's any reasonable way for someone to interface with/link against those meaningfully. They can only be used by CREATE EXTENSION in the server process.

For certain third party extensions, I know they do ship libraries and headers that can be consumed by other extensions. But that case is handled fine: because those expressions will act normally, have .lib and .dev outputs, and be part of your buildInputs if you need them.

I don't think there are any realistic use cases where someone would want all the server libs, but not the binaries in .bin. And if you did -- you'd probably want the include files anyway, but there's another catch here which is that they are inside .bin too, not in a .dev output. So having a .server-libs output is a net loss: you still need .bin anyway for the header files.

---

So, if we cleaned up things as I mentioned earlier -- by putting server extensions in .bin -- and someone did want them and did want to use them in some theoretical scenario, we can just say "use the .bin output in your buildInputs and it will work". No need for a separate .server-libs.

But I think that's a particular, rare-or-impossible enough bridge that we can cross that one when (if) we get to it.

[matthewbauer]

I don't think this is a good idea. systemd.lib is only 1.8M and seems like a reasonable thing to have a runtime dependency on:

$ du -sch $(nix-store -qR /nix/store/558zw2lrdiqwgy0i5vq779j700iph882-postgresql-9.6.12-lib)
152K	/nix/store/r4bbg2bjmsvyachy2mn97xri6bcib4lj-zlib-1.2.11
160K	/nix/store/kq8jb3g4xzfbc3d7m818ri3293j2nsy3-libossp-uuid-1.6.2
1.4M	/nix/store/2ch3pwfv40ls6qhdmzwsz9wky9pq6cyl-bash-4.4-p23
1.7M	/nix/store/hsh8afbalh1blsj2s4n8g2dvaj2j5aai-libxml2-2.9.9
3.6M	/nix/store/2wj5dw9sslnqxkznyzp8jhic8ywybs72-openssl-1.0.2r
6.2M	/nix/store/558zw2lrdiqwgy0i5vq779j700iph882-postgresql-9.6.12-lib
30M	/nix/store/siks2gcfwx6qwh27m7c5r5lixcr621bd-glibc-2.27
43M	total

Splitting packages into multiple parts makes things much more confusing to reason about. Just because other distros do it doesn't mean it's a good idea. Multiple outputs are there to avoid this mess. We should use package splitting as a last result!

On the other hand, disabling systemd support when you are building with Musl seems reasonable. But it would be much better to just fix musl support in systemd.

[thoughtpolice]

@matthewbauer FWIW, see #61581 for my proposed changes to @nh2's proposed changes -- which effectively do what you ask (systemd is always available, unless stdenv.hostPlatform.isMusl == true || stdenv.isDarwin).

I also do not think systemd upstream has any interest in building with musl; some googling leads to various patches which seem to be rather invasive, and I think it would be better to avoid cluttering our own systemd patchset with a ~dozen more. It's possible to get a subset of systemd's library code working with musl (a lot of the most useful stuff like socket activation helper code, sd_log etc are all in just a single header file), but anything that truly needs libudev or libsystemd is going to be tricky.

[danbst]

I can think of another good reason of splitting postgresql and postgresql_lib. Currently we have postgres = postgresql96, and we can't bump it because it will bump server version for those, who use postgresql as server. Well... we can bump it, but that will be breaking change for many (last time I've checked there were tens of direct pg server usages through postgresql)...

If we could have postgresql_lib as separate top-level attribute, we could bump it's major version every year, as libpq has backwards compat. Is there any need for libpq9.6 when we already have libpq11?

[danbst]

I wrote "splitting" but in fact I just want a separate top-level attribute for libpq. Which should be libpq = postgresql_latest.lib;. So I'm all for multiple outputs, just libpq deserves a separate reference.

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[nh2]

still important to me

[stale]

I marked this as stale due to inactivity. → More info

[YoshiRulz]

I think the first task in the OP can be checked off?

[szlend]

The postgresql.lib output also doesn't include pkg-config definitions which can screw up builds from non-sandboxed environments like the devshell.

[szlend]

In my opinion, a separate libpq derivation still makes the most sense. Here's why:

• libpq is one of the most popular libraries out there. It's a dependency of every single application that connects to a postgres database. As such I feel it deserves special considerations.
• libpq will likely be used in variety of different ways (static linking, cross-compilation, etc.). Being an output of the entire postgresql derivation causes additional friction as it has to build a ton of additional dependencies that may or may not work in those configurations. By minimizing the number of build-time dependencies we make the whole build process much smoother.
  • Having something like pkgsCross.aarch64-multiplatform.libpq just work is much nicer than playing whack-a-mole with (pkgsCross.aarch64-multiplatform.postgresql.overrides { ... }).lib, trying to figure out which magic set of overrides it will work with (or break in the next nixpkgs release).
• Changing postgresql.lib to only include libpq is a breaking change and has so far caused a ton of bike-shedding conversations on what it should be, while 4 years later we're the only major package repository that doesn't ship this.
• We don't need multiple packages of libpq. libpq is guaranteed to be backwards compatible so it only needs to be based on postgresql_<latest>.src.
• Having one additional package that builds in under a minute in a package repository of 80k+ is not a serious concern
• libpq is really easy to package on its own.

I opened a Draft MR based on @teburd's work to hopefully move something forward: #234470

[rupurt]

Well said @szlend. Couldn't agree more

[wolfgangwalther]

I'm summarizing the current state:

• Since postgresql: Make systemd support configurable #61581 systemd support is only enabled if supported on the platform.
• Since postgresql: split -lib and -dev outputs cleanly #294504 the lib output is only 724K.
• In pkgsStatic.postgresql: fix build #345289, pkgsStatic.postgresql will successfully build - which surely was a big motivator for this issue, too.

That's all stuff that everybody seemed to agree on was a good idea.

It was also proposed to have a separate libpq attribute to be able to update libpq users independently from users of the server package. libpq is backwards-compatible in general, so it generally makes sense to use the latest libpq whereever possible. This would make it easier to do so.

In #294504, we discussed different approaches without building a separate derivation, but ultimately took it out before merge, because this approach just doesn't work well:

• libpq = postgresql_latest.lib will not do it, because downstream users might also need the -dev output, e.g. for pkg-config files.
• libpq = postgresql_latest is pointless - because this would include the whole server, which just opens the door to accidentally use it for things where postgresql should have been used - thus giving us no better guarantee about updatability of libpq to a new major version.
• Any variant where libpq uses the pg_config from postgresql_latest (i.e. via symlinking, copying libs to a new derivation or so) would always have the problem that pg_config will refer to the postgresql output directly, not to libpq. This will break any reference checking (e.g. disallowedReferences = libpq).
  TLDR: This isn't going to be good.

Multiple proposals / PRs have been made to build libpq as a separate derivation and not just an alias to postgresql_latest. @szlend provided good arguments in #61580 (comment).

Those proposals don't say anything about whether libpq.so or libpq.a would remain in the postgresql package at the same time? There were some arguments that seemed to assume that this wasn't the case, and thus downstream users would need to use buildInputs = [ libpq postgresql ], if they needed more than what is provided by the libpq. I don't think this makes sense - the postgresql package should still provide the same libraries. Once you depend on the full package, the additional 700K in size really don't matter at all.

This leaves us with only argument against building a separate libpq: Number of (re-)builds. It was argued that a separate package would lead to more rebuilds required. I think the opposite is true. I worked on the postgresql package a lot in the last couple of months, and every change to its derivation causes a huge number of rebuilds. Most of the dependencies are only because downstream users depend on libpq - but almost none of my changes were lippq related. CI having to rebuild more than 5k packages each time, just because we make a small change to the server part? A waste of resources. If we were to split a separate libpq derivation, I'm pretty sure it would be changed much less - and the postgresql package should have much fewer reverse dependencies. Win, win!

A separate libpq would surely help with cross-compiling efforts, but we could take this one step further: One hazard for cross-compilation is surely pg_config, because it can't be executed if the hostPlatform is different enough from the buildPlatform. We could just not provide pg_config for libpq and instead rely on pkg-config instead. This would mean that only packages not using pg_config could depend on libpq, thus increasing the likelihood of support cross compilation. All packages, which rely on pg_config, can still use postgresql like they do right now. Furthermore, we could use the new meson based build system for the libpq package, which was shipped for the first time with PostgreSQL 16. We can't really use it for postgreql, yet, because it's not feature complete - but for everything libpq related it should be more than enough. In general cross-compiling should be much better supported if done via meson.

[Ma27]

more, we could use the new meson based build system for the libpq package, which was shipped for the first time with PostgreSQL 16.

Does that mean we can build libpq-only with meson without manual make hackery as in https://github.com/NixOS/nixpkgs/pull/234470/files#diff-e2876a6fc20e0333a7dd9be0c4b081f9ad0c3c5b12aa032469e42dda2a291701R39 ?

[wolfgangwalther]

Does that mean we can build libpq-only with meson without manual make hackery

Unfortunately, at least right now, not. I didn't even find a way to do this with meson hackery, yet. So we should probably settle for a autoconf build at first and then iterate later.

[Ma27]

I worked on the postgresql package a lot in the last couple of months, and every change to its derivation causes a huge number of rebuilds.

Just to make sure I follow your reasoning here, that means you'd suggest a fresh build (that just uses postgresql_latest.src[1]) and no derivation that extracts the relevant parts out of postgresql.lib & postgresql.dev?

Unfortunately, at least right now, not. I didn't even find a way to do this with meson hackery, yet. So we should probably settle for a autoconf build at first and then iterate later.

Even though I'm not too happy with this, I agree.

Is there any upstream bugreport already to make this better? IIRC part of the problem with the current build system was that there's a make target for libpq, but it pulls in the server part anyways(?)

[1] Now that I think of it, it might make sense to allow temporarily hardcoding the source in libpq to fast-track patches for server-only vulns assuming that the rebuild number of pkgs.postgresql drops low enough.

[wolfgangwalther]

Just to make sure I follow your reasoning here, that means you'd suggest a fresh build (that just uses postgresql_latest.src[1]) and no derivation that extracts the relevant parts out of postgresql.lib & postgresql.dev?

Correct.

Is there any upstream bugreport already to make this better? IIRC part of the problem with the current build system was that there's a make target for libpq, but it pulls in the server part anyways(?)

Not sure about the autoconf build there, but that's precisely the problem with meson right now. You can build libpq + client binaries only, but installing afterwards will trigger a full build of all server stuff as well.

I don't think there is any upstream report about this, yet. It's on my list, but I would like to figure out a possible solution first and then propose that at the same time.

[1] Now that I think of it, it might make sense to allow temporarily hardcoding the source in libpq to fast-track patches for server-only vulns assuming that the rebuild number of pkgs.postgresql drops low enough.

I can't think of a nice solution to allow this temporarily. The cleanest approach would be to reference the same source twice explicitly and update it independently, without relying on postgresql_latest.

This would also allow merging a new major version directly into master, because it wouldn't trigger the full rebuild via libpq. While we should update libpq asap, it's kind of annoying to deal with both in the same PR (e.g. #345260).

[szlend]

Decoupling libpq from postgresql_latest.src makes a lot of sense given the reasons above. I'm fully in support of this!