pkgsStatic.postgresql: fix build #345289

wolfgangwalther · 2024-09-29T13:02:50Z

The underlying problem was fixed as a side-effect of 7797728, for reasons unknown to me. In the current state, it's enough to disable a few breaking dependencies to make the build pass.

Note, that this builds the full package, including backend. However, the backend is not working, yet: Loading shared modules, which PostgreSQL heavily depends is still broken. Further, all binaries in the default output, even client binaries such as psql, are currently dynamically linked against libpq.so. While the current autoconf based build system doesn't support changing this, this might be possible in the future with meson.

However, not all is bad: Fixing the build allows using the static libpq.a library, which is probably the one thing that most users want from pkgsStatic.postgresql anyway.

Resolves #191920

Note: I also experimented with a meson-based static build, only to then notice that autoconf works with the basics as well. Making the switch to meson eventually should make things better, though: I was able to build a working backend, too, at least enough to run basic test suites for postgresql drivers with this. We can't fully switch to meson as a build system for 16 or 17, yet, because JIT will not work correctly. The meson build system doesn't build the required bitcode files, yet.

Things done

Add a 👍 reaction to pull requests you find important.

Ma27

Looks good to me, thank you!

However, this will need to target staging.

@alyssais do you want to take a look? Would wait a few days with merging.

pkgs/servers/sql/postgresql/generic.nix

wolfgangwalther · 2024-09-29T18:17:36Z

However, this will need to target staging.

I hope that we don't need any regular rebuilds and only affect the static build, that's why I targeted master. If my latest fix doesn't do that, yet, I will look at the eval diff - could only be some whitespace in the script or so.

pkgs/servers/sql/postgresql/generic.nix

alyssais · 2024-09-29T18:28:57Z

Would be good to understand what's wrong with the broken dependencies. Are they not compatible with being built statically in some fundamental way? If so, that should be marked in badPlatforms (see e.g. systemd), and then we should use lib.meta.availableOn rather than using an explicit isStatic check, so that platform info for those dependents can have a single source of truth.

wolfgangwalther · 2024-09-29T18:30:07Z

Would be good to understand what's wrong with the broken dependencies. Are they not compatible with being built statically in some fundamental way?

They both build fine in pkgsStatic, but PostgreSQL in combination doesn't.

(Edit: I will dig out the error messages and add them as comments or so, let me see)

wolfgangwalther · 2024-09-29T18:58:48Z

Tried to build with each of (icu, linux-pam) and put the error messages into a comment. I also tried some variations of NIX_LDFLAGS listing the various libs in both package's pkg-config files (of course, both are missing Libs.private and Requires.private ...), but wasn't able to fix it that way.

wolfgangwalther · 2024-09-29T18:59:47Z

However, this will need to target staging.

Done.

Ma27

Good enough for me now.

But if the server part is inherently broken on static builds, would it make sense to entirely skip the build? Haven't checkout how good this is doable though.

wolfgangwalther · 2024-09-29T20:22:15Z

But if the server part is inherently broken on static builds, would it make sense to entirely skip the build? Haven't checkout how good this is doable though.

I think it would be more invasive to skip the build than I'd want to go at this stage. I just tried, but one thing leads to the next and it will result in quite a few isStatic conditions all over the file.

I'd rather move towards a separate libpq package eventually (see #61580 (comment)). I'd still do the fix as proposed here, with the intention to eventually make the static postgresql backend actually work.

alyssais · 2024-09-29T20:56:07Z

pkgs/servers/sql/postgresql/generic.nix

+      #   /nix/store/3s55icpsbc36sgn7sa8q3qq4z6al6rlr-linux-pam-static-x86_64-unknown-linux-musl-1.6.1/lib/libpam.a(pam_audit.o):
+      #     in function `pam_modutil_audit_write':(.text+0x571):
+      #     undefined reference to `audit_close'
+      ++ lib.optionals (stdenv'.hostPlatform.isLinux && !stdenv'.hostPlatform.isStatic) [ "--with-pam" ]


This probably needs the following in its .pc file (upstream), if you're interested in fixing it:

Requires.private: audit

Given it's autotools, something like this: rockdaboot/libpsl#246

I'll have a look, but I'm not sure how useful that would be. I just looked and pkgsStatic.linux-pam contains only static libraries. But AFAICT, PAM is built around the idea of loading those authentication modules as shared libraries at run-time, depending on config. You'd probably have to link whatever authentication module you're going to use statically into the binary as well, but I'm not sure whether PAM would actually make use of that and how you'd configure it in that case.

Maybe it's a "linux-pam inherently doesn't make sense in pkgsStatic" as you earlier asked for, despite it being built fine on it's own.

3 weeks ago, upstream merged a PR to build with meson. meson generally generates better .pc files. So it might be worth trying that.

Yes, that should do it.

A new version of linux-pam was released, which only supports meson. I wasn't able to make that work, yet, so couldn't try whether that would improve this static build. Something for another day.

Work at all or work with postgres?

Couldn't make linux-pam build with meson, so was not able to update linux-pam itself in the first place. I never got to testing with postgres.

Couldn't make linux-pam build with meson, so was not able to update linux-pam itself in the first place.

I can push my intermediate state of that, if you're interested.

wolfgangwalther · 2024-09-30T07:03:05Z

pkgs/servers/sql/postgresql/generic.nix

+      # Building with icu in pkgsStatic gives tons of "undefined reference" errors like this:
+      #   /nix/store/452lkaak37d3mzzn3p9ak7aa3wzhdqaj-icu4c-74.2-x86_64-unknown-linux-musl/lib/libicuuc.a(chariter.ao):
+      #    (.data.rel.ro._ZTIN6icu_7417CharacterIteratorE[_ZTIN6icu_7417CharacterIteratorE]+0x0):
+      #    undefined reference to `vtable for __cxxabiv1::__si_class_type_info'
+      (lib.withFeature (!stdenv'.hostPlatform.isStatic) "icu")


For icu, there has been a PR merged a week ago which adds some Requires.private and Libs.private to their pkg config files.

So maybe once this is available to us, this will be possible.

Might be worth backporting now so we remember. (But doesn't have to block this PR even if we do.)

I just tried this... but there is something odd going on. Those .private modifiers are only added for shared linking, not when linking statically. This is kind of the opposite of what we need. I'll try to dig deeper later and possibly raise this upstream.

wolfgangwalther · 2024-11-28T19:57:32Z

Rebased, now it's simply a matter of disabling the new feature flags we have. Should be good to go, nothing more we can do here for now.

Ma27 · 2024-11-30T12:07:59Z

The idea was to have a package that builds at least so that you can the libpq.a, correct?
For me, it still fails with

***
ERROR: `xmllint' is missing on your system.
***
make[3]: *** [Makefile:72: postgres-full.xml] Error 1
make[3]: Leaving directory '/build/postgresql-17.2/doc/src/sgml'
make[2]: *** [Makefile:8: all] Error 2
make[2]: Leaving directory '/build/postgresql-17.2/doc/src'
make[1]: *** [Makefile:16: all] Error 2
make[1]: Leaving directory '/build/postgresql-17.2/doc'
make: *** [GNUmakefile:16: world-doc-recurse] Error 2
make: *** Waiting for unfinished jobs....

wolfgangwalther · 2024-11-30T13:12:04Z

The idea was to have a package that builds at least so that you can the libpq.a, correct?

Absolutely 🙈

Will look into it.

The underlying problem was fixed as a side-effect of [1], for reasons unknown to me. In the current state, it's enough to disable a few breaking dependencies to make the build pass. Note, that this builds the full package, including backend. However, the backend is not working, yet: Loading shared modules, which PostgreSQL heavily depends is still broken. Further, all binaries in the default output, even client binaries such as psql, are currently dynamically linked against libpq.so. While the current autoconf based build system doesn't support changing this, this might be possible in the future with meson. However, not all is bad: Fixing the build allows using the static libpq.a library, which is probably the one thing that most users want from pkgsStatic.postgresql anyway. Resolves NixOS#191920 [1]: 7797728

wolfgangwalther · 2024-11-30T16:02:44Z

For me, it still fails with

Right, so while I opened this PR against staging, I was still testing on master, where v16 was the default. This only breaks for v17, where we added new tools for building the docs:

nixpkgs/pkgs/servers/sql/postgresql/generic.nix

Line 113 in 60a1ffe

    
           ++ lib.optionals (atLeast "17") [ bison flex perl docbook_xml_dtd_45 docbook-xsl-nons libxslt ];

Seems like we should have technically added a nativeBuildInput of libxml2, too. But since this was in buildInputs already, and we don't build with strictDeps = true - this was not a problem as long as not building cross. Also confirmed that postgresql_17 is failing with strictDeps for the same reason on master.

Thus: I enabled strictDeps and added libxml2 as a nativeBuildInput. This should fix it and prevent regressions.

pkgs/servers/sql/postgresql/generic.nix

wolfgangwalther requested a review from thoughtpolice as a code owner September 29, 2024 13:02

wolfgangwalther requested review from Ma27 and removed request for thoughtpolice September 29, 2024 13:03

wolfgangwalther mentioned this pull request Sep 29, 2024

Make a libpq package #61580

Open

4 tasks

ofborg bot requested review from ivan, danbst, globin and thoughtpolice September 29, 2024 14:50

ofborg bot added 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 501+ 10.rebuild-darwin: 2501-5000 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Sep 29, 2024

Ma27 requested changes Sep 29, 2024

View reviewed changes

wolfgangwalther commented Sep 29, 2024

View reviewed changes

pkgs/servers/sql/postgresql/generic.nix Outdated Show resolved Hide resolved

wolfgangwalther force-pushed the postgresql-static branch from e176720 to 17abff5 Compare September 29, 2024 18:16

wolfgangwalther commented Sep 29, 2024

View reviewed changes

pkgs/servers/sql/postgresql/generic.nix Show resolved Hide resolved

wolfgangwalther marked this pull request as draft September 29, 2024 18:26

wolfgangwalther force-pushed the postgresql-static branch from 17abff5 to d578a79 Compare September 29, 2024 18:26

wolfgangwalther changed the base branch from master to staging September 29, 2024 18:27

alyssais added the 2.status: merge conflict This PR has merge conflicts with the target branch label Sep 29, 2024

wolfgangwalther force-pushed the postgresql-static branch from d578a79 to 53b64d8 Compare September 29, 2024 18:28

wolfgangwalther force-pushed the postgresql-static branch from 53b64d8 to 128d27c Compare September 29, 2024 18:56

wolfgangwalther marked this pull request as ready for review September 29, 2024 18:56

ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Sep 29, 2024

Ma27 approved these changes Sep 29, 2024

View reviewed changes

ofborg bot requested a review from Ma27 September 29, 2024 20:35

alyssais approved these changes Sep 29, 2024

View reviewed changes

alyssais reviewed Sep 29, 2024

View reviewed changes

wolfgangwalther commented Sep 30, 2024

View reviewed changes

wegank added 12.approvals: 1 This PR was reviewed and approved by one reputable person 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in the package labels Sep 30, 2024

This was referenced Nov 18, 2024

postgresql: split -lib and -dev outputs cleanly #294504

Merged

postgresql: add new feature options #358238

Merged

wolfgangwalther force-pushed the postgresql-static branch from 128d27c to c1bbf8c Compare November 28, 2024 19:55

wegank removed 12.approvals: 1 This PR was reviewed and approved by one reputable person 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in the package labels Nov 29, 2024

wolfgangwalther added 2 commits November 30, 2024 17:02

postgresql: enable strictDeps

445371f

wolfgangwalther force-pushed the postgresql-static branch from c1bbf8c to 593bc6e Compare November 30, 2024 16:02

alyssais reviewed Nov 30, 2024

View reviewed changes

pkgs/servers/sql/postgresql/generic.nix Outdated Show resolved Hide resolved

wolfgangwalther force-pushed the postgresql-static branch from 593bc6e to 6547f93 Compare November 30, 2024 16:46

postgresql: add -export_dynamic on darwin only for v16

a77adf3

wolfgangwalther force-pushed the postgresql-static branch from 6547f93 to a77adf3 Compare November 30, 2024 16:53

Ma27 merged commit fc5a956 into NixOS:staging Dec 1, 2024
31 of 33 checks passed

wolfgangwalther mentioned this pull request Dec 1, 2024

static build of postgresql (and thus libpq) fails #191920

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pkgsStatic.postgresql: fix build #345289

pkgsStatic.postgresql: fix build #345289

wolfgangwalther commented Sep 29, 2024

Ma27 left a comment

wolfgangwalther commented Sep 29, 2024

alyssais commented Sep 29, 2024

wolfgangwalther commented Sep 29, 2024 •

edited

Loading

wolfgangwalther commented Sep 29, 2024

wolfgangwalther commented Sep 29, 2024

Ma27 left a comment

wolfgangwalther commented Sep 29, 2024

alyssais Sep 29, 2024

alyssais Sep 29, 2024

wolfgangwalther Sep 30, 2024 •

edited

Loading

wolfgangwalther Sep 30, 2024

alyssais Sep 30, 2024

wolfgangwalther Nov 28, 2024

alyssais Nov 30, 2024

wolfgangwalther Nov 30, 2024 •

edited

Loading

wolfgangwalther Nov 30, 2024 •

edited

Loading

wolfgangwalther Sep 30, 2024

alyssais Sep 30, 2024

wolfgangwalther Sep 30, 2024

wolfgangwalther commented Nov 28, 2024

Ma27 commented Nov 30, 2024

wolfgangwalther commented Nov 30, 2024

wolfgangwalther commented Nov 30, 2024

pkgsStatic.postgresql: fix build #345289

pkgsStatic.postgresql: fix build #345289

Conversation

wolfgangwalther commented Sep 29, 2024

Things done

Ma27 left a comment

Choose a reason for hiding this comment

wolfgangwalther commented Sep 29, 2024

alyssais commented Sep 29, 2024

wolfgangwalther commented Sep 29, 2024 • edited Loading

wolfgangwalther commented Sep 29, 2024

wolfgangwalther commented Sep 29, 2024

Ma27 left a comment

Choose a reason for hiding this comment

wolfgangwalther commented Sep 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wolfgangwalther Sep 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wolfgangwalther Nov 30, 2024 • edited Loading

Choose a reason for hiding this comment

wolfgangwalther Nov 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wolfgangwalther commented Nov 28, 2024

Ma27 commented Nov 30, 2024

wolfgangwalther commented Nov 30, 2024

wolfgangwalther commented Nov 30, 2024

wolfgangwalther commented Sep 29, 2024 •

edited

Loading

wolfgangwalther Sep 30, 2024 •

edited

Loading

wolfgangwalther Nov 30, 2024 •

edited

Loading

wolfgangwalther Nov 30, 2024 •

edited

Loading