Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to specify a location to a flatpak ref for a application? #78

Closed
SpiderUnderUrBed opened this issue Sep 17, 2024 · 27 comments · Fixed by #80
Closed

Ability to specify a location to a flatpak ref for a application? #78

SpiderUnderUrBed opened this issue Sep 17, 2024 · 27 comments · Fixed by #80
Assignees
Labels
enhancement New feature or request

Comments

@SpiderUnderUrBed
Copy link

Hello, I want to install sober declaratively using nix flatpak, however sober is not published on flathub or any repo, so I was wondering how i could specify the link to the ref file and have it automatically install? is this possible currently? if not then can it be worked on?
Here is the how they ship their flatpak ref:
https://sober.vinegarhq.org/sober.flatpakref

@gmodena
Copy link
Owner

gmodena commented Sep 18, 2024

Hey @SpiderUnderUrBed ,

I'm working on adding the capability to install from local flatpaks (as a way to improve testing).

I need to look into supporting generic URI, but that sounds like a nice feature to have.

@gmodena gmodena self-assigned this Sep 18, 2024
@gmodena gmodena added the enhancement New feature or request label Sep 18, 2024
@gmodena
Copy link
Owner

gmodena commented Oct 4, 2024

I need to look into supporting generic URI, but that sounds like a nice feature to have.

Supporting installation from a flatpakref URL is pretty straightforward.

What complicates things slightly is this behavior:
flatpak/flatpak#5460

flatpak fails to --reinstall or skip already installed apps from a flatpakref.
You can reproduce this issue by trying the following:

flatpak --user install https://sober.vinegarhq.org/sober.flatpakref
flatpak --user install --reinstall https://sober.vinegarhq.org/sober.flatpakref

The second command should throw an error. This would be nix-flatpak install/update lifecycle.

The only workaround I can think of is downloading the .flatpakref file and mapping the URL to the declared Name and RuntimeRepo.

@SpiderUnderUrBed
Copy link
Author

SpiderUnderUrBed commented Oct 4, 2024

What does this look like in nix? Like you specify:
https://sober.vinegarhq.org/sober.flatpakref
as a app, does nix flatpak even support that?

Also does this mean generic URI's cant be supported? A way of implementation would require a potentially different install/update lifecycle for file paths. Or URLS to flatpakrefs.

@Bear-03
Copy link

Bear-03 commented Oct 6, 2024

This is my current workaround for this issue.

I noticed that .flatpakref and .flatpakrepo files have very similar keys, the only significant thing that changes is the header itself, so I just added the ref file as an input to my flake, and then I replace the header before feeding it to nix-flatpak so it can read it as a .flatpakrepo.

Seems to work flawlessly so far.

@gmodena
Copy link
Owner

gmodena commented Oct 6, 2024

The only workaround I can think of is downloading the .flatpakref file and mapping the URL to the declared Name and RuntimeRepo.

I did some further testing. This might require some adjustments to the state management logic to avoid breaking configurations that declare services.flatpak.uninstallUnmanaged = true.

When you run flatpak install --from <flatpakref>, a new repository is added to Flatpak's configuration.

For example, when I install with

flatpak --user install --from https://sober.vinegarhq.org/sober.flatpakref
I'm promted with:

Should the remote be kept for future installations? [Y/n]

Even If I select n, flatpak will still add a remote:

$ flatpak remotes
Name         Options
flathub      system
flathub-beta system
flathub      user
sober-origin user,no-enumerate

Unless the repository is explicitly declared in services.flatpak.remotes, it will be deleted during the next activation, and the application will also be deleted. Howevert, unless explicitly removed from the packages declaration, the application would still be re-installed as part of the activation. That's a rather wasteful operation IMHO (the application would need to be re-downloaded each time an activation is triggered).

@Bear-03 gave a neat example of how to automate remote declaration configs.

A flatpakref will contain all the info you need to declare your remote and application manually. Let's take a look at Sober:

$ curl -s "https://sober.vinegarhq.org/sober.flatpakref"
[Flatpak Ref]
Title=Sober
Name=org.vinegarhq.Sober
Url=https://sober.vinegarhq.org/repo/
SuggestRemoteName=sober
Homepage=https://sober.vinegarhq.org
Icon=https://sober.vinegarhq.org/sober.svg
RuntimeRepo=https://dl.flathub.org/repo/flathub.flatpakrepo
IsRuntime=false
GPGKey=<redacted>

With this, you can declare a remote and package as:

services.flatpak.remotes = lib.mkOptionDefault [{
    name = "sober"; # from SuggestRemoteName
    location = "https://sober.vinegarhq.org/repo/"; # from Url
  }];

services.flatpak.packages = [
   { appId = "org.vinegarhq.Sober"; origin = "sober"; }
];

Where appId comes from the flatpakref Name field and origin is the name of the remote we added.

You could follow the example @Bear-03 provided and automate parsing of the flapakref.

All in all, I think this approach is less error prone than trying to manage remote /application state implicitly.

Would it be useful if nix-flatpak provided helper functions to work with flatpakrepo / flatpakref files?

@gmodena gmodena pinned this issue Oct 6, 2024
@Bear-03
Copy link

Bear-03 commented Oct 6, 2024

With this, you can declare a remote and package as:

services.flatpak.remotes = lib.mkOptionDefault [{
    name = "sober"; # from SuggestRemoteName
    location = "https://sober.vinegarhq.org/repo/"; # from Url
  }];

services.flatpak.packages = [
     { appId = "org.vinegarhq.Sober"; origin = "sober"; }
];

I did try this before and unfortunately, in my case it causes the following error:

oct 06 23:27:01 bear-laptop systemd[1]: Starting flatpak-managed-install.service...
oct 06 23:27:02 bear-laptop flatpak[46214]: system: Added remote sober to https://sober.vinegarhq.org/repo/
oct 06 23:27:02 bear-laptop 7yzcg37k8xn5lvryngwdrpjzyl1m4a12-flatpak-managed-install[46214]: error: Signature made dom 15 sep 2024 18:55:44 using RSA key ID DAF8BEF8138D4B65
oct 06 23:27:02 bear-laptop 7yzcg37k8xn5lvryngwdrpjzyl1m4a12-flatpak-managed-install[46214]: Can't check signature: public key not found
oct 06 23:27:02 bear-laptop systemd[1]: flatpak-managed-install.service: Main process exited, code=exited, status=1/FAILURE

As I understand it, it's missing the GPG key tht is included in the .flatpakref. That's why my next approach was to just keep the file and patch the header.

I don't know how nix-flatpak works internally, but wouldn't it be possible to know that a certain repo does not have to be uninstalled because it is (implicitly) needed for a flatpakref package? If the package list is traversed and some packages are flatpakrefs, the names of their repos should be extracted from the files and even if uninstallUnmanaged = true, nix-flatpak would be able to use those names to ignore and keep the repos, just like with those in the remotes list.

@SpiderUnderUrBed
Copy link
Author

Also to answer your proposal, nix-flatpak should definitely add helper functions, I tried Bear-03's configuration and it yielded no errors, so I assume its done its job (I didn't re-install sober but I will try later). However, the current method is quite unintuitive and probably with caveats.

@gmodena
Copy link
Owner

gmodena commented Oct 7, 2024

Hey @Bear-03

As I understand it, it's missing the GPG key tht is included in the .flatpakref. That's why my next approach was to just keep the file and patch the header.

Good call. I think the pub key got cached in my experiments. I tried on a freshly built vm, and can repro that error.

I don't know how nix-flatpak works internally, but wouldn't it be possible to know that a certain repo does not have to be uninstalled because it is (implicitly) needed for a flatpakref package? I

Yep, it can be done pretty much the way you describe it. I have a PoC that needs a little flashing out (mostly internal APIs) and some more testing.

I do like your approach though, because it makes remote declaration explicit.

I was about to suggest you could pass the GPG key by injecting custom args = "..." in the remote declaration, but I am afraid that option is broken right now (it's something I've kept around for dev/tests).

@Bear-03
Copy link

Bear-03 commented Oct 7, 2024

I was about to suggest you could pass the GPG key by injecting custom args = "..." in the remote declaration, but I am afraid that option is broken right now (it's something I've kept around for dev/tests).

Even then, in the long run I think it would be much cleaner to allow passing .flatpakrefs to nix-flatpak directly (either via the packages option or a completely new one) instead of having to manually extract the data from within them, which is harder to maintain and more error prone.

@SpiderUnderUrBed
Copy link
Author

Also the ability to specify .flatpakrepo's could come in handy (in the case of local flatpak development or there might be a case where you don't have a repo at all). Specifying a .flatpakrepo should also be a feature at some point.

@Bear-03
Copy link

Bear-03 commented Oct 7, 2024

Isn't that already possible though? you can provide a local path with file://<absolute path to .flatpakrepo> like I did in my config. Although now that I think of it, the protocol could be implicit if a path is provided instead of a url...

@gmodena gmodena linked a pull request Oct 8, 2024 that will close this issue
2 tasks
@gmodena
Copy link
Owner

gmodena commented Oct 8, 2024

Isn't that already possible though? you can provide a local path with file://<absolute path to .flatpakrepo> like I did in my config. Although now that I think of it, the protocol could be implicit if a path is provided instead of a url...

Specifying a protocol is, in my opinion, good practice, but in case of remotes configs it is not strictly required. The location attribute is passed to flatpak remote-add, and Flatpak will resolve a local path even if the protocol is not specified.

@gmodena gmodena changed the title Abbility to specify a location to a flatpak ref for a application? Ability to specify a location to a flatpak ref for a application? Oct 9, 2024
@gmodena
Copy link
Owner

gmodena commented Oct 10, 2024

@Bear-03 @SpiderUnderUrBed I still need to add some unit tests before resolving, but if you want to give it a go #80 should be in a decent enough shape.

You can test the feature by tracking the install-flatpakref branch in your config:

...
nix-flatpak.url = "github:gmodena/nix-flatpak/install-flatpakref";
...

@SpiderUnderUrBed
Copy link
Author

I am getting an issue with the whole install/reinstall cycle as you said:

Oct 11 00:48:44 daspidercave systemd[1]: Starting flatpak-managed-install.service...
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: The following refs are installed from remote 'sober':
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]:    1) app/org.vinegarhq.Sober/x86_64/master
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: Remove them? [y/n]: n
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: error: Can't remove remote 'sober' with installed refs
Oct 11 00:48:48 daspidercave systemd[1]: flatpak-managed-install.service: Main process exited, code=exited, status=1/FAILURE
Oct 11 00:48:48 daspidercave systemd[1]: flatpak-managed-install.service: Failed with result 'exit-code'.
Oct 11 00:48:48 daspidercave systemd[1]: Failed to start flatpak-managed-install.service.
Oct 11 00:48:48 daspidercave systemd[1]: flatpak-managed-install.service: Consumed 373ms CPU time, 16.9M memory peak, 44.7K incoming IP traffic, 9.6K outgoing IP traffic.

Here is my nixos config files (under flatpak.nix, also forgive me if it looks a little weird, I haven't cleaned up the comments and had a rebase mishap):
https://github.com/SpiderUnderUrBed/nixos-files

@gmodena
Copy link
Owner

gmodena commented Oct 10, 2024

@SpiderUnderUrBed thanks for giving it a go!

Could you paste me the content of $HOME/.local/state/home-manager/gcroots/flatpak-state.json (if you load nix-flatpak as an HM module) or /nix/var/nix/gcroots/flatpak-state.json (if you load nix-flatpak as a nixos module)?

@SpiderUnderUrBed
Copy link
Author

I load nix-flatpak as a nixos module:
[spiderunderurbed@daspidercave:~]$ cat /nix/var/nix/gcroots/flatpak-state.json
{"overrides":{},"packages":["moe.launcher.the-honkers-railway-launcher","moe.launcher.an-anime-game-launcher","org.vinegarhq.Sober","io.gitlab.idevecore.Pomodoro","org.inkscape.Inkscape","org.signal.Signal","org.kde.knights","com.opera.Opera","org.nickvision.tubeconverter","io.kapsa.drive","com.rawtherapee.RawTherapee","org.eclipse.Java","org.rncbc.qpwgraph","com.google.Chrome","net.lutris.Lutris","com.valvesoftware.Steam","org.gnome.Notes","io.github.seadve.Mousai","org.torproject.torbrowser-launcher","org.kde.kdenlive","im.fluffychat.Fluffychat","org.kde.krita","com.system76.Popsicle","xyz.xclicker.xclicker","com.github.tchx84.Flatseal","com.obsproject.Studio","org.gimp.GIMP","com.visualstudio.code","org.qbittorrent.qBittorrent","org.nickvision.tubeconverter","de.haeckerfelix.Shortwave"],"remotes":["flathub","gol","flathub-beta","flathub","sober"]}

@gmodena
Copy link
Owner

gmodena commented Oct 13, 2024

@SpiderUnderUrBed thanks for sharing. flatpak-state.json look ok to me. Unfortunately I have not been able to repro your error:

Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: The following refs are installed from remote 'sober':
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]:    1) app/org.vinegarhq.Sober/x86_64/master
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: Remove them? [y/n]: n
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: error: Can't remove remote 'sober' with installed refs

the remote should have been removed only after the package was uninstall. Could you look in your system logs for failures wrt uninstalling org.vinegarhq.Sober ?

Could you try to uninstall the package and remote manually with:

flatpak uninstall org.vinegarhq.Sober
flatpak remote-delete sober

@SpiderUnderUrBed
Copy link
Author

SpiderUnderUrBed commented Oct 14, 2024

This error should be replicated if you install a remote, like "sober" which is a remote automatically made when you install from a flatpak ref. Then you install something from that remote, which is sober itself (from the flatpakref). I am confident this happens because uninstallUnmanaged is true, I don't know if you want this behavior to exist though, should it just skip uninstalling remotes with an existing ref (flatpakref installed from it)? or should it try to uninstall the ref first then the remote?

@gmodena
Copy link
Owner

gmodena commented Oct 14, 2024

This error should be replicated if you install a remote, like "sober" which is a remote automatically made when you install from a flatpak ref.

I did try to repro with uninstallUnmanaged=true, and a remote pointing to soberPatchedPath (same string replacement logic you linked).

First activation looks like this:

services.flatpak.uninstallUnmanaged = true;
services.flatpak.remotes = [
           {
               name = "flathub";
               location = "https://dl.flathub.org/repo/flathub.flatpakrepo";
           }
           {
               name = "sober";
               location = "file://${soberPatchedPath}";
           }
 ];
services.flatpak.packages = [
   { appId = "org.vinegarhq.Sober"; origin = "sober"; }
];

Remote and application have been installed.

Second activation:

services.flatpak.uninstallUnmanaged = true;
services.flatpak.remotes = [
            {
                name = "flathub";
                location = "https://dl.flathub.org/repo/flathub.flatpakrepo";
            }
  ];

services.flatpak.packages = [];

Remote and application are not installed in this generation (they have both been uninstalled on activation).

should it just skip uninstalling remotes with an existing ref (flatpakref installed from it)? or should it try to uninstall the ref first then the remote?

The expected behavior is that the flatpak-managed-install script will first attempt to uninstall the application and then remove the remote. If uninstalling the application fails, I would expect the script to abort.

I don't know why your system ended up in a state where the remote is uninstallable because of a dangling ref.

Could it be that at some point you installed the remote/app both declaratively and manually with flatpak install ...? Do you see multiple references to sober origins in flatpak remotes -v ? Similarly, do you see multiple Sober instances in flatpak list --app -v?

Maybe a failsafe is needed for this scenario:

Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: The following refs are installed from remote 'sober':
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]:    1) app/org.vinegarhq.Sober/x86_64/master
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: Remove them? [y/n]: n

Defaulting to y when flatpak ask to remove the refs should clean up state.

@SpiderUnderUrBed
Copy link
Author

SpiderUnderUrBed commented Oct 15, 2024


[spiderunderurbed@daspidercave:~]$ flatpak remotes -v
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/spiderunderurbed/.local/share/flatpak
flathub system
flathub-beta    system
gol     system
sober   system
flathub user
sober   user

https://pastebin.com/HBDFHbYv
The actual issue you eluded too is present:

Remote ‘sober’ found in multiple installations:

   1) system
   2) user

and I already sent a link to my repo, I think I found the source of the issue, its like you said, multiple remotes. I don't know if you tested dangling refs/my issue but I think you should be able to get the same error if you reproduce it that way. Then you can address this bug with your suggested failsafe.

@gmodena
Copy link
Owner

gmodena commented Oct 15, 2024

Thanks for confirming.

Then you can address this bug with your suggested failsafe.

I'll open a dedicate PR for this.

@Bear-03
Copy link

Bear-03 commented Nov 2, 2024

Hey @gmodena, I recently migrated to the new way of specifying .flatpakrefs and I am getting the following error on every rebuild.

warning: the following units failed: flatpak-managed-install.service
× flatpak-managed-install.service
     Loaded: loaded (/etc/systemd/system/flatpak-managed-install.service; enabled; preset: ignored)
     Active: failed (Result: exit-code) since Sat 2024-11-02 18:59:42 CET; 266ms ago
 Invocation: 174620ef2f1d406a85a6770998b17e25
    Process: 186875 ExecStart=/nix/store/vxfi6c4dkc9vz67lhvk95v8f6cic2k42-flatpak-managed-install (code=exited, status=1/FAILURE)
   Main PID: 186875 (code=exited, status=1/FAILURE)
         IP: 37.6K in, 4.2K out
   Mem peak: 17.8M
        CPU: 213ms

nov 02 18:59:42 bear-laptop systemd[1]: Starting flatpak-managed-install.service...
nov 02 18:59:42 bear-laptop flatpak[186927]: system: Removed remote sober
nov 02 18:59:42 bear-laptop vxfi6c4dkc9vz67lhvk95v8f6cic2k42-flatpak-managed-install[186943]: error: No remote refs found for ‘org.vinegarhq.Sober’
nov 02 18:59:42 bear-laptop systemd[1]: flatpak-managed-install.service: Main process exited, code=exited, status=1/FAILURE
nov 02 18:59:42 bear-laptop systemd[1]: flatpak-managed-install.service: Failed with result 'exit-code'.
nov 02 18:59:42 bear-laptop systemd[1]: Failed to start flatpak-managed-install.service.
nov 02 18:59:42 bear-laptop systemd[1]: flatpak-managed-install.service: Consumed 213ms CPU time, 17.8M memory peak, 37.6K incoming IP traffic, 4.2K outgoing IP traffic.
warning: error(s) occurred while switching to the new configuration

The issue is fixed if i run flatpak uninstall --all, but it fails again the next time I try to rebuild.

@gmodena
Copy link
Owner

gmodena commented Nov 3, 2024

Hey @Bear-03,

This sounds related to #90.

The issue is fixed if i run flatpak uninstall --all, but it fails again the next time I try to rebuild.

Ack. This is weird. I'm investigating and will follow up in #90.

@Bear-03
Copy link

Bear-03 commented Nov 3, 2024

Okay perfect, I'll keep an eye on #90 then in case any help is needed.

@gmodena
Copy link
Owner

gmodena commented Nov 11, 2024

Okay perfect, I'll keep an eye on #90 then in case any help is needed.

hey @Bear-03 , if you have some time to spare, I could use help with testing #93.
It's a few fixes that should help with the issue you reported (and with correctly setting remote names, when .flatpakref does not declare a SuggestRemoteName).

Would you mind tying an activation tracking the fix-flatpakref-install branch?

@Bear-03
Copy link

Bear-03 commented Nov 11, 2024

Hey @gmodena! I just tested it and it still fails.

# flake.nix
# ...
nix-flatpak.url = "github:gmodena/nix-flatpak?ref=fix-flatpakref-install";
# ...
restarting the following units: home-manager-bear.service
the following new units were started: run-credentials-systemd\x2dtmpfiles\x2dresetup.service.mount, sysinit-reactivation.target, systemd-localed.service, systemd-tmpfiles-resetup.service
warning: the following units failed: flatpak-managed-install.service
× flatpak-managed-install.service
     Loaded: loaded (/etc/systemd/system/flatpak-managed-install.service; enabled; preset: ignored)
     Active: failed (Result: exit-code) since Mon 2024-11-11 19:52:00 CET; 274ms ago
 Invocation: c2c956570a8a42b38c56687c4b6ad2f5
    Process: 5603 ExecStart=/nix/store/i5q4grhah3gbfjdhnmmnhdwxbiha8iz8-flatpak-managed-install (code=exited, status=1/FAILURE)
   Main PID: 5603 (code=exited, status=1/FAILURE)
         IP: 38.2K in, 4.4K out
         IO: 4K read, 0B written
   Mem peak: 7.9M
        CPU: 428ms

nov 11 19:51:59 bear-laptop systemd[1]: Starting flatpak-managed-install.service...
nov 11 19:52:00 bear-laptop i5q4grhah3gbfjdhnmmnhdwxbiha8iz8-flatpak-managed-install[5666]: error: No remote refs found for ‘org.vinegarhq.Sober’
nov 11 19:52:00 bear-laptop systemd[1]: flatpak-managed-install.service: Main process exited, code=exited, status=1/FAILURE
nov 11 19:52:00 bear-laptop systemd[1]: flatpak-managed-install.service: Failed with result 'exit-code'.
nov 11 19:52:00 bear-laptop systemd[1]: Failed to start flatpak-managed-install.service.
nov 11 19:52:00 bear-laptop systemd[1]: flatpak-managed-install.service: Consumed 428ms CPU time, 7.9M memory peak, 4K read from disk, 38.2K incoming IP traffic, 4.4K outgoing IP traffic.
warning: error(s) occurred while switching to the new configuration

@gmodena
Copy link
Owner

gmodena commented Nov 12, 2024

@Bear-03 many thanks for testing. I'm a bit at a loss on this one.

I've been daily driving that branch for a few days now (with Sober), and did not stumble upon the No remote refs found for ‘org.vinegarhq.Sober’ error.

Did you try manually uninstalling sober before the activation? Could you maybe check in flake.lock that the right nix-flatpak has been pinned?

This comments documents how I'm testing that branch in a VM:
#93 (comment)

Could you maybe help me pin-point what I might be doing different than you, either in terms of config or activation steps?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants