Unable to deploy to Digital Ocean - 401 Unauthorized

[codepilotsf]

For some reason, after authenticating with doctl auth init --context myapp I'm hitting this error:

 > exporting to image:
------
ERROR: failed to solve: failed to push registry.digitalocean.com/myapp-registry/myapp:17f228b9bd594c42e92dfcaa593d770d210090b3: failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://api.digitalocean.com/v2/registry/auth?scope=repository%3Acodepilotsf%2Ftestme%3Apull&scope=repository%3Amyapp-registry%2Fmyapp%3Apull%2Cpush&service=registry.digitalocean.com: 401 Unauthorized

The really weird thing is that the URL in the error seems to show old connection info "codepilotsf/testme" in addition to the new one. This is all such a black box to me. I thought Kamal was supposed to simplify deployment. :-)

Any clues?

[codepilotsf]

I wanted to answer my own question here (sorta) in case it's helpful to someone else. What I've learned is that DO container registry is a pain in the ass. I think almost all of my initial frustration with Kamal was due to issues with doctl and various related auth issues. I switched over to DockerHub and everything worked great straight away. Then I learned that my self-hosted GitLab also has a built in container registry service which is just as simple, works perfectly, and costs nothing. Tldr; Love DO, but their container registry service kinda blows.

[rschultheis]

I am having this same apparent issue with DigitalOcean registry.

When I attempt a kamal deploy or more simply a kamal build push it fails with:

#22 ERROR: failed to push registry.digitalocean.com/xxxx/yyyy:c7420fb0197c08e7d29d3ebb15b295af533f1fb0: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://api.digitalocean.com/v2/registry/auth?scope=repository%3Axxxx%2Fyyyy%3Apull%2Cpush&service=registry.digitalocean.com: 401 Unauthorized

When I do kamal registry login it works no problem.

This was all working so nicely the other day!

What I know so far: It does not matter if I use kamal 2.3.0 or 2.4.0, I still get the same result.

@codepilotsf did you find some kind of cache file or other thing relating to DO API/doctl that helps here? I would prefer not to switch registries.

[nickhammond]

@codepilotsf Kamal doesn't use doctl, it's just using docker push commands with the given server, username, and password. This thread has a good full example of using DO as your registry #804 (comment).

[rschultheis]

I've got a few apps on kamal, all setup basically the same with DigitalOcean, just different accounts. I have them setup like the example you link to @nickhammond and its been working fine for months. When I moved to a new laptop and set up all the .kamal/secrets only one is now not working. the .kamal/secrets is identical to the one on my my old laptop where it does work! (shasum .kamal/secrets gives idential sha)

This is the error

#22 pushing layers
#22 pushing layers 1.0s done
#22 ERROR: failed to push registry.digitalocean.com/xxxx/yyyy:c7420fb0197c08e7d29d3ebb15b295af533f1fb0: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://api.digitalocean.com/v2/registry/auth?scope=repository%xxxx%yyyy%3Apull%2Cpush&service=registry.digitalocean.com: 401 Unauthorized
------
 > exporting to image:
------
ERROR: failed to solve: failed to push registry.digitalocean.com/xxxx/yyyy:c7420fb0197c08e7d29d3ebb15b295af533f1fb0: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://api.digitalocean.com/v2/registry/auth?scope=repository%xxxx%2Fyyyy%3Apull%2Cpush&service=registry.digitalocean.com: 401 Unauthorized

I am able to use the exact same credentials that are not working in kamal to do a manual docker login registry.digitalocean.com and a docker push. I am using the account email as the username.

I think there is something weird about the digitalocean registry here. I remember when I was initially setting this app up I had a similar issue and just fiddled and retried and it eventually worked. I wasn't sure what acually ended up fixing it.

I'll update back here if I get a solution.

[rschultheis]

OK, I have made progress but I still do not understand this issue.

I am sometimes getting an error when pushing registry.digitalocean.com that looks like this:

#22 ERROR: failed to push registry.digitalocean.com/xxxx/yyyy:c7420fb0197c08e7d29d3ebb15b295af533f1fb0: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://api.digitalocean.com/v2/registry/auth?scope=repository%xxxx%yyyy%3Apull%2Cpush&service=registry.digitalocean.com: 401 Unauthorized

I can successfully kamal registry login so my digitalocean registry config is good. But when I do a kamal deploy or kamal build push I get that blasted 401 error above.

Major clue: I am using OSX and Docker Desktop (by chance were you too @codepilotsf ?).

When I change kamal config so it uses remote linux builder, everything works. I change nothing else, just add the remote: line to the builder config and it just works (like it used to on my OSX machine).

I believe there is some funny business around the "credStore" config feature in docker for OSX. Unfortunately, it seems to be very very difficult if not impossible to disable that feature now in the newest versions of docker desktop, so I went way down the rabbit hole of trying to figure how to disable it and ultimately it was never confident that it was actually disabled. I am not a super docker expert nor do I really want to be one.

Anyways.... my workaround is to just use a remote linux builder and everything works perfectly. I am moving on... I really tried to get to the bottom of this. If there are any Docker OSX experts out there, this could use a look (maybe @nickhammond ?). I don't think this is a kamal issue at this point but my confidence in that is low.

I love Kamal, but man this docker issue was a major PITA!

[codepilotsf]

@rschultheis Yes. MacOS with Docker Desktop over here. But here's an important point: When I switched away from DO's registry and tried using DockerHub, then GitLab's built-in registry, then later using Gitea's built-in registry, all three of those non-DO options worked flawlessly even with MacOS Docker Desktop.

[rschultheis]

I think the difference with DigitalOcean is that every private repo uses the same url: registry.digitalocean.com where as GitLab gives you a distinct URL for each private repo?

I have multiple kamal projects hosted on DigitalOcean, and they use distinct DigitalOcean registries. Somewhere, something is caching login credentials for registry.digitalocean.com and not accounting for the fact that we have multiple accounts for multiple different repos.

My best guess is the credStore part of Docker Desktop, but I just can't pin down the exact place this cacheing is happening.