Pushing to registry

[azkiii]

Hi, whenever I try to push to the registry from my local machine I receive the following error:

The push refers to repository [localhost:5000/nginx]
Get "http://localhost:5000/v2/": dial tcp [::1]:5000: connect: connection refused

However, curl seems to work fine:

$ curl http://localhost:5000/v2/_catalog
{"repositories":[]}

Here's a describe.

$ kubectl describe pod docker-registry-6cf76d94bf-kq798 -n kind-docker-registry
Name:         docker-registry-6cf76d94bf-kq798
Namespace:    kind-docker-registry
Priority:     0
Node:         kind-development-worker/172.19.0.2
Start Time:   Fri, 11 Feb 2022 21:36:34 +0100
Labels:       app=docker-registry
              pod-template-hash=6cf76d94bf
              release=docker-registry
Annotations:  checksum/config: bec82c70a7bf4930eb1a6ec665bcfa11d585217e0b3efdcb0da3b513f73d4ff4
              checksum/secret: 783c8c8f8c223b48d2dcaa50990312d3bd819671bdfdae90170363a5b1ece3fd
Status:       Running
IP:           10.244.1.5
IPs:
  IP:           10.244.1.5
Controlled By:  ReplicaSet/docker-registry-6cf76d94bf
Containers:
  docker-registry:
    Container ID:  containerd://bb645bfb4fce0d28f5c1a3747dfa05ccace283915db080ebf1fea7c08f65abd5
    Image:         registry:2.7.1
    Image ID:      docker.io/library/registry@sha256:169211e20e2f2d5d115674681eb79d21a217b296b43374b8e39f97fcf866b375
    Port:          5000/TCP
    Host Port:     0/TCP
    Command:
      /bin/registry
      serve
      /etc/docker/registry/config.yml
    State:          Running
      Started:      Fri, 11 Feb 2022 21:36:40 +0100
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:5000/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:5000/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      REGISTRY_HTTP_SECRET:                       <set to the key 'haSharedSecret' in secret 'docker-registry-secret'>  Optional: false
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY:  /var/lib/registry
    Mounts:
      /etc/docker/registry from docker-registry-config (rw)
      /var/lib/registry/ from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4m4tf (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  docker-registry-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      docker-registry-config
    Optional:  false
  kube-api-access-4m4tf:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  6m41s  default-scheduler  Successfully assigned kind-docker-registry/docker-registry-6cf76d94bf-kq798 to kind-development-worker
  Normal  Pulling    6m41s  kubelet            Pulling image "registry:2.7.1"
  Normal  Pulled     6m35s  kubelet            Successfully pulled image "registry:2.7.1" in 6.3038477s
  Normal  Created    6m35s  kubelet            Created container docker-registry
  Normal  Started    6m35s  kubelet            Started container docker-registry

Does anyone know why this is happening?

[canterberry]

Based on the error message, it looks like Docker is resolving localhost to an IPv6 address, whereas I imagine curl is resolving to IPv4. To confirm, try curl with the -v flag to see which IP localhost resolves to. Then, use that IP instead of localhost when attempting to push the image. I'd venture to guess that IP's going to be 127.0.0.1.

[azkiii]

Unfortunately that doesn't seem to fix the issue :(

$ docker push 127.0.0.1:5000/nginx:latest
Get "http://127.0.0.1:5000/v2/":
dial tcp 127.0.0.1:5000: connect: connection refused

Also, the registry also seems to work with curl on IPv6.

curl [::1]:5000/v2/_catalog
{"repositories":[]}

[canterberry]

In that case, my next thought would be that your Docker daemon is operating on a different network, or a different host on the same network. 127.0.0.1 and localhost would resolve differently on a per-host basis. You just need to find an IP that's reachable to the Docker daemon. You can do that through trial and error by trying the registry's Service IP, one of its Pod IPs, or one of the cluster's Node IPs (making sure to use the appropriate port). To experiment, you can docker run -it --rm alpine sh to launch a shell on the same network as the Docker daemon. From there, you can apk add curl then try out curl with different urls until you find one that works. That'll give you the IP/port/hostname to use to successfully push/pull images from the registry. Note the IP you find that works may not be reachable if you try to curl that from your host. That's okay, as long as docker can reach it.

[azkiii]

That was indeed the case. Thanks a lot!

[krancour]

I've run into this issue as well within the last few days. Didn't manage to solve it, but discovered I get the same results if I skip the chart and just run the registry in the local Docker daemon. I think the issue is with the registry image and/or Docker client configuration and not the chart.

[krancour]

Inexplicably, my problem went away. I think I've rebooted since I last encountered the issue, so I'm assuming I jolted something in my Docker client config when I rebooted.

@azkiii try this and see what happens. It'll take Helm and k8s out of the equation, so it may narrow down where the issue is.

$ docker run -d -p 5000:5000 registry:2.8.0 
$ docker pull alpine:latest
$ docker tag alpine:latest localhost:5000/alpine:latest
$ docker push localhost:5000/alpine:latest

[azkiii]

Running a registry in Docker seems to work fine.

docker push localhost:5000/alpine:latest
The push refers to repository [localhost:5000/alpine]
8d3ac3489996: Pushed
latest: digest: sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3 size: 528

[krancour]

Alrighty. Then the issue's in the chart or in k8s. Sorry for initially steering you in the wrong direction!