Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing Charts #31039

Closed
Rumbles opened this issue Dec 13, 2024 · 7 comments
Closed

Missing Charts #31039

Rumbles opened this issue Dec 13, 2024 · 7 comments
Assignees
Labels
keycloak solved tech-issues The user has a technical issue about an application triage Triage is needed

Comments

@Rumbles
Copy link

Rumbles commented Dec 13, 2024

Name and Version

bitnami/keycloak 24.3.0

What architecture are you using?

None

What steps will reproduce the bug?

Yesterday I set up a new keycloak server using 24.3.0, and I this morning I deleted everything to start again, but I suddenly get:

Error: invalid_reference: invalid tag

I tried manually to confirm and I get the same error locally, and the last 3 chart versions are also not available when I check, only 24.2.0 was available.

Is this to be expected? I thought tags should not be removed like this? Was there some rollback and some versions lost?

Sorry if this is the wrong place to report, but I was a bit confused

Are you using any custom parameters or values?

No response

What is the expected behavior?

Charts should be available

What do you see instead?

$ helm pull --repo https://charts.bitnami.com/bitnami keycloak --version=24.3.0
Error: invalid_reference: invalid tag
$ helm search repo bitnami/keycloak --versions
bitnami/keycloak        24.3.0          26.0.7          Keycloak is a high performance Java-based ident...
bitnami/keycloak        24.2.3          26.0.7          Keycloak is a high performance Java-based ident...
bitnami/keycloak        24.2.2          26.0.6          Keycloak is a high performance Java-based ident...
bitnami/keycloak        24.2.1          26.0.5          Keycloak is a high performance Java-based ident...
bitnami/keycloak        24.2.0          26.0.5          Keycloak is a high performance Java-based ident...
bitnami/keycloak        24.1.0          26.0.5          Keycloak is a high performance Java-based ident...
bitnami/keycloak        24.0.5          26.0.5          Keycloak is a high performance Java-based ident...
bitnami/keycloak        24.0.4          26.0.5          Keycloak is a high performance Java-based ident...
bitnami/keycloak        24.0.3          26.0.4          Keycloak is a high performance Java-based ident...
bitnami/keycloak        24.0.2          26.0.2          Keycloak is a high performance Java-based ident...

however 24.2.0 is the first version I can pull, 24.2.1, 24.2.2, 24.2.3 and 24.3.0 are all invalid tag

Additional information

No response

@Rumbles Rumbles added the tech-issues The user has a technical issue about an application label Dec 13, 2024
@github-actions github-actions bot added the triage Triage is needed label Dec 13, 2024
@SeanKilleen
Copy link
Contributor

SeanKilleen commented Dec 14, 2024

Just wanted to chime in that I'm seeing the same behavior. I found this issue after receiving the same issue attempting to check renovatebot PRs for both the minio and redis-cluster charts.

I've noticed the tags do appear to be available in docker.io when looking at the tags for docker images in a given chart. I only use the chart default versions so this is surprising we'd see this error.

I'm wondering if this might have something to do with needing to authenticate with a Docker account (purely a guess in this case). I'm going to look into how to achieve that.

@SeanKilleen
Copy link
Contributor

SeanKilleen commented Dec 14, 2024

I took the following steps (spoiler -- still not successful)

  • Created a docker access token with my account
  • Added a TF resource for a secret for it within k8s:
resource "kubernetes_secret" "docker_io_access_token" {
  metadata {
    name      = "docker-io-access-token"
    # NOTE: this is my namespace, yours will of course differ
    namespace = kubernetes_namespace.products.metadata[0].name 
  }

  type = "kubernetes.io/dockerconfigjson"

  data = {
    ".dockerconfigjson" = jsonencode({
      auths = {
        "ghcr.io" = {
          "username" = "MY_USERNAME"
          "password" = "MY_ACCESS_TOKEN" # Docker Personal Access token scoped to read packages only
          "email"    = "MY_EMAIL"
          "auth"     = base64encode("MY_USERNAME:MY_ACCESS_TOKEN")
        }
      }
    })
  }
}
  • Added that pull secret to the chart by setting the list of values for image.pullSecrets (I think there's also global.imagePullSecrets for most Bitnami charts).
  set_list {
    name  = "image.pullSecrets"
    value = [kubernetes_secret.docker_io_access_token.metadata[0].name]
  }
  • Applied the chart as-is with no update to the latest version -- pulled image and worked correctly
  • Updated the version and applied the chart. Received the same invalid_reference: invalid tag error.

So (I think?) at this point I've ruled out it having anything to do with docker creds.

@SeanKilleen
Copy link
Contributor

Found this, which seems relevant: helm/helm#13466

The Bitnami charts are listed here I think: https://repo.broadcom.com/bitnami-files/index.yaml

The issue seems to be that the OCI chart definition includes the version -- here's redis-cluster, for example:

image

@SeanKilleen
Copy link
Contributor

SeanKilleen commented Dec 14, 2024

From what I can tell, this part of #30402 might be the cause (and subsequent modifications). Looks like the index has the OCI reference including a tag, and then also includes the version, which helm then parses into a duplicate tag because it uses URL + version:

https://github.com/bitnami/charts/pull/30402/files#diff-c5f77f0bd6fa2d85047cc8657ea0b16c6c65cbaa6f168ee7e9c38b9a8966d2d1R87

I'll see if I can maybe submit a pull request, assuming I'm not somehow way off.

@SeanKilleen
Copy link
Contributor

SeanKilleen commented Dec 14, 2024

Nevermind -- I see here that the issue is that Bitnami appears to be defaulting to OCI formats: #30519 (comment)

In the https format of the URL, the Bitnami index includes the tag for the version, which then has the version additionally appended to it by Helm, causing in invalid tag error.

I updated my repos in helm to use the OCI format (e.g. from https://charts.bitnami.com/bitnami to oci://registry-1.docker.io/bitnamicharts) and this appears to resolve the symptom.

Sorry for all the comments while I flailed about. 😅

@Rumbles
Copy link
Author

Rumbles commented Dec 14, 2024

Hi @SeanKilleen thanks for the info, I was not aware of this change.

The strange part for me was that it worked one day, then stopped working the next

@carrodher
Copy link
Member

As you rightly said and as reported in helm/helm#13466, this problem stems from how the helm ... --repo ... command handles an index.yaml file containing OCI URLs.

Why is this issue appearing now?

Historically, Bitnami Helm charts were accessed by adding the Bitnami repository to the Helm CLI:

$ helm repo add bitnami https://charts.bitnami.com/bitnami

or by directly installing/pulling a specific Helm chart using the --repo flag:

$ helm pull --repo https://charts.bitnami.com/bitnami foo

In April 2023, Bitnami introduced support for the OCI format, recommending OCI-based installation as the primary method going forward. For example:

helm pull oci://registry-1.docker.io/bitnamicharts/external-dns --version 8.6.0

As detailed in our recent announcement and #30110, the traditional method is now being deprecated. Consequently, the index.yaml served at https://charts.bitnami.com/bitnami now uses OCI URLs, exposing the above mentioned issue in the Helm CLI's handling of these URLs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
keycloak solved tech-issues The user has a technical issue about an application triage Triage is needed
Projects
None yet
Development

No branches or pull requests

3 participants