GitLab CI fails whenever repo is not public #267

kegalo · 2024-10-28T20:55:40Z

Describe the bug

After setting up a GitLab runner, whether managed through YunoHost or not, any and all build jobs on a non-public repo fail as soon the runner attempts to pull your repo.

Context

Hardware: Hetzner dedicated root server
YunoHost version: 12.0.4.1 (but this issue existed before I switched to beta)
I have access to my server: Through SSH, through the webadmin, through IPMI
Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: Yes
- Currently on the bookworm beta, but this issue existed for me before I upgraded.
Using, or trying to install package version/branch: main

Steps to reproduce

Install GitLab
Register runner
Create repo, set it to internal or private
Add .gitlab-ci.yml

Expected behavior

The runner should pull your repo using an authentication token. The runner does attempt to use the token, but it seems like something is stopping GitLab from seeing this token - I can't find any log containing the ci-runner:token@gitlab I expected. I do find two 401s in the rails production logs matching up to the build job, though.

Logs

GitLab CI log:

remote: HTTP Basic: Access denied. If a password was provided for Git authentication, the password was incorrect or you're required to use a token instead of a password. If a token was provided, it was either incorrect, expired, or improperly scoped. See https://docs.gitlab.com/17.5/ee/topics/git/troubleshooting_git.html#error-on-git-fetch-http-basic-access-denied

/var/log/gitlab/gitlab-rails/production_json.log:

{
    "method": "GET",
    "path": "/user/repo.git/info/refs",
    "format": "*/*",
    "controller": "Repositories::GitHttpController",
    "action": "info_refs",
    "status": 401,
    "time": "2024-10-28T20:24:34.046Z",
    "params": [
        {
            "key": "service",
            "value": "git-upload-pack"
        },
        {
            "key": "repository_path",
            "value": "user/repo.git"
        }
    ],
    "correlation_id": "01JBACTFYKJWQCNDGWRZCDRR8W",
    "meta.caller_id": "Repositories::GitHttpController#info_refs",
    "meta.feature_category": "source_code_management",
    "repository_storage": "default",
    "remote_ip": "XXX",
    "ua": "gitlab-runner 17.5.2 linux/amd64",
    "request_urgency": "default",
    "target_duration_s": 1,
    "redis_calls": 2,
    "redis_duration_s": 0.000639,
    "redis_read_bytes": 398,
    "redis_write_bytes": 131,
    "redis_feature_flag_calls": 2,
    "redis_feature_flag_duration_s": 0.000639,
    "redis_feature_flag_read_bytes": 398,
    "redis_feature_flag_write_bytes": 131,
    "db_count": 3,
    "db_write_count": 0,
    "db_cached_count": 0,
    "db_txn_count": 0,
    "db_replica_txn_count": 0,
    "db_primary_txn_count": 0,
    "db_main_txn_count": 0,
    "db_ci_txn_count": 0,
    "db_main_replica_txn_count": 0,
    "db_ci_replica_txn_count": 0,
    "db_replica_count": 0,
    "db_primary_count": 3,
    "db_main_count": 3,
    "db_ci_count": 0,
    "db_main_replica_count": 0,
    "db_ci_replica_count": 0,
    "db_replica_write_count": 0,
    "db_primary_write_count": 0,
    "db_main_write_count": 0,
    "db_ci_write_count": 0,
    "db_main_replica_write_count": 0,
    "db_ci_replica_write_count": 0,
    "db_replica_cached_count": 0,
    "db_primary_cached_count": 0,
    "db_main_cached_count": 0,
    "db_ci_cached_count": 0,
    "db_main_replica_cached_count": 0,
    "db_ci_replica_cached_count": 0,
    "db_replica_wal_count": 0,
    "db_primary_wal_count": 0,
    "db_main_wal_count": 0,
    "db_ci_wal_count": 0,
    "db_main_replica_wal_count": 0,
    "db_ci_replica_wal_count": 0,
    "db_replica_wal_cached_count": 0,
    "db_primary_wal_cached_count": 0,
    "db_main_wal_cached_count": 0,
    "db_ci_wal_cached_count": 0,
    "db_main_replica_wal_cached_count": 0,
    "db_ci_replica_wal_cached_count": 0,
    "db_replica_txn_max_duration_s": 0,
    "db_primary_txn_max_duration_s": 0,
    "db_main_txn_max_duration_s": 0,
    "db_ci_txn_max_duration_s": 0,
    "db_main_replica_txn_max_duration_s": 0,
    "db_ci_replica_txn_max_duration_s": 0,
    "db_replica_txn_duration_s": 0,
    "db_primary_txn_duration_s": 0,
    "db_main_txn_duration_s": 0,
    "db_ci_txn_duration_s": 0,
    "db_main_replica_txn_duration_s": 0,
    "db_ci_replica_txn_duration_s": 0,
    "db_replica_duration_s": 0,
    "db_primary_duration_s": 0.003,
    "db_main_duration_s": 0.003,
    "db_ci_duration_s": 0,
    "db_main_replica_duration_s": 0,
    "db_ci_replica_duration_s": 0,
    "cpu_s": 0.039546,
    "mem_objects": 10424,
    "mem_bytes": 1166640,
    "mem_mallocs": 4039,
    "mem_total_bytes": 1583600,
    "pid": 2119943,
    "worker_id": "puma_5",
    "rate_limiting_gates": [],
    "db_duration_s": 0.00317,
    "view_duration_s": 0.00657,
    "duration_s": 0.02882
}
{
    "method": "GET",
    "path": "/user/repo.git/info/refs",
    "format": "*/*",
    "controller": "Repositories::GitHttpController",
    "action": "info_refs",
    "status": 401,
    "time": "2024-10-28T20:24:34.249Z",
    "params": [
        {
            "key": "service",
            "value": "git-upload-pack"
        },
        {
            "key": "repository_path",
            "value": "user/repo.git"
        }
    ],
    "correlation_id": "01JBACTG5E4DTDJFWZ0E50GMA8",
    "meta.caller_id": "Repositories::GitHttpController#info_refs",
    "meta.feature_category": "source_code_management",
    "repository_storage": "default",
    "remote_ip": "XXX",
    "ua": "gitlab-runner 17.5.2 linux/amd64",
    "request_urgency": "default",
    "target_duration_s": 1,
    "db_count": 3,
    "db_write_count": 0,
    "db_cached_count": 0,
    "db_txn_count": 0,
    "db_replica_txn_count": 0,
    "db_primary_txn_count": 0,
    "db_main_txn_count": 0,
    "db_ci_txn_count": 0,
    "db_main_replica_txn_count": 0,
    "db_ci_replica_txn_count": 0,
    "db_replica_count": 0,
    "db_primary_count": 3,
    "db_main_count": 3,
    "db_ci_count": 0,
    "db_main_replica_count": 0,
    "db_ci_replica_count": 0,
    "db_replica_write_count": 0,
    "db_primary_write_count": 0,
    "db_main_write_count": 0,
    "db_ci_write_count": 0,
    "db_main_replica_write_count": 0,
    "db_ci_replica_write_count": 0,
    "db_replica_cached_count": 0,
    "db_primary_cached_count": 0,
    "db_main_cached_count": 0,
    "db_ci_cached_count": 0,
    "db_main_replica_cached_count": 0,
    "db_ci_replica_cached_count": 0,
    "db_replica_wal_count": 0,
    "db_primary_wal_count": 0,
    "db_main_wal_count": 0,
    "db_ci_wal_count": 0,
    "db_main_replica_wal_count": 0,
    "db_ci_replica_wal_count": 0,
    "db_replica_wal_cached_count": 0,
    "db_primary_wal_cached_count": 0,
    "db_main_wal_cached_count": 0,
    "db_ci_wal_cached_count": 0,
    "db_main_replica_wal_cached_count": 0,
    "db_ci_replica_wal_cached_count": 0,
    "db_replica_txn_max_duration_s": 0,
    "db_primary_txn_max_duration_s": 0,
    "db_main_txn_max_duration_s": 0,
    "db_ci_txn_max_duration_s": 0,
    "db_main_replica_txn_max_duration_s": 0,
    "db_ci_replica_txn_max_duration_s": 0,
    "db_replica_txn_duration_s": 0,
    "db_primary_txn_duration_s": 0,
    "db_main_txn_duration_s": 0,
    "db_ci_txn_duration_s": 0,
    "db_main_replica_txn_duration_s": 0,
    "db_ci_replica_txn_duration_s": 0,
    "db_replica_duration_s": 0,
    "db_primary_duration_s": 0.002,
    "db_main_duration_s": 0.002,
    "db_ci_duration_s": 0,
    "db_main_replica_duration_s": 0,
    "db_ci_replica_duration_s": 0,
    "cpu_s": 0.024558,
    "mem_objects": 9414,
    "mem_bytes": 1156880,
    "mem_mallocs": 3905,
    "mem_total_bytes": 1533440,
    "pid": 2119945,
    "worker_id": "puma_6",
    "rate_limiting_gates": [],
    "db_duration_s": 0.00197,
    "view_duration_s": 0.00496,
    "duration_s": 0.0167
}

The text was updated successfully, but these errors were encountered:

kay0u · 2024-10-29T03:53:48Z

Does it work on a public repository?

Can you provide the cli used to register the runner? How did you get your token?

It's weird because it works at home™.

kegalo · 2024-10-29T05:03:08Z

It does work on a public repo, and it does work when GitLab is not installed alongside YunoHost.

I used the interactive registration and am using the docker executor.

Right now, I'm using the docker image GitLab provides (gitlab/gitlab-runner), but the issue also existed for me when I used the GitLab runner provided by the app catalog.

I got the token by going through Admin > CI/CD > Runners > New instance Runner.

I reinstalled GitLab a number of times, and tried modifying various nginx configs just to see if anything would change, but nothing did change. I reset everything back, and it still doesn't work.

Let me know if there are any more logs I can provide - I am honestly at a loss at this point, none of the logs I went through gave me any idea of why it isn't working for me.

kay0u · 2024-10-29T06:46:53Z

Ok, I spoke too fast, I can reproduce your issue on my server. It'll be way easier for me to debug.

It's probably because of the Yunohost's sso.

…c auth spoofing to fix #267

kay0u · 2024-10-31T03:09:06Z

I manually fix this issue by hacking the ssowatt conf file and adding to gitlab.main permission :
"protect_against_basic_auth_spoofing": false

If you don't know what I'm talking about, just wait the next release ;)

kegalo · 2024-10-31T03:36:34Z

works perfectly for me, thank you!

kay0u · 2024-10-31T03:52:25Z

FYI, the conf will be overwritten on each app install/upgrade/remove

kegalo changed the title ~~GitLab CI fails (potentially due to not receiving runner token?) whenever repo is not public~~ GitLab CI fails whenever repo is not public Oct 28, 2024

kay0u added a commit that referenced this issue Oct 31, 2024

basic auth doesn't work on gitlab, and remove protection against basi…

07f611c

…c auth spoofing to fix #267

kay0u linked a pull request Nov 1, 2024 that will close this issue

Testing #268

Open

2 tasks

philippstappert mentioned this issue Nov 18, 2024

Git over HTTPS not working after upgrade to Yunohost 12 #269

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitLab CI fails whenever repo is not public #267

GitLab CI fails whenever repo is not public #267

kegalo commented Oct 28, 2024

kay0u commented Oct 29, 2024

kegalo commented Oct 29, 2024

kay0u commented Oct 29, 2024

kay0u commented Oct 31, 2024

kegalo commented Oct 31, 2024

kay0u commented Oct 31, 2024

GitLab CI fails whenever repo is not public #267

GitLab CI fails whenever repo is not public #267

Comments

kegalo commented Oct 28, 2024

Describe the bug

Context

Steps to reproduce

Expected behavior

Logs

kay0u commented Oct 29, 2024

kegalo commented Oct 29, 2024

kay0u commented Oct 29, 2024

kay0u commented Oct 31, 2024

kegalo commented Oct 31, 2024

kay0u commented Oct 31, 2024