Increasing timeout of the wait for apiserver

[lzap]

Hello,

my hardware which is Dell Z400 workstation with 12GB RAM and Xeon with 4 cores does meet the minimum requirements, I have a bare-metal installation of RHEL8 fully updated yet CRC start does error out with:

DEBU retry loop: attempt 52
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
DEBU SSH command results: err: Process exited with status 124, output:
DEBU
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s
DEBU RetryAfter timeout after 53 tries
DEBU Making call to close driver server
DEBU (crc) Calling .Close
DEBU Successfully made call to close driver server
DEBU Making call to close connection to plugin binary
Error waiting for apiserver: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124

Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 (x17)
Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 (x35)

Full log: https://gist.github.com/lzap/d07920acbd77f80cb8a72516fa03470b
Debug diagnostics: https://gist.github.com/lzap/0e9abc22958c597c11fc381299b55a57

The machine does not have a SSD that might be the issue (regular HDD), I can clearly see that minutes after the loop timeouts I can see the VM still performing some CPU tasks.

Is there a way to increase the timeout or amount of attempts for the call? Thanks.

Minimum requirements I refer to: https://access.redhat.com/documentation/en-us/red_hat_codeready_containers/1.40/html-single/getting_started_guide/index#minimum-system-requirements_gsg

[cfergeau]

Could you add the full log? a bit hard to parse with just this.
If you ssh into the VM while this is running and after this failed, and run oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig manually, does this succeed?

[lzap]

Further log according to the debugging wiki: https://gist.github.com/lzap/0e9abc22958c597c11fc381299b55a57

[cfergeau]

Did you grab these logs before/or just after crc start timed out? Or did you let your cluster plenty of time to settle? because all looks good here.

[lzap]

Yeah its weird, everything looks fine, I think the issue is that the hardware have quite slow HDD and it was booting for such a long time.

I started it up, waited and then I grabbed logs after the error.

[lzap]

So its solved: my hardware does not simply have enough memory, it has 10 GB (I thought it had 12 GB). Thanks for help, closing.

I tried this:

[crc@zzzap ~]$ crc config set skip-check-ram true
Successfully configured skip-check-ram to true
[crc@zzzap ~]$ crc start -m 7500 -c 3
requires memory in MiB >= 9216

Did not work out. Also noticed there is some Podman profile/preset:

[crc@zzzap ~]$ crc config set preset podman
Configuration property 'preset' does not exist

I guess this is only in the nightly build?

[cfergeau]

The 2.0.1 release has this podman preset.

[lzap]

Correction: the hardware has 10 GB of memory not 12 GB, it still meets the minimum requirements in the docs, but I found out that it is swapping. This is top after 8 hours after crc start with no actions taken, load is 1.5-2.0 and memory is 90% full.

top - 10:51:43 up 20:26,  1 user,  load average: 1.47, 1.74, 1.85
Tasks: 171 total,   1 running, 170 sleeping,   0 stopped,   0 zombie
%Cpu(s): 28.6 us,  9.1 sy,  0.0 ni, 58.2 id,  2.6 wa,  1.4 hi,  0.2 si,  0.0 st
MiB Mem :   9765.3 total,    136.5 free,   7434.6 used,   2194.3 buff/cache
MiB Swap:   5044.0 total,   4192.4 free,    851.6 used.   2076.4 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   8327 qemu      20   0   11.9g   8.3g   6300 S 153.5  87.3   1802:08 qemu-kvm
   8027 root      20   0  813400  42944  12260 S   0.0   0.4   0:24.09 packagekitd

By the way, libvirt domain defines 9.5 GB RAM for the VM while minimum requirements are 9 GB, that does not sound right. I feel like it shoud have been bumped - the OS needs some memory too. So I guess this hardware is not sufficient, well, it's an old workstation of mine from like 2012.

[lzap]

I tried:

[crc@zzzap ~]$ crc start -m 7500 -c 3
requires memory in MiB >= 9216

So really it looks like I am done on this hardware I guess :-)

[gbraad]

libvirt domain defines 9.5 GB RAM for the VM while minimum requirements are 9 GB

libvirt defines in GiB and the defaults requests are in MB or GB, right?

[cfergeau]

The VM has <currentMemory unit='KiB'>9437184</currentMemory> which is exactly 9 GiB, but is more than 9,5 GB. The amount of RAM in a computer is usually expressed in GiB, so 9 is the more accurate figure imo.

[gbraad]

So no changes there either

[gbraad]

Correction: the hardware has 10 GB of memory not 12 GB,

You need 9GB of free memory on the machine. The swapping in that case might be from the host and not the VM.