Error while deploying openness release 20.12 for edge node. #77
Comments
|
Hi @pushpraj527! This might be happening due to slow download speed. Can you give it another try and see if the issue still persists? |
|
Hi @amr-mokhtar I tried two times and always getting this issue. Pod status and secret status are mentioned below. NAMESPACE NAME READY STATUS RESTARTS AGE secrets.txt |
|
Hi @pushpraj527 |
|
Hi @tomaszwesolowski I tried it on the fresh machine only. Nothing manually done there. |
|
Hi, could you share with us file Openness_experience_kit_archive.tar.gz that was created after deployment? |
|
Hello @tomaszwesolowski I have a the same problem as the issue starter when deploying the edge node (currently in the same step at 34 retries left and counting). Please, find my Openness_experience_kit_archive.tar.gz here: 2021_01_13_11_44_59_Openness_experience_kit_archive.tar.gz In case it's relevant, here's the "describe pod" output of the only pod I found in the kafka namespace: https://pastebin.com/raw/gPZsTYp4 My knowledge about Kubernetes is limited but according to the output below my edge node seems to be ready and has no taints: In the edge node, the docker logs of the only container matching "docker container ls | grep kafka" are empty. I would appreciate any pointers towards finding a solution for this issue. Thanks in advance. Edit: It seems to be an underlaying networking issue from my part unrelated to taints and tolerations where nodes just can't reach pods from a different node. I'll edit again confirming it if I can fix it. |
|
My solution is run “kubectl taint nodes - -all node-role.kubernetes.io/master-“ and then run deploy.py file again |
I am facing exact same issue...Can you please elaborate on what you did?
Details would be very much appreciated. Thanks, |
|
@NishankK In my case the error was just caused because the pods in my controller node had no connectivity with pods in the edge node. I would check whether that's your case too. (I don't remember exactly right now, but in my case I had the controller in a OpenStack VM and it was some issue with port security rules I think). If I'm understanding correctly, removing the master taint from all nodes as suggested above would enable the EAA pod (or whatever it is) to be deployed on the controller instead of the edge, which would place it in the same node as the Kafka pod but I don't think it's correct (but I could be missing something, sorry i nadvance if that's the case). |
Thanks for replying Jorge..Actually I am getting this exact error while deploying Openness on Azure..I got confused. In Azure deployment case, with the click of a button the Edge as well as controller node is supposed to be deployed on Azure. Okay I have 1 other query, if you can help, please. Is it possible to deploy Openness controller and Edge on VM rather than server? If yes, what is the recommended VM configuration like how may GBs, CPU etc..Can't find a clear answer to this anywhere. Many thanks |
|
I'm no expert by any means, so I'll just link you to this thread in OpenNESS-dev (Question number 2 covers that): https://mail.openness.org/archives/developer/2021-January/000225.html |
Hi,
We tried deploying openness-experience-kit(v20.12). We were able to deploy controller successfully, but while deploying node faced the below mention issue. I am also attaching deployment log here(
2020-12-21_14-31-23_ansible.log
). Please let me if any thing else is required.
2020-12-21 14:53:50,037 p=17432 u=root n=ansible | TASK [openness/node : wait for Kafka CA and User secrets] **********************$restartCounts; do\n if [ $ ((restartCount + 0)) -gt 10 ]; then\n exit -1\n fi\ndone\nkubectl get secret cluster-cluster-ca-cert -n kafka && kubectl get secret eaa-kafka -n kafka\n",
2020-12-21 14:53:50,037 p=17432 u=root n=ansible | task path: /home/centos/aman/openness-experience-kits/roles/openness/node/tasks/prebuild/kafka_certs.yml:23
2020-12-21 14:53:50,892 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (60 retries left).
2020-12-21 14:54:52,117 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (59 retries left).
2020-12-21 14:55:53,300 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (58 retries left).
2020-12-21 14:56:54,469 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (57 retries left).
2020-12-21 14:57:58,617 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (56 retries left).
2020-12-21 14:59:00,383 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (55 retries left).
2020-12-21 15:00:01,794 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (54 retries left).
2020-12-21 15:01:03,138 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (53 retries left).
2020-12-21 15:02:04,407 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (52 retries left).
2020-12-21 15:03:05,760 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (51 retries left).
2020-12-21 15:04:07,640 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (50 retries left).
2020-12-21 15:05:09,128 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (49 retries left).
2020-12-21 15:06:10,436 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (48 retries left).
2020-12-21 15:07:11,940 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (47 retries left).
2020-12-21 15:08:15,510 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (46 retries left).
2020-12-21 15:09:16,786 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (45 retries left).
2020-12-21 15:10:18,205 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (44 retries left).
2020-12-21 15:11:19,646 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (43 retries left).
2020-12-21 15:12:20,947 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (42 retries left).
2020-12-21 15:13:22,296 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (41 retries left).
2020-12-21 15:14:23,920 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (40 retries left).
2020-12-21 15:15:25,327 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (39 retries left).
2020-12-21 15:16:26,751 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (38 retries left).
2020-12-21 15:17:28,155 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (37 retries left).
2020-12-21 15:18:32,401 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (36 retries left).
2020-12-21 15:19:33,872 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (35 retries left).
2020-12-21 15:20:36,335 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (34 retries left).
2020-12-21 15:21:37,702 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (33 retries left).
2020-12-21 15:22:39,036 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (32 retries left).
2020-12-21 15:23:40,375 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (31 retries left).
2020-12-21 15:24:41,715 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (30 retries left).
2020-12-21 15:25:43,159 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (29 retries left).
2020-12-21 15:26:45,648 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (28 retries left).
2020-12-21 15:27:47,039 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (27 retries left).
2020-12-21 15:28:50,458 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (26 retries left).
2020-12-21 15:29:52,203 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (25 retries left).
2020-12-21 15:30:53,609 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (24 retries left).
2020-12-21 15:31:54,969 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (23 retries left).
2020-12-21 15:32:56,339 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (22 retries left).
2020-12-21 15:33:57,995 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (21 retries left).
2020-12-21 15:34:59,529 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (20 retries left).
2020-12-21 15:36:00,889 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (19 retries left).
2020-12-21 15:37:02,278 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (18 retries left).
2020-12-21 15:38:03,674 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (17 retries left).
2020-12-21 15:39:10,494 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (16 retries left).
2020-12-21 15:40:11,852 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (15 retries left).
2020-12-21 15:41:14,020 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (14 retries left).
2020-12-21 15:42:15,470 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (13 retries left).
2020-12-21 15:43:16,845 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (12 retries left).
2020-12-21 15:44:18,169 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (11 retries left).
2020-12-21 15:45:19,598 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (10 retries left).
2020-12-21 15:46:20,973 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (9 retries left).
2020-12-21 15:47:22,383 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (8 retries left).
2020-12-21 15:48:23,708 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (7 retries left).
2020-12-21 15:49:28,033 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (6 retries left).
2020-12-21 15:50:29,519 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (5 retries left).
2020-12-21 15:51:31,098 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (4 retries left).
2020-12-21 15:52:32,647 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (3 retries left).
2020-12-21 15:53:34,163 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (2 retries left).
2020-12-21 15:54:35,627 p=17432 u=root n=ansible | FAILED - RETRYING: wait for Kafka CA and User secrets (1 retries left).
2020-12-21 15:55:37,504 p=17432 u=root n=ansible | fatal: [node01 -> 192.168.0.16]: FAILED! => {
"attempts": 60,
"changed": false,
"cmd": "restartCounts=
kubectl get pods -n kafka -o json | jq -r '.items[] | [.status.containerStatuses[].restartCount] | @sh'\nfor restartCount in"delta": "0:00:00.713457",
"end": "2020-12-21 15:55:37.251058",
"rc": 1,
"start": "2020-12-21 15:55:36.537601"
}
STDOUT:
NAME TYPE DATA AGE
cluster-cluster-ca-cert Opaque 3 56m
STDERR:
jq: error (at :1756): Cannot iterate over null (null)
Error from server (NotFound): secrets "eaa-kafka" not found
MSG:
non-zero return code
The text was updated successfully, but these errors were encountered: