HI You can check what's the failure by this command firstly. $ Kubectl describe po rook-ceph-mon-a-65b58876db-98jg5 -n kube-system BR! Martin, Chen IOTG, Software Engineer 021-61164330 -----Original Message----- From: starlingx-discuss-request@lists.starlingx.io <starlingx-discuss-request@lists.starlingx.io> Sent: Thursday, July 1, 2021 8:30 PM To: starlingx-discuss@lists.starlingx.io Subject: Starlingx-discuss Digest, Vol 38, Issue 2 Send Starlingx-discuss mailing list submissions to starlingx-discuss@lists.starlingx.io To subscribe or unsubscribe via the World Wide Web, visit http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss or, via email, send a message with subject or body 'help' to starlingx-discuss-request@lists.starlingx.io You can reach the person managing the list at starlingx-discuss-owner@lists.starlingx.io When replying, please edit your Subject line so it is more specific than "Re: Contents of Starlingx-discuss digest..." Today's Topics: 1. rook-ceph deployment failure (Embedded Devel) ---------------------------------------------------------------------- Message: 1 Date: Thu, 01 Jul 2021 02:39:10 +0000 From: Embedded Devel <lists@optimcloud.com> To: Starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] rook-ceph deployment failure Message-ID: <1625106963669.939601506.1886162123@optimcloud.com> Content-Type: text/plain; charset=utf-8; format=flowed stx 5.0 simplex bare metal fais so 1) how to resolv, or 2) how to back out and just use normal ceph kube-system rook-ceph-crashcollector-controller-0-96bb8b6df-cz5qf 1/1 Running 1 45h kube-system rook-ceph-mgr-a-7b84cc65d6-bt5mq 1/1 Running 55 45h kube-system rook-ceph-mon-a-65b58876db-98jg5 0/1 CrashLoopBackOff 520 45h kube-system rook-ceph-operator-79fb8559-k7w8z 1/1 Running 1 45h kube-system rook-ceph-tools-5778d7f6c-d5pwp 1/1 Running 1 45h kube-system rook-discover-djv7r 1/1 Running 1 45h controller-0:~$ cat /var/log/armada/rook-ceph-apps-apply_2021-06-30-12-41-10.log 2021-06-30 12:41:11.316 68 DEBUG armada.handlers.document [-] Resolving reference /tmp/manifests/rook-ceph-apps/1.0-5/rook-ceph-apps-manifest.yaml. resolve_reference /usr/local/lib/python3.6/dist-packages/armada/handlers/document.py:49 2021-06-30 12:41:11.327 68 DEBUG armada.handlers.tiller [-] Using Tiller host IP: 127.0.0.1 _get_tiller_ip /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:165 2021-06-30 12:41:11.328 68 DEBUG armada.handlers.tiller [-] Using Tiller host port: 24134 _get_tiller_port /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:174 2021-06-30 12:41:11.328 68 DEBUG armada.handlers.tiller [-] Tiller getting gRPC insecure channel at 127.0.0.1:24134 with options: [grpc.max_send_message_length=429496729, grpc.max_receive_message_length=429496729] get_channel /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:127 2021-06-30 12:41:11.334 68 DEBUG armada.handlers.tiller [-] Armada is using Tiller at: 127.0.0.1:24134, namespace=kube-system, timeout=300 __init__ /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:107 2021-06-30 12:41:11.334 68 INFO armada.handlers.lock [-] Acquiring lock 2021-06-30 12:41:11.350 68 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] kube-system-rook-operator validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108 2021-06-30 12:41:11.351 68 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] kube-system-rook-ceph validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108 2021-06-30 12:41:11.351 68 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] kube-system-rook-ceph-provisioner validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108 2021-06-30 12:41:11.351 68 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] starlingx-rook-charts validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108 2021-06-30 12:41:11.351 68 DEBUG armada.utils.validate [-] Validating document [armada/Manifest/v1] rook-ceph-manifest validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108 2021-06-30 12:41:11.352 68 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] kube-system-rook-operator validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108 2021-06-30 12:41:11.352 68 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] kube-system-rook-ceph validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108 2021-06-30 12:41:11.352 68 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] kube-system-rook-ceph-provisioner validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108 2021-06-30 12:41:11.353 68 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] starlingx-rook-charts validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108 2021-06-30 12:41:11.353 68 DEBUG armada.utils.validate [-] Validating document [armada/Manifest/v1] rook-ceph-manifest validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108 2021-06-30 12:41:11.353 68 INFO armada.handlers.armada [-] Performing pre-flight operations. 2021-06-30 12:41:11.353 68 DEBUG armada.handlers.tiller [-] Using Tiller host IP: 127.0.0.1 _get_tiller_ip /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:165 2021-06-30 12:41:11.353 68 DEBUG armada.handlers.tiller [-] Getting Tiller Status: Tiller exists tiller_status /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:186 2021-06-30 12:41:11.353 68 INFO armada.handlers.armada [-] Downloading tarball from: http://192.168.206.1:8080/helm_charts/stx-platform/rook-operator-0.1.0.tgz 2021-06-30 12:41:11.354 68 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts 2021-06-30 12:41:11.359 68 INFO armada.handlers.armada [-] Downloading tarball from: http://192.168.206.1:8080/helm_charts/stx-platform/rook-ceph-0.1.0.tgz 2021-06-30 12:41:11.359 68 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts 2021-06-30 12:41:11.362 68 INFO armada.handlers.armada [-] Downloading tarball from: http://192.168.206.1:8080/helm_charts/stx-platform/rook-ceph-provisioner-0.1... 2021-06-30 12:41:11.362 68 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts 2021-06-30 12:41:11.365 68 DEBUG armada.handlers.tiller [-] Tiller ListReleases() with timeout=300, request=limit: 32 status_codes: UNKNOWN status_codes: DEPLOYED status_codes: DELETED status_codes: DELETING status_codes: FAILED status_codes: PENDING_INSTALL status_codes: PENDING_UPGRADE status_codes: PENDING_ROLLBACK get_results /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:215 2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release stx-rook-ceph, version 1, status: DEPLOYED list_releases /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276 2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release stx-rook-operator, version 1, status: DEPLOYED list_releases /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276 2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release osh-openstack-mariadb, version 1, status: PENDING_INSTALL list_releases /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276 2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release osh-openstack-ingress, version 1, status: DEPLOYED list_releases /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276 2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release osh-openstack-nginx-ports-control, version 1, status: DEPLOYED list_releases /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276 2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release osh-openstack-psp-rolebinding, version 1, status: DEPLOYED list_releases /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276 2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release stx-cephfs-provisioner, version 1, status: DEPLOYED list_releases /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276 2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release stx-ceph-pools-audit, version 1, status: DEPLOYED list_releases /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276 2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release stx-rbd-provisioner, version 1, status: DEPLOYED list_releases /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276 2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release cm-cert-manager-psp-rolebinding, version 1, status: DEPLOYED list_releases /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276 2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release cm-cert-manager, version 1, status: DEPLOYED list_releases /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276 2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release ic-nginx-ingress, version 1, status: DEPLOYED list_releases /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276 2021-06-30 12:41:11.403 68 INFO armada.handlers.armada [-] Processing ChartGroup: starlingx-rook-charts (StarlingX Rook Ceph Charts), sequenced=True 2021-06-30 12:41:11.403 68 INFO armada.handlers.chart_deploy [-] [chart=kube-system-rook-operator]: Processing Chart, release=stx-rook-operator 2021-06-30 12:41:11.403 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-operator]: Resolved `wait.resources` list: [{'type': 'pod', 'labels': {'app': 'rook-ceph-operator'}}] __init__ /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:89 2021-06-30 12:41:11.404 68 INFO armada.handlers.chart_deploy [-] [chart=kube-system-rook-operator]: Existing release stx-rook-operator found in namespace kube-system 2021-06-30 12:41:11.406 68 INFO armada.handlers.chart_deploy [-] [chart=kube-system-rook-operator]: Checking for updates to chart release inputs. 2021-06-30 12:41:11.425 68 INFO armada.handlers.chart_deploy [-] [chart=kube-system-rook-operator]: Found no updates to chart release inputs 2021-06-30 12:41:11.425 68 INFO armada.handlers.wait [-] [chart=kube-system-rook-operator]: Waiting for resource type=pod, namespace=kube-system labels=app=rook-ceph-operator required=True for 1800s 2021-06-30 12:41:11.425 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-operator]: Starting to wait on: namespace=kube-system, resource type=pod, label_selector=(app=rook-ceph-operator), timeout=1800 _watch_resource_completions /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:367 2021-06-30 12:41:11.435 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-operator]: pod rook-ceph-operator-79fb8559-k7w8z is ready! handle_resource /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:258 2021-06-30 12:41:11.436 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-operator]: Found no modified resources. wait /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:302 2021-06-30 12:41:11.436 68 INFO armada.handlers.chart_deploy [-] [chart=kube-system-rook-ceph]: Processing Chart, release=stx-rook-ceph 2021-06-30 12:41:11.436 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Resolved `wait.resources` list: [{'type': 'pod', 'labels': {'app': 'rook-ceph-mgr'}}, {'type': 'pod', 'labels': {'app': 'rook-ceph-mon'}}, {'type': 'pod', 'labels': {'app': 'rook-ceph-osd'}}] __init__ /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:89 2021-06-30 12:41:11.437 68 INFO armada.handlers.chart_deploy [-] [chart=kube-system-rook-ceph]: Existing release stx-rook-ceph found in namespace kube-system 2021-06-30 12:41:11.439 68 INFO armada.handlers.chart_deploy [-] [chart=kube-system-rook-ceph]: Checking for updates to chart release inputs. 2021-06-30 12:41:11.448 68 INFO armada.handlers.chart_deploy [-] [chart=kube-system-rook-ceph]: Found no updates to chart release inputs 2021-06-30 12:41:11.448 68 INFO armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Waiting for resource type=pod, namespace=kube-system labels=app=rook-ceph-mgr required=True for 1800s 2021-06-30 12:41:11.448 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Starting to wait on: namespace=kube-system, resource type=pod, label_selector=(app=rook-ceph-mgr), timeout=1800 _watch_resource_completions /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:367 2021-06-30 12:41:11.476 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: pod rook-ceph-mgr-a-7b84cc65d6-bt5mq is ready! handle_resource /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:258 2021-06-30 12:41:11.476 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Found no modified resources. wait /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:302 2021-06-30 12:41:11.476 68 INFO armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Waiting for resource type=pod, namespace=kube-system labels=app=rook-ceph-mon required=True for 1800s 2021-06-30 12:41:11.476 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Starting to wait on: namespace=kube-system, resource type=pod, label_selector=(app=rook-ceph-mon), timeout=1800 _watch_resource_completions /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:367 2021-06-30 12:41:11.483 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: pod rook-ceph-mon-a-65b58876db-98jg5 not ready: Waiting for pod rook-ceph-mon-a-65b58876db-98jg5 to be ready... handle_resource /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:260 2021-06-30 12:41:22.131 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Watch event: type=MODIFIED, name=rook-ceph-mon-a-65b58876db-98jg5, namespace=kube-system,resource_version=2005320 _watch_resource_completions /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:412 2021-06-30 12:41:22.131 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: pod rook-ceph-mon-a-65b58876db-98jg5 not ready: Waiting for pod rook-ceph-mon-a-65b58876db-98jg5 to be ready... handle_resource /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:260 2021-06-30 12:41:35.357 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Watch event: type=MODIFIED, name=rook-ceph-mon-a-65b58876db-98jg5, namespace=kube-system,resource_version=2005430 _watch_resource_completions /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:412 2021-06-30 12:41:35.358 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: pod rook-ceph-mon-a-65b58876db-98jg5 is ready! handle_resource /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:258 2021-06-30 12:41:35.358 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Found modified resources: ['rook-ceph-mon-a-65b58876db-98jg5'] wait /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:299 2021-06-30 12:41:35.358 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Continuing to wait: 0 consecutive attempts without modified resources of 1 required. wait /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:310 2021-06-30 12:41:36.359 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Starting to wait on: namespace=kube-system, resource type=pod, label_selector=(app=rook-ceph-mon), timeout=1775 _watch_resource_completions /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:367 2021-06-30 12:41:36.367 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: pod rook-ceph-mon-a-65b58876db-98jg5 is ready! handle_resource /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:258 2021-06-30 12:41:36.367 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Found no modified resources. wait /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:302 2021-06-30 12:41:36.367 68 INFO armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Waiting for resource type=pod, namespace=kube-system labels=app=rook-ceph-osd required=True for 1775s 2021-06-30 12:41:36.367 68 DEBUG armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Starting to wait on: namespace=kube-system, resource type=pod, label_selector=(app=rook-ceph-osd), timeout=1775 _watch_resource_completions /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:367 2021-06-30 12:42:11.404 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:43:11.466 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:44:11.528 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:45:11.589 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:46:11.651 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:47:11.712 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:48:11.774 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:49:11.835 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:50:11.897 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:51:11.958 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:52:12.019 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:53:12.081 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:54:12.144 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:55:12.205 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:56:12.268 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:57:12.329 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:58:12.392 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 12:59:12.454 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 13:00:12.517 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 13:01:12.578 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 13:02:12.639 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 13:03:12.701 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 13:04:12.762 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 13:05:12.824 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 13:06:12.886 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 13:07:12.946 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 13:08:13.007 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 13:09:13.069 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 13:10:13.132 68 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176 2021-06-30 13:11:11.371 68 ERROR armada.handlers.wait [-] [chart=kube-system-rook-ceph]: Timed out waiting for pods (namespace=kube-system, labels=(app=rook-ceph-osd)). None found! Are `wait.labels` correct? Does `wait.resources` need to exclude `type: pod`? 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada [-] Chart deploy [kube-system-rook-ceph] failed: armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for pods (namespace=kube-system, labels=(app=rook-ceph-osd)). None found! Are `wait.labels` correct? Does `wait.resources` need to exclude `type: pod`? 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada Traceback (most recent call last): 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 225, in handle_result 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada result = get_result() 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 236, in <lambda> 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada if (handle_result(chart, lambda: deploy_chart(chart))): 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 214, in deploy_chart 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada chart, cg_test_all_charts, prefix, known_releases) 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/chart_deploy.py", line 248, in execute 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada chart_wait.wait(timer) 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 134, in wait 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada wait.wait(timeout=timeout) 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 294, in wait 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada modified = self._wait(deadline) 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 354, in _wait 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada raise k8s_exceptions.KubernetesWatchTimeoutException(error) 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for pods (namespace=kube-system, labels=(app=rook-ceph-osd)). None found! Are `wait.labels` correct? Does `wait.resources` need to exclude `type: pod`? 2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada 2021-06-30 13:11:11.373 68 ERROR armada.handlers.armada [-] Chart deploy(s) failed: ['kube-system-rook-ceph'] 2021-06-30 13:11:12.193 68 INFO armada.handlers.lock [-] Releasing lock 2021-06-30 13:11:12.196 68 ERROR armada.cli [-] Caught internal exception: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['kube-system-rook-ceph'] 2021-06-30 13:11:12.196 68 ERROR armada.cli Traceback (most recent call last): 2021-06-30 13:11:12.196 68 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke 2021-06-30 13:11:12.196 68 ERROR armada.cli self.invoke() 2021-06-30 13:11:12.196 68 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke 2021-06-30 13:11:12.196 68 ERROR armada.cli resp = self.handle(documents, tiller) 2021-06-30 13:11:12.196 68 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper 2021-06-30 13:11:12.196 68 ERROR armada.cli return future.result() 2021-06-30 13:11:12.196 68 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result 2021-06-30 13:11:12.196 68 ERROR armada.cli return self.__get_result() 2021-06-30 13:11:12.196 68 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-06-30 13:11:12.196 68 ERROR armada.cli raise self._exception 2021-06-30 13:11:12.196 68 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run 2021-06-30 13:11:12.196 68 ERROR armada.cli result = self.fn(*self.args, **self.kwargs) 2021-06-30 13:11:12.196 68 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle 2021-06-30 13:11:12.196 68 ERROR armada.cli return armada.sync() 2021-06-30 13:11:12.196 68 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 252, in sync 2021-06-30 13:11:12.196 68 ERROR armada.cli raise armada_exceptions.ChartDeployException(failures) 2021-06-30 13:11:12.196 68 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['kube-system-rook-ceph'] 2021-06-30 13:11:12.196 68 ERROR armada.cli command terminated with exit code 1 -- Sent with Vivaldi Mail. Download Vivaldi for free at vivaldi.com ------------------------------ Subject: Digest Footer _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss ------------------------------ End of Starlingx-discuss Digest, Vol 38, Issue 2 ************************************************