[Starlingx-discuss] rook-ceph deployment failure

Embedded Devel lists at optimcloud.com
Thu Jul 1 02:39:10 UTC 2021


stx 5.0 simplex bare metal fais so 1) how to resolv, or 2) how to back out 
and just use normal ceph

kube-system    rook-ceph-crashcollector-controller-0-96bb8b6df-cz5qf   1/1  
   Running             1          45h
kube-system    rook-ceph-mgr-a-7b84cc65d6-bt5mq                        1/1  
   Running             55         45h
kube-system    rook-ceph-mon-a-65b58876db-98jg5                        0/1  
   CrashLoopBackOff    520        45h
kube-system    rook-ceph-operator-79fb8559-k7w8z                       1/1  
   Running             1          45h
kube-system    rook-ceph-tools-5778d7f6c-d5pwp                         1/1  
   Running             1          45h
kube-system    rook-discover-djv7r                                     1/1  
   Running             1          45h

controller-0:~$ cat 
/var/log/armada/rook-ceph-apps-apply_2021-06-30-12-41-10.log
2021-06-30 12:41:11.316 68 DEBUG armada.handlers.document [-] Resolving 
reference /tmp/manifests/rook-ceph-apps/1.0-5/rook-ceph-apps-manifest.yaml. 
resolve_reference 
/usr/local/lib/python3.6/dist-packages/armada/handlers/document.py:49
2021-06-30 12:41:11.327 68 DEBUG armada.handlers.tiller [-] Using Tiller 
host IP: 127.0.0.1 _get_tiller_ip 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:165
2021-06-30 12:41:11.328 68 DEBUG armada.handlers.tiller [-] Using Tiller 
host port: 24134 _get_tiller_port 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:174
2021-06-30 12:41:11.328 68 DEBUG armada.handlers.tiller [-] Tiller getting 
gRPC insecure channel at 127.0.0.1:24134 with options: 
[grpc.max_send_message_length=429496729, 
grpc.max_receive_message_length=429496729] get_channel 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:127
2021-06-30 12:41:11.334 68 DEBUG armada.handlers.tiller [-] Armada is using 
Tiller at: 127.0.0.1:24134, namespace=kube-system, timeout=300 __init__ 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:107
2021-06-30 12:41:11.334 68 INFO armada.handlers.lock [-] Acquiring lock
2021-06-30 12:41:11.350 68 DEBUG armada.utils.validate [-] Validating 
document [armada/Chart/v1] kube-system-rook-operator 
validate_armada_document 
/usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2021-06-30 12:41:11.351 68 DEBUG armada.utils.validate [-] Validating 
document [armada/Chart/v1] kube-system-rook-ceph validate_armada_document 
/usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2021-06-30 12:41:11.351 68 DEBUG armada.utils.validate [-] Validating 
document [armada/Chart/v1] kube-system-rook-ceph-provisioner 
validate_armada_document 
/usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2021-06-30 12:41:11.351 68 DEBUG armada.utils.validate [-] Validating 
document [armada/ChartGroup/v1] starlingx-rook-charts 
validate_armada_document 
/usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2021-06-30 12:41:11.351 68 DEBUG armada.utils.validate [-] Validating 
document [armada/Manifest/v1] rook-ceph-manifest validate_armada_document 
/usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2021-06-30 12:41:11.352 68 DEBUG armada.utils.validate [-] Validating 
document [armada/Chart/v1] kube-system-rook-operator 
validate_armada_document 
/usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2021-06-30 12:41:11.352 68 DEBUG armada.utils.validate [-] Validating 
document [armada/Chart/v1] kube-system-rook-ceph validate_armada_document 
/usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2021-06-30 12:41:11.352 68 DEBUG armada.utils.validate [-] Validating 
document [armada/Chart/v1] kube-system-rook-ceph-provisioner 
validate_armada_document 
/usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2021-06-30 12:41:11.353 68 DEBUG armada.utils.validate [-] Validating 
document [armada/ChartGroup/v1] starlingx-rook-charts 
validate_armada_document 
/usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2021-06-30 12:41:11.353 68 DEBUG armada.utils.validate [-] Validating 
document [armada/Manifest/v1] rook-ceph-manifest validate_armada_document 
/usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2021-06-30 12:41:11.353 68 INFO armada.handlers.armada [-] Performing 
pre-flight operations.
2021-06-30 12:41:11.353 68 DEBUG armada.handlers.tiller [-] Using Tiller 
host IP: 127.0.0.1 _get_tiller_ip 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:165
2021-06-30 12:41:11.353 68 DEBUG armada.handlers.tiller [-] Getting Tiller 
Status: Tiller exists tiller_status 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:186
2021-06-30 12:41:11.353 68 INFO armada.handlers.armada [-] Downloading 
tarball from: 
http://192.168.206.1:8080/helm_charts/stx-platform/rook-operator-0.1.0.tgz
2021-06-30 12:41:11.354 68 WARNING armada.handlers.armada [-] Disabling 
server validation certs to extract charts
2021-06-30 12:41:11.359 68 INFO armada.handlers.armada [-] Downloading 
tarball from: 
http://192.168.206.1:8080/helm_charts/stx-platform/rook-ceph-0.1.0.tgz
2021-06-30 12:41:11.359 68 WARNING armada.handlers.armada [-] Disabling 
server validation certs to extract charts
2021-06-30 12:41:11.362 68 INFO armada.handlers.armada [-] Downloading 
tarball from: 
http://192.168.206.1:8080/helm_charts/stx-platform/rook-ceph-provisioner-0.1.0.tgz

2021-06-30 12:41:11.362 68 WARNING armada.handlers.armada [-] Disabling 
server validation certs to extract charts
2021-06-30 12:41:11.365 68 DEBUG armada.handlers.tiller [-] Tiller 
ListReleases() with timeout=300, request=limit: 32
status_codes: UNKNOWN
status_codes: DEPLOYED
status_codes: DELETED
status_codes: DELETING
status_codes: FAILED
status_codes: PENDING_INSTALL
status_codes: PENDING_UPGRADE
status_codes: PENDING_ROLLBACK
  get_results 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:215
2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release 
stx-rook-ceph, version 1, status: DEPLOYED list_releases 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276
2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release 
stx-rook-operator, version 1, status: DEPLOYED list_releases 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276
2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release 
osh-openstack-mariadb, version 1, status: PENDING_INSTALL list_releases 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276
2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release 
osh-openstack-ingress, version 1, status: DEPLOYED list_releases 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276
2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release 
osh-openstack-nginx-ports-control, version 1, status: DEPLOYED 
list_releases 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276
2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release 
osh-openstack-psp-rolebinding, version 1, status: DEPLOYED list_releases 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276
2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release 
stx-cephfs-provisioner, version 1, status: DEPLOYED list_releases 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276
2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release 
stx-ceph-pools-audit, version 1, status: DEPLOYED list_releases 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276
2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release 
stx-rbd-provisioner, version 1, status: DEPLOYED list_releases 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276
2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release 
cm-cert-manager-psp-rolebinding, version 1, status: DEPLOYED list_releases 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276
2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release 
cm-cert-manager, version 1, status: DEPLOYED list_releases 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276
2021-06-30 12:41:11.402 68 DEBUG armada.handlers.tiller [-] Found release 
ic-nginx-ingress, version 1, status: DEPLOYED list_releases 
/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:276
2021-06-30 12:41:11.403 68 INFO armada.handlers.armada [-] Processing 
ChartGroup: starlingx-rook-charts (StarlingX Rook Ceph Charts), 
sequenced=True
2021-06-30 12:41:11.403 68 INFO armada.handlers.chart_deploy [-] 
[chart=kube-system-rook-operator]: Processing Chart, 
release=stx-rook-operator
2021-06-30 12:41:11.403 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-operator]: Resolved `wait.resources` list: 
[{'type': 'pod', 'labels': {'app': 'rook-ceph-operator'}}] __init__ 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:89
2021-06-30 12:41:11.404 68 INFO armada.handlers.chart_deploy [-] 
[chart=kube-system-rook-operator]: Existing release stx-rook-operator found 
in namespace kube-system
2021-06-30 12:41:11.406 68 INFO armada.handlers.chart_deploy [-] 
[chart=kube-system-rook-operator]: Checking for updates to chart release 
inputs.
2021-06-30 12:41:11.425 68 INFO armada.handlers.chart_deploy [-] 
[chart=kube-system-rook-operator]: Found no updates to chart release inputs
2021-06-30 12:41:11.425 68 INFO armada.handlers.wait [-] 
[chart=kube-system-rook-operator]: Waiting for resource type=pod, 
namespace=kube-system labels=app=rook-ceph-operator required=True for 1800s
2021-06-30 12:41:11.425 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-operator]: Starting to wait on: 
namespace=kube-system, resource type=pod, 
label_selector=(app=rook-ceph-operator), timeout=1800 
_watch_resource_completions 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:367
2021-06-30 12:41:11.435 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-operator]: pod rook-ceph-operator-79fb8559-k7w8z is 
ready! handle_resource 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:258
2021-06-30 12:41:11.436 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-operator]: Found no modified resources. wait 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:302
2021-06-30 12:41:11.436 68 INFO armada.handlers.chart_deploy [-] 
[chart=kube-system-rook-ceph]: Processing Chart, release=stx-rook-ceph
2021-06-30 12:41:11.436 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Resolved `wait.resources` list: [{'type': 
'pod', 'labels': {'app': 'rook-ceph-mgr'}}, {'type': 'pod', 'labels': 
{'app': 'rook-ceph-mon'}}, {'type': 'pod', 'labels': {'app': 
'rook-ceph-osd'}}] __init__ 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:89
2021-06-30 12:41:11.437 68 INFO armada.handlers.chart_deploy [-] 
[chart=kube-system-rook-ceph]: Existing release stx-rook-ceph found in 
namespace kube-system
2021-06-30 12:41:11.439 68 INFO armada.handlers.chart_deploy [-] 
[chart=kube-system-rook-ceph]: Checking for updates to chart release 
inputs.
2021-06-30 12:41:11.448 68 INFO armada.handlers.chart_deploy [-] 
[chart=kube-system-rook-ceph]: Found no updates to chart release inputs
2021-06-30 12:41:11.448 68 INFO armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Waiting for resource type=pod, 
namespace=kube-system labels=app=rook-ceph-mgr required=True for 1800s
2021-06-30 12:41:11.448 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Starting to wait on: namespace=kube-system, 
resource type=pod, label_selector=(app=rook-ceph-mgr), timeout=1800 
_watch_resource_completions 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:367
2021-06-30 12:41:11.476 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: pod rook-ceph-mgr-a-7b84cc65d6-bt5mq is 
ready! handle_resource 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:258
2021-06-30 12:41:11.476 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Found no modified resources. wait 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:302
2021-06-30 12:41:11.476 68 INFO armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Waiting for resource type=pod, 
namespace=kube-system labels=app=rook-ceph-mon required=True for 1800s
2021-06-30 12:41:11.476 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Starting to wait on: namespace=kube-system, 
resource type=pod, label_selector=(app=rook-ceph-mon), timeout=1800 
_watch_resource_completions 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:367
2021-06-30 12:41:11.483 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: pod rook-ceph-mon-a-65b58876db-98jg5 not 
ready: Waiting for pod rook-ceph-mon-a-65b58876db-98jg5 to be ready... 
handle_resource 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:260
2021-06-30 12:41:22.131 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Watch event: type=MODIFIED, 
name=rook-ceph-mon-a-65b58876db-98jg5, 
namespace=kube-system,resource_version=2005320 _watch_resource_completions 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:412
2021-06-30 12:41:22.131 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: pod rook-ceph-mon-a-65b58876db-98jg5 not 
ready: Waiting for pod rook-ceph-mon-a-65b58876db-98jg5 to be ready... 
handle_resource 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:260
2021-06-30 12:41:35.357 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Watch event: type=MODIFIED, 
name=rook-ceph-mon-a-65b58876db-98jg5, 
namespace=kube-system,resource_version=2005430 _watch_resource_completions 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:412
2021-06-30 12:41:35.358 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: pod rook-ceph-mon-a-65b58876db-98jg5 is 
ready! handle_resource 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:258
2021-06-30 12:41:35.358 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Found modified resources: 
['rook-ceph-mon-a-65b58876db-98jg5'] wait 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:299
2021-06-30 12:41:35.358 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Continuing to wait: 0 consecutive attempts 
without modified resources of 1 required. wait 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:310
2021-06-30 12:41:36.359 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Starting to wait on: namespace=kube-system, 
resource type=pod, label_selector=(app=rook-ceph-mon), timeout=1775 
_watch_resource_completions 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:367
2021-06-30 12:41:36.367 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: pod rook-ceph-mon-a-65b58876db-98jg5 is 
ready! handle_resource 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:258
2021-06-30 12:41:36.367 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Found no modified resources. wait 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:302
2021-06-30 12:41:36.367 68 INFO armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Waiting for resource type=pod, 
namespace=kube-system labels=app=rook-ceph-osd required=True for 1775s
2021-06-30 12:41:36.367 68 DEBUG armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Starting to wait on: namespace=kube-system, 
resource type=pod, label_selector=(app=rook-ceph-osd), timeout=1775 
_watch_resource_completions 
/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:367
2021-06-30 12:42:11.404 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:43:11.466 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:44:11.528 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:45:11.589 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:46:11.651 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:47:11.712 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:48:11.774 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:49:11.835 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:50:11.897 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:51:11.958 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:52:12.019 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:53:12.081 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:54:12.144 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:55:12.205 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:56:12.268 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:57:12.329 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:58:12.392 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 12:59:12.454 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 13:00:12.517 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 13:01:12.578 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 13:02:12.639 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 13:03:12.701 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 13:04:12.762 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 13:05:12.824 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 13:06:12.886 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 13:07:12.946 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 13:08:13.007 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 13:09:13.069 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 13:10:13.132 68 DEBUG armada.handlers.lock [-] Updating lock 
update_lock 
/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176
2021-06-30 13:11:11.371 68 ERROR armada.handlers.wait [-] 
[chart=kube-system-rook-ceph]: Timed out waiting for pods 
(namespace=kube-system, labels=(app=rook-ceph-osd)). None found! Are 
`wait.labels` correct? Does `wait.resources` need to exclude `type: pod`?
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada [-] Chart deploy 
[kube-system-rook-ceph] failed: 
armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out 
waiting for pods (namespace=kube-system, labels=(app=rook-ceph-osd)). None 
found! Are `wait.labels` correct? Does `wait.resources` need to exclude 
`type: pod`?
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada Traceback (most 
recent call last):
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada   File 
"/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 
225, in handle_result
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada     result = 
get_result()
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada   File 
"/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 
236, in <lambda>
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada     if 
(handle_result(chart, lambda: deploy_chart(chart))):
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada   File 
"/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 
214, in deploy_chart
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada     chart, 
cg_test_all_charts, prefix, known_releases)
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada   File 
"/usr/local/lib/python3.6/dist-packages/armada/handlers/chart_deploy.py", 
line 248, in execute
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada     
chart_wait.wait(timer)
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada   File 
"/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 134, 
in wait
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada     
wait.wait(timeout=timeout)
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada   File 
"/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 294, 
in wait
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada     modified = 
self._wait(deadline)
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada   File 
"/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 354, 
in _wait
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada     raise 
k8s_exceptions.KubernetesWatchTimeoutException(error)
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada 
armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out 
waiting for pods (namespace=kube-system, labels=(app=rook-ceph-osd)). None 
found! Are `wait.labels` correct? Does `wait.resources` need to exclude 
`type: pod`?
2021-06-30 13:11:11.371 68 ERROR armada.handlers.armada 
2021-06-30 13:11:11.373 68 ERROR armada.handlers.armada [-] Chart deploy(s) 
failed: ['kube-system-rook-ceph']
2021-06-30 13:11:12.193 68 INFO armada.handlers.lock [-] Releasing lock
2021-06-30 13:11:12.196 68 ERROR armada.cli [-] Caught internal exception: 
armada.exceptions.armada_exceptions.ChartDeployException: Exception 
deploying charts: ['kube-system-rook-ceph']
2021-06-30 13:11:12.196 68 ERROR armada.cli Traceback (most recent call 
last):
2021-06-30 13:11:12.196 68 ERROR armada.cli   File 
"/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, 
in safe_invoke
2021-06-30 13:11:12.196 68 ERROR armada.cli     self.invoke()
2021-06-30 13:11:12.196 68 ERROR armada.cli   File 
"/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in 
invoke
2021-06-30 13:11:12.196 68 ERROR armada.cli     resp = 
self.handle(documents, tiller)
2021-06-30 13:11:12.196 68 ERROR armada.cli   File 
"/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, 
in func_wrapper
2021-06-30 13:11:12.196 68 ERROR armada.cli     return future.result()
2021-06-30 13:11:12.196 68 ERROR armada.cli   File 
"/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2021-06-30 13:11:12.196 68 ERROR armada.cli     return self.__get_result()
2021-06-30 13:11:12.196 68 ERROR armada.cli   File 
"/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-06-30 13:11:12.196 68 ERROR armada.cli     raise self._exception
2021-06-30 13:11:12.196 68 ERROR armada.cli   File 
"/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2021-06-30 13:11:12.196 68 ERROR armada.cli     result = 
self.fn(*self.args, **self.kwargs)
2021-06-30 13:11:12.196 68 ERROR armada.cli   File 
"/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in 
handle
2021-06-30 13:11:12.196 68 ERROR armada.cli     return armada.sync()
2021-06-30 13:11:12.196 68 ERROR armada.cli   File 
"/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 
252, in sync
2021-06-30 13:11:12.196 68 ERROR armada.cli     raise 
armada_exceptions.ChartDeployException(failures)
2021-06-30 13:11:12.196 68 ERROR armada.cli 
armada.exceptions.armada_exceptions.ChartDeployException: Exception 
deploying charts: ['kube-system-rook-ceph']
2021-06-30 13:11:12.196 68 ERROR armada.cli 
command terminated with exit code 1


-- 
Sent with Vivaldi Mail. Download Vivaldi for free at vivaldi.com



More information about the Starlingx-discuss mailing list