[Starlingx-discuss] Help understanding an application-appy failure

Saul Wold sgw at linux.intel.com
Wed Aug 7 17:08:56 UTC 2019 

Folks,

I am getting a failure during application-apply on a new Baremetal 
install. I installed based on the image built over the weekend.

It appears to be a timeout (see the attached log).

Thoughts, suggestions to debug this further and if a bug is needed

Thanks
     Sau!


> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller [-] [chart=openstack-libvirt]: Error while installing release osh-openstack-libvirt: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
>         status = StatusCode.UNKNOWN
>         details = "release osh-openstack-libvirt failed: timed out waiting for the condition"
>         debug_error_string = "{"created":"@1565150558.052442767","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-libvirt failed: timed out waiting for the condition","grpc_status":2}"
>>
> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller Traceback (most recent call last):
> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 465, in install_release
> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller     metadata=self.metadata)
> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__
> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller     return _end_unary_response_blocking(state, call, False, None)
> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller     raise _Rendezvous(state, None, None, deadline)
> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller      status = StatusCode.UNKNOWN
> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller      details = "release osh-openstack-libvirt failed: timed out waiting for the condition"
> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller      debug_error_string = "{"created":"@1565150558.052442767","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-libvirt failed: timed out waiting for the condition","grpc_status":2}"
> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller >
> 2019-08-07 04:02:38.052 85578 ERROR armada.handlers.tiller ^[[00m
> 2019-08-07 04:02:38.053 85578 DEBUG armada.handlers.tiller [-] [chart=openstack-libvirt]: Helm getting release status for release=osh-openstack-libvirt, version=0 get_release_status /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:531^[[00m
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.wait [-] [chart=openstack-openvswitch]: Timed out waiting for pods (namespace=openstack, labels=(release_group=osh-openstack-openvswitch)). None found! Are `wait.labels` correct? Does `wait.resources` need to exclude `type: pod`?^[[00m
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada [-] Chart deploy [openstack-openvswitch] failed: armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for pods (namespace=openstack, labels=(release_group=osh-openstack-openvswitch)). None found! Are `wait.labels` correct? Does `wait.resources` need to exclude `type: pod`?
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada Traceback (most recent call last):
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 223, in handle_result
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada     result = get_result()
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada   File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada     return self.__get_result()
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada   File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada     raise self._exception
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada   File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada     result = self.fn(*self.args, **self.kwargs)
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 212, in deploy_chart
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada     prefix, known_releases)
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/chart_deploy.py", line 242, in execute
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada     chart_wait.wait(timer)
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 130, in wait
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada     wait.wait(timeout=timeout)
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 292, in wait
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada     modified = self._wait(deadline)
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 349, in _wait
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada     raise k8s_exceptions.KubernetesWatchTimeoutException(error)
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for pods (namespace=openstack, labels=(release_group=osh-openstack-openvswitch)). None found! Are `wait.labels` correct? Does `wait.resources` need to exclude `type: pod`?
> 2019-08-07 04:02:38.129 85578 ERROR armada.handlers.armada ^[[00m
> 2019-08-07 04:02:38.162 85578 DEBUG armada.handlers.tiller [-] [chart=openstack-libvirt]: GetReleaseStatus= name: "osh-openstack-libvirt"
> info {
>   status {
>     code: FAILED
>   }
>   first_deployed {
>     seconds: 1565148757
>     nanos: 968267140
>   }
>   last_deployed {
>     seconds: 1565148757
>     nanos: 968267140
>   }
>   Description: "Release \"osh-openstack-libvirt\" failed: timed out waiting for the condition"
> }
> namespace: "openstack"
>  get_release_status /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:539^[[00m

-------------- next part --------------
A non-text attachment was scrubbed...
Name: stx-openstack-apply.log
Type: text/x-log
Size: 118413 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190807/a1ef02f6/attachment-0001.bin>

More information about the Starlingx-discuss mailing list