Re: [Starlingx-discuss] DUPLEX: applying stx-openstack fails with timeout for mariadb

7 May 2019


      I have used the one mentioned in Installation doc from here:

http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke...
...
"Hu, Yong" <yong.hu@intel.com> hat am 7. Mai 2019 um 14:28 geschrieben:
A few weeks ago, I met the similar issue that it was stuck when applying osh-openstack-mariadb helm chart.
At that time, the issue was caused by a failure of mariadb depending on a ceph osd pool (kube-rbd).
Given you were using 0429 image (http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190429T233...),
did you download helm chart from the same build? - http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190429T233...
These 2 have to be aligned.
On 07/05/2019, 7:42 PM, "Marcel Schaible" <marcel@schaible-consulting.de> wrote:
Hi,
we have got the image from 20190429 up and running in a duplex configuration.
When I'll try to bring up the application Services with:
system application-apply stx-openstack
I am getting the error and log file is below.
# system application-list
    stx-openstack | armada-manifest | manifest.yaml | apply-failed | operation aborted, check logs for detail |
I am using this stx-application: stx-openstack-1.0-11-centos-stable-latest.tgz
Thanks
Marcel
2019-05-06 09:13:34.447 11348 INFO armada.handlers.test [-] [chart=ceph-pools-audit]: PASSED: osh-openstack-ceph-pools-audit
    2019-05-06 09:13:34.447 11348 INFO armada.handlers.armada [-] All Charts applied in ChartGroup provisioner.
    2019-05-06 09:13:34.447 11348 INFO armada.handlers.armada [-] Processing ChartGroup: openstack-mariadb (Mariadb), sequenced=True
    2019-05-06 09:13:34.447 11348 INFO armada.handlers.chart_deploy [-] [chart=mariadb]: Processing Chart, release=osh-openstack-mariadb
    2019-05-06 09:13:34.447 11348 INFO armada.handlers.chart_deploy [-] [chart=mariadb]: known: [], release_name: osh-openstack-mariadb
    2019-05-06 09:13:34.448 11348 INFO armada.handlers.chartbuilder [-] [chart=mariadb]: Building dependency chart helm-toolkit for release openstack-mariadb.
    2019-05-06 09:13:34.463 11348 INFO armada.handlers.chart_deploy [-] [chart=mariadb]: Installing release osh-openstack-mariadb in namespace openstack, wait=True, timeout=1800s
    2019-05-06 09:13:34.466 11348 INFO armada.handlers.tiller [-] [chart=mariadb]: Helm install release: wait=True, timeout=1800
    2019-05-06 09:13:59.605 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:14:59.668 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:15:59.730 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:16:59.792 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:17:59.852 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:18:59.906 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:19:59.967 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:21:00.031 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:22:00.094 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:23:00.154 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:24:00.216 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:25:00.279 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:26:00.341 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:27:00.405 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:28:00.468 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:29:00.530 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:30:00.595 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:31:00.651 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:32:00.712 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:33:00.769 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:34:00.826 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:35:00.890 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:36:00.949 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:37:01.009 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:38:01.069 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:39:01.130 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:40:01.194 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:41:01.257 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:42:01.320 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:43:01.385 11348 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:155
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller [-] [chart=mariadb]: Error while installing release osh-openstack-mariadb: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
            status = StatusCode.UNKNOWN
            details = "release osh-openstack-mariadb failed: timed out waiting for the condition"
            debug_error_string = "{"created":"@1557135814.815037033","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-mariadb failed: timed out waiting for the condition","grpc_status":2}"
    >
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller Traceback (most recent call last):
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 462, in install_release
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller     metadata=self.metadata)
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller     return _end_unary_response_blocking(state, call, False, None)
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller     raise _Rendezvous(state, None, None, deadline)
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller      status = StatusCode.UNKNOWN
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller      details = "release osh-openstack-mariadb failed: timed out waiting for the condition"
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller      debug_error_string = "{"created":"@1557135814.815037033","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-mariadb failed: timed out waiting for the condition","grpc_status":2}"
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller >
    2019-05-06 09:43:34.815 11348 ERROR armada.handlers.tiller
    2019-05-06 09:43:34.816 11348 DEBUG armada.handlers.tiller [-] [chart=mariadb]: Helm getting release status for release=osh-openstack-mariadb, version=0 get_release_status /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:528
    2019-05-06 09:43:34.995 11348 DEBUG armada.handlers.tiller [-] [chart=mariadb]: GetReleaseStatus= name: "osh-openstack-mariadb"
    info {
      status {
        code: FAILED
      }
      first_deployed {
        seconds: 1557134014
        nanos: 472287926
      }
      last_deployed {
        seconds: 1557134014
        nanos: 472287926
      }
      Description: "Release \"osh-openstack-mariadb\" failed: timed out waiting for the condition"
    }
    namespace: "openstack"
     get_release_status /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:536
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada [-] Chart deploy [mariadb] failed: armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: osh-openstack-mariadb - Tiller Message: b'Release "osh-openstack-mariadb" failed: timed out waiting for the condition'
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada Traceback (most recent call last):
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 462, in install_release
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada     metadata=self.metadata)
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada     return _end_unary_response_blocking(state, call, False, None)
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada     raise _Rendezvous(state, None, None, deadline)
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada      status = StatusCode.UNKNOWN
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada      details = "release osh-openstack-mariadb failed: timed out waiting for the condition"
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada      debug_error_string = "{"created":"@1557135814.815037033","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-mariadb failed: timed out waiting for the condition","grpc_status":2}"
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada >
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada During handling of the above exception, another exception occurred:
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada
    2019-05-06 09:43:34.995 11348 ERROR armada.handlers.armada Traceback (most recent call last):
_______________________________________________
    Starlingx-discuss mailing list
    Starlingx-discuss@lists.starlingx.io
    http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss