You might get some hints by inspecting the pod by: # kubectl -n openstack describe pod horizon-d777d8b7d-cpvq7 # kubectl -n openstack logs horizon-d777d8b7d-cpvq7 From: "Kalvala, Haridhar" <haridhar.kalvala@intel.com> Date: Monday, January 11, 2021 at 4:19 PM To: "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Cc: "Bailey, Henry Albert (Al)" <Al.Bailey@windriver.com>, "Penney, Don" <Don.Penney@windriver.com>, "Miller, Frank" <Frank.Miller@windriver.com>, "Mukherjee, Sanjay K" <sanjay.k.mukherjee@intel.com>, "Bhat, Gopalkrishna" <gopalkrishna.bhat@intel.com> Subject: [Starlingx-discuss] Horizon container deployment, Pod status (CrashLoopBackOff) issue. Hello all, Any pointers on how can I debug below CrashLoopBackOff status issue for horizon container image. Pod status : horizon-d777d8b7d-cpvq7 0/1 CrashLoopBackOff 43 3h17m horizon-db-init-4pcd4 0/1 Completed 0 3h17m horizon-db-sync-26ltd 0/1 Completed 0 3h17m Error Log: 2021-01-11 16:06:21.812 414 DEBUG armada.handlers.wait [-] [chart=openstack-horizon]: pod horizon-d777d8b7d-cpvq7 not ready: Waiting for pod horizon-d777d8b7d-cpvq7 to be ready... handle_resource /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:260[00m 2021-01-11 16:07:17.488 414 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176[00m 2021-01-11 16:08:17.612 414 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176[00m 2021-01-11 16:08:19.221 414 ERROR armada.handlers.wait [-] [chart=openstack-horizon]: Timed out waiting for pods (namespace=openstack, labels=(release_group=osh-openstack-horizon)). These pods were not ready=['horizon-d777d8b7d-cpvq7'][00m 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada [-] Chart deploy [openstack-horizon] failed: armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for pods (namespace=openstack, labels=(release_group=osh-openstack-horizon)). These pods were not ready=['horizon-d777d8b7d-cpvq7'] 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada Traceback (most recent call last): 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 225, in handle_result 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada result = get_result() 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada return self.__get_result() 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada raise self._exception 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada result = self.fn(*self.args, **self.kwargs) 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 214, in deploy_chart 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada chart, cg_test_all_charts, prefix, known_releases) 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/chart_deploy.py", line 248, in execute 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada chart_wait.wait(timer) 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 134, in wait 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada wait.wait(timeout=timeout) 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 294, in wait 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada modified = self._wait(deadline) 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 354, in _wait 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada raise k8s_exceptions.KubernetesWatchTimeoutException(error) 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for pods (namespace=openstack, labels=(release_group=osh-openstack-horizon)). These pods were not ready=['horizon-d777d8b7d-cpvq7'] 2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada [00m 2021-01-11 16:08:19.223 414 ERROR armada.handlers.armada [-] Chart deploy(s) failed: ['openstack-horizon'][00m 2021-01-11 16:08:19.625 414 INFO armada.handlers.lock [-] Releasing lock[00m 2021-01-11 16:08:19.668 414 ERROR armada.cli [-] Caught internal exception: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openstack-horizon'] 2021-01-11 16:08:19.668 414 ERROR armada.cli Traceback (most recent call last): 2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke 2021-01-11 16:08:19.668 414 ERROR armada.cli self.invoke() 2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke 2021-01-11 16:08:19.668 414 ERROR armada.cli resp = self.handle(documents, tiller) 2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper 2021-01-11 16:08:19.668 414 ERROR armada.cli return future.result() 2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result 2021-01-11 16:08:19.668 414 ERROR armada.cli return self.__get_result() 2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2021-01-11 16:08:19.668 414 ERROR armada.cli raise self._exception 2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run 2021-01-11 16:08:19.668 414 ERROR armada.cli result = self.fn(*self.args, **self.kwargs) 2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle 2021-01-11 16:08:19.668 414 ERROR armada.cli return armada.sync() 2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 252, in sync 2021-01-11 16:08:19.668 414 ERROR armada.cli raise armada_exceptions.ChartDeployException(failures) 2021-01-11 16:08:19.668 414 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openstack-horizon'] 2021-01-11 16:08:19.668 414 ERROR armada.cli [00m command terminated with exit code 1 Regards, Haridhar Kalvala