Hi,
Jfyi..
Below issue of mysqlite with Django on openstack is resolved applying “Chen,Haochuan Z” patch from upstream.
https://stackoverflow.com/questions/55657752/django-installing-mysqlclient-error-mysqlclient-1-3-13-or-newer-is-required
Fix with guide from stackoverflow. After openstack image
upgrade to ussuri, Django upgrade to 2.2, which request
mysqlclient newer than 1.3.13, conflict with version
0.9.3 in current image. Fix with WA in above link, and
currently we use pymysql not mysql.
Closes-Bug: 1908117
Change-Id: Ic7054c6736993394d92bb0aec25397fd22f84d31
Signed-off-by: Chen, Haochuan Z <haochuan.z.chen@intel.com>
??From <https://opendev.org/starlingx/upstream> ??
Regards,
Haridhar Kalvala
From: Kalvala, Haridhar
Sent: Tuesday, January 19, 2021 11:50 AM
To: Bai, Haiqing <Haiqing.Bai@windriver.com>; Hu, Yong <yong.hu@intel.com>; starlingx-discuss@lists.starlingx.io
Cc: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Penney, Don <Don.Penney@windriver.com>; Miller, Frank <Frank.Miller@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>
Subject: RE: [Starlingx-discuss] Horizon container deployment, Pod status (CrashLoopBackOff) issue.
Thank you Haiqing. Yeah, But this is part of scss python package tool(Django-pyscss etc may also need to change) which is used by many others modules which are python2 compiled. Is there any way to specify to use particular use wheel for
one module ?
Regards,
Haridhar Kalvala
From: Bai, Haiqing <Haiqing.Bai@windriver.com>
Sent: Tuesday, January 19, 2021 11:43 AM
To: Kalvala, Haridhar <haridhar.kalvala@intel.com>; Hu, Yong <yong.hu@intel.com>;
starlingx-discuss@lists.starlingx.io
Cc: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Penney, Don <Don.Penney@windriver.com>; Miller, Frank <Frank.Miller@windriver.com>;
Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>
Subject: RE: [Starlingx-discuss] Horizon container deployment, Pod status (CrashLoopBackOff) issue.
This is py3 upgrade issue. Maybe you should change the code from “inspect.getargspec()” to “inspect.getfullargspec()”.
I once fixed the same kind of issue of centos8 on stx/distributedcloud-client/dcmanagerclient/openstack/common/cliutils.py: line 60
distributedcloud-client$ git diff distributedcloud-client/dcmanagerclient/openstack/common/cliutils.py
diff --git a/distributedcloud-client/dcmanagerclient/openstack/common/cliutils.py b/distributedcloud-client/dcmanagerclient/openstack/common/cliutils.py
index d40d6ee..e59f52f 100644
--- a/distributedcloud-client/dcmanagerclient/openstack/common/cliutils.py
+++ b/distributedcloud-client/dcmanagerclient/openstack/common/cliutils.py
@@ -57,7 +57,7 @@ def validate_args(fn, *args, **kwargs):
:param arg: the positional arguments supplied
:param kwargs: the keyword arguments supplied
"""
-
argspec = inspect.getargspec(fn)
+
argspec = inspect.getfullargspec(fn)
num_defaults = len(argspec.defaults or [])
++
python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())'
+
SITE_PACKAGES_ROOT=/var/lib/openstack/lib/python3.6/site-packages
+
rm -f /var/lib/openstack/lib/python3.6/site-packages/openstack_dashboard/local/local_settings.py
+
chown -R horizon /var/lib/openstack/lib/python3.6/site-packages/openstack_dashboard/local/
/tmp/horizon.sh:
line 17: :wq: command not found
+
'[' -f /etc/apache2/envvars ']'
+
rm -rf /var/run/apache2/htcacheclean
+
PANEL_DIR=/var/lib/openstack/lib/python3.6/site-packages/heat_dashboard/enabled
+
'[' -d /var/lib/openstack/lib/python3.6/site-packages/heat_dashboard/enabled ']'
+
PANEL_DIR=/var/lib/openstack/lib/python3.6/site-packages/neutron_taas_dashboard/enabled
+
'[' -d /var/lib/openstack/lib/python3.6/site-packages/neutron_taas_dashboard/enabled ']'
+
cd /var/lib/openstack/lib/python3.6/site-packages/openstack_dashboard
+
/tmp/manage.py compilemessages
result = tb * (i + nesting) + "@media -sass-debug-info{filename{font-family:file\:\/\/%s}line{font-family:\\00003%s}}" % (filename, lineno) + nl
/var/lib/openstack/lib64/python3.6/site-packages/scss/cssdefs.py:516: DeprecationWarning: invalid escape sequence \s
''', re.VERBOSE)
/var/lib/openstack/lib64/python3.6/site-packages/scss/namespace.py:172: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature()
or inspect.getfullargspec()
argspec = inspect.getargspec(function)
Traceback (most recent call last):
File "/tmp/manage.py", line 19, in <module>
execute_from_command_line(sys.argv)
File "/var/lib/openstack/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/var/lib/openstack/lib/python3.6/site-packages/django/core/management/__init__.py", line 357, in execute
django.setup()
File "/var/lib/openstack/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/var/lib/openstack/lib/python3.6/site-packages/django/apps/registry.py", line 114, in populate
app_config.import_models()
File "/var/lib/openstack/lib/python3.6/site-packages/django/apps/config.py", line 211, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/var/lib/openstack/lib/python3.6/site-packages/django/contrib/auth/models.py", line 2, in <module>
from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
File "/var/lib/openstack/lib/python3.6/site-packages/django/contrib/auth/base_user.py", line 47, in <module>
class AbstractBaseUser(models.Model):
File "/var/lib/openstack/lib/python3.6/site-packages/django/db/models/base.py", line 117, in __new__
new_class.add_to_class('_meta', Options(meta, app_label))
File "/var/lib/openstack/lib/python3.6/site-packages/django/db/models/base.py", line 321, in add_to_class
value.contribute_to_class(cls, name)
File "/var/lib/openstack/lib/python3.6/site-packages/django/db/models/options.py", line 204, in contribute_to_class
self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
File "/var/lib/openstack/lib/python3.6/site-packages/django/db/__init__.py", line 28, in __getattr__
return getattr(connections[DEFAULT_DB_ALIAS], item)
File "/var/lib/openstack/lib/python3.6/site-packages/django/db/utils.py", line 201, in __getitem__
backend = load_backend(db['ENGINE'])
File "/var/lib/openstack/lib/python3.6/site-packages/django/db/utils.py", line 110, in load_backend
return import_module('%s.base' % backend_name)
File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/var/lib/openstack/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 36, in <module>
raise ImproperlyConfigured('mysqlclient 1.3.13 or newer is required; you have %s.' % Database.__version__)
django.core.exceptions.ImproperlyConfigured: mysqlclient 1.3.13 or newer is required; you have 0.9.3.
Regards,
Haridhar Kalvala
From: Hu, Yong <yong.hu@intel.com>
Sent: Monday, January 11, 2021 1:58 PM
To: Kalvala, Haridhar <haridhar.kalvala@intel.com>;
starlingx-discuss@lists.starlingx.io
Cc: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Penney, Don <Don.Penney@windriver.com>; Miller, Frank <Frank.Miller@windriver.com>;
Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>
Subject: Re: [Starlingx-discuss] Horizon container deployment, Pod status (CrashLoopBackOff) issue.
You might get some hints by inspecting the pod by:
# kubectl -n openstack describe pod
horizon-d777d8b7d-cpvq7
# kubectl -n openstack logs horizon-d777d8b7d-cpvq7
From: "Kalvala, Haridhar" <haridhar.kalvala@intel.com>
Date: Monday, January 11, 2021 at 4:19 PM
To: "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io>
Cc: "Bailey, Henry Albert (Al)" <Al.Bailey@windriver.com>, "Penney, Don" <Don.Penney@windriver.com>, "Miller, Frank" <Frank.Miller@windriver.com>,
"Mukherjee, Sanjay K" <sanjay.k.mukherjee@intel.com>, "Bhat, Gopalkrishna" <gopalkrishna.bhat@intel.com>
Subject: [Starlingx-discuss] Horizon container deployment, Pod status (CrashLoopBackOff) issue.
Hello all,
Any pointers on how can I debug below CrashLoopBackOff status issue for horizon container image.
Pod status :
horizon-d777d8b7d-cpvq7 0/1 CrashLoopBackOff 43 3h17m
horizon-db-init-4pcd4 0/1 Completed 0 3h17m
horizon-db-sync-26ltd 0/1 Completed 0 3h17m
Error Log:
2021-01-11 16:06:21.812 414 DEBUG armada.handlers.wait [-] [chart=openstack-horizon]: pod horizon-d777d8b7d-cpvq7 not ready: Waiting for pod horizon-d777d8b7d-cpvq7 to be ready... handle_resource /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:260[00m
2021-01-11 16:07:17.488 414 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176[00m
2021-01-11 16:08:17.612 414 DEBUG armada.handlers.lock [-] Updating lock update_lock /usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py:176[00m
2021-01-11 16:08:19.221 414 ERROR armada.handlers.wait [-] [chart=openstack-horizon]: Timed out waiting for pods (namespace=openstack, labels=(release_group=osh-openstack-horizon)). These pods were not ready=['horizon-d777d8b7d-cpvq7'][00m
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada [-] Chart deploy [openstack-horizon] failed: armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for pods (namespace=openstack,
labels=(release_group=osh-openstack-horizon)). These pods were not ready=['horizon-d777d8b7d-cpvq7']
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada Traceback (most recent call last):
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 225, in handle_result
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada result = get_result()
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada return self.__get_result()
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada raise self._exception
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada result = self.fn(*self.args, **self.kwargs)
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 214, in deploy_chart
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada chart, cg_test_all_charts, prefix, known_releases)
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/chart_deploy.py", line 248, in execute
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada chart_wait.wait(timer)
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 134, in wait
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada wait.wait(timeout=timeout)
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 294, in wait
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada modified = self._wait(deadline)
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py", line 354, in _wait
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada raise k8s_exceptions.KubernetesWatchTimeoutException(error)
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for pods (namespace=openstack, labels=(release_group=osh-openstack-horizon)).
These pods were not ready=['horizon-d777d8b7d-cpvq7']
2021-01-11 16:08:19.222 414 ERROR armada.handlers.armada [00m
2021-01-11 16:08:19.223 414 ERROR armada.handlers.armada [-] Chart deploy(s) failed: ['openstack-horizon'][00m
2021-01-11 16:08:19.625 414 INFO armada.handlers.lock [-] Releasing lock[00m
2021-01-11 16:08:19.668 414 ERROR armada.cli [-] Caught internal exception: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openstack-horizon']
2021-01-11 16:08:19.668 414 ERROR armada.cli Traceback (most recent call last):
2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2021-01-11 16:08:19.668 414 ERROR armada.cli self.invoke()
2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2021-01-11 16:08:19.668 414 ERROR armada.cli resp = self.handle(documents, tiller)
2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2021-01-11 16:08:19.668 414 ERROR armada.cli return future.result()
2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2021-01-11 16:08:19.668 414 ERROR armada.cli return self.__get_result()
2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-01-11 16:08:19.668 414 ERROR armada.cli raise self._exception
2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2021-01-11 16:08:19.668 414 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle
2021-01-11 16:08:19.668 414 ERROR armada.cli return armada.sync()
2021-01-11 16:08:19.668 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 252, in sync
2021-01-11 16:08:19.668 414 ERROR armada.cli raise armada_exceptions.ChartDeployException(failures)
2021-01-11 16:08:19.668 414 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['openstack-horizon']
2021-01-11 16:08:19.668 414 ERROR armada.cli [00m
command terminated with exit code 1
Regards,
Haridhar Kalvala