[Starlingx-discuss] Flock components python3 migration query on master.
Bailey, Henry Albert (Al)
Al.Bailey at windriver.com
Wed Jan 6 15:00:39 UTC 2021
Those extra classifier entries in setup.py are redundant and can just be removed.
setup.cfg provides them already.
sysinv also would want this type of cleanup
Also note, python 3.5 entries should probably also be cleaned up.
Normally python 3.6 is where python3 compatability would start. I see tsconfig is listing only py3.5
Al
________________________________
From: Kalvala, Haridhar <haridhar.kalvala at intel.com>
Sent: Tuesday, January 5, 2021 11:06 PM
To: Penney, Don <Don.Penney at windriver.com>; Bailey, Henry Albert (Al) <Al.Bailey at windriver.com>; starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>; Wold, Saul <Saul.Wold at windriver.com>; Jones, Bruce E <bruce.e.jones at intel.com>; Sun, Austin <austin.sun at intel.com>
Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com>; Kumar, Sharath <sharath.kumar at intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>; Wensley, Barton <Barton.Wensley at windriver.com>
Subject: RE: [Starlingx-discuss] Flock components python3 migration query on master.
[Please note this e-mail is from an EXTERNAL e-mail address]
Hi Don Penney,
Thank you, I will try only (will revert existing modifications) with these changes for the components.
But setup.py doesn’t need similar changes (Ex: cgts-client/setup.py) ?
Thank you,
Haridhar Kalvala
From: Penney, Don <Don.Penney at windriver.com>
Sent: Tuesday, January 5, 2021 8:33 PM
To: Kalvala, Haridhar <haridhar.kalvala at intel.com>; Bailey, Henry Albert (Al) <Al.Bailey at windriver.com>; starlingx-discuss at lists.starlingx.io; Wold, Saul <Saul.Wold at windriver.com>; Jones, Bruce E <bruce.e.jones at intel.com>; Sun, Austin <austin.sun at intel.com>
Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com>; Kumar, Sharath <sharath.kumar at intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>; Wensley, Barton <Barton.Wensley at windriver.com>
Subject: RE: [Starlingx-discuss] Flock components python3 migration query on master.
I think Al’s original suggestion was to avoid changing the on-host installation of packages. Rather, to ensure that the modules required for the container are building a py3-compatible wheel that can be used for the image. Some of the wheels are already py2.py3 and available for use in a py3 image.
The fm-common module, for example, has a compiled library and is py2-specific:
/wheels/fm_core-1.0-cp27-cp27mu-linux_x86_64.whl
So updating the spec to use %py3_build_wheel allows us to build a wheel that should be compatible with a py3 image:
$ git diff
diff --git a/fm-common/centos/fm-common.spec b/fm-common/centos/fm-common.spec
index a5c3618..f8d4ac9 100644
--- a/fm-common/centos/fm-common.spec
+++ b/fm-common/centos/fm-common.spec
@@ -19,6 +19,8 @@ BuildRequires: python-devel
BuildRequires: python-setuptools
BuildRequires: python2-pip
BuildRequires: python2-wheel
+BuildRequires: python3-pip
+BuildRequires: python3-wheel
Buildrequires: json-c-devel
Requires: json-c
@@ -55,6 +57,7 @@ MINOR=`echo $VER | awk -F . '{print $2}'`
make MAJOR=$MAJOR MINOR=$MINOR %{?_smp_mflags}
%{__python} setup.py build
%py2_build_wheel
+%py3_build_wheel
%install
rm -rf %{buildroot}
With that change, we end up with wheels for both py2 and py3:
$ rpm -qpl std/rpmbuild/RPMS/fm-common-wheels-1.0-50.tis.x86_64.rpm
/wheels/fm_core-1.0-cp27-cp27mu-linux_x86_64.whl
/wheels/fm_core-1.0-cp36-cp36m-linux_x86_64.whl
Additionally, some modules may need an update to their setup.cfg to add py3 support. For example, cgts-client is currently py2-only:
https://opendev.org/starlingx/config/src/branch/master/sysinv/cgts-client/cgts-client/setup.cfg
So adding the following would be needed to get the wheel to support both py2 and py3, with a universal wheel:
$ git diff
diff --git a/sysinv/cgts-client/cgts-client/setup.cfg b/sysinv/cgts-client/cgts-client/setup.cfg
index e1ee3e7b..0ca4697f 100644
--- a/sysinv/cgts-client/cgts-client/setup.cfg
+++ b/sysinv/cgts-client/cgts-client/setup.cfg
@@ -13,6 +13,8 @@ classifier =
Programming Language :: Python
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
+ Programming Language :: Python :: 3
+ Programming Language :: Python :: 3.6
[global]
setup-hooks =
@@ -33,3 +35,6 @@ autodoc_index_modules = False
tag_build =
tag_date = 0
tag_svn_revision = 0
+
+[wheel]
+universal = 1
With this change, I now see the following:
$ rpm -qpl std/rpmbuild/RPMS/cgts-client-wheels-1.0-199.tis.x86_64.rpm
/wheels/cgtsclient-1.0-py2.py3-none-any.whl
A quick check of our wheel RPMs can show us which don’t currently support py3. Most of these would be similar to the cgtsclient change, where the setup.cfg would need to be update to build a universal wheel that supports py2 and py3:
$ for f in std/rpmbuild/RPMS/*-wheels* ; do rpm -qpl $f | grep -q -e '-py3-' -e '-py2.py3-' -e cp36 || echo $f; done
std/rpmbuild/RPMS/ceph-manager-wheels-1.0-20.tis.x86_64.rpm
std/rpmbuild/RPMS/cgts-client-wheels-1.0-198.tis.x86_64.rpm
std/rpmbuild/RPMS/controllerconfig-wheels-1.0-222.tis.x86_64.rpm
std/rpmbuild/RPMS/distributedcloud-client-wheels-1.0.0-1.tis.22.noarch.rpm
std/rpmbuild/RPMS/distributedcloud-wheels-1.0.0-1.tis.215.noarch.rpm
std/rpmbuild/RPMS/libvirt-python-wheels-4.7.0-1.tis.5.x86_64.rpm
std/rpmbuild/RPMS/nfv-wheels-1.0-163.tis.x86_64.rpm
std/rpmbuild/RPMS/nova-api-proxy-wheels-1.0-30.tis.x86_64.rpm
std/rpmbuild/RPMS/platform-util-wheels-1.0-52.tis.noarch.rpm
std/rpmbuild/RPMS/python-3parclient-wheels-4.2.3-0.tis.3.noarch.rpm
std/rpmbuild/RPMS/python-barbicanclient-wheels-4.9.0-1.tis.3.noarch.rpm
std/rpmbuild/RPMS/python-cephclient-wheels-13.2.2.0-10.tis.noarch.rpm
std/rpmbuild/RPMS/python-django-horizon-wheels-15.1.0-1.tis.44.noarch.rpm
std/rpmbuild/RPMS/python-lefthandclient-wheels-2.1.0-0.tis.3.noarch.rpm
std/rpmbuild/RPMS/python-ryu-wheels-4.19-0.tis.5.noarch.rpm
std/rpmbuild/RPMS/sm-api-wheels-1.0-35.tis.x86_64.rpm
std/rpmbuild/RPMS/sm-client-wheels-1.0-24.tis.x86_64.rpm
std/rpmbuild/RPMS/sm-tools-wheels-1.0-17.tis.x86_64.rpm
std/rpmbuild/RPMS/starlingx-dashboard-wheels-1.0-251.tis.noarch.rpm
std/rpmbuild/RPMS/sysinv-wheels-1.0-1857.tis.x86_64.rpm
std/rpmbuild/RPMS/vm-topology-wheels-1.0-17.tis.noarch.rpm
From: Kalvala, Haridhar <haridhar.kalvala at intel.com<mailto:haridhar.kalvala at intel.com>>
Sent: Tuesday, January 5, 2021 8:40 AM
To: Bailey, Henry Albert (Al) <Al.Bailey at windriver.com<mailto:Al.Bailey at windriver.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Wold, Saul <Saul.Wold at windriver.com<mailto:Saul.Wold at windriver.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>
Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com<mailto:sanjay.k.mukherjee at intel.com>>; Kumar, Sharath <sharath.kumar at intel.com<mailto:sharath.kumar at intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com<mailto:gopalkrishna.bhat at intel.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Wensley, Barton <Barton.Wensley at windriver.com<mailto:Barton.Wensley at windriver.com>>
Subject: Re: [Starlingx-discuss] Flock components python3 migration query on master.
[Please note this e-mail is from an EXTERNAL e-mail address]
Yes Henry. Also #131/#133
And also when we migrate(as python3.6) fmclient, cgtsclient, distributed cloud etc… deployment fails because sysinv can’t find the modules in 2.7.
Thank you,
Haridhar Kalvala
From: Bailey, Henry Albert (Al) <Al.Bailey at windriver.com<mailto:Al.Bailey at windriver.com>>
Sent: Tuesday, January 5, 2021 6:47 PM
To: Kalvala, Haridhar <haridhar.kalvala at intel.com<mailto:haridhar.kalvala at intel.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Wold, Saul <Saul.Wold at windriver.com<mailto:Saul.Wold at windriver.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>
Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com<mailto:sanjay.k.mukherjee at intel.com>>; Kumar, Sharath <sharath.kumar at intel.com<mailto:sharath.kumar at intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com<mailto:gopalkrishna.bhat at intel.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Wensley, Barton <Barton.Wensley at windriver.com<mailto:Barton.Wensley at windriver.com>>
Subject: Re: [Starlingx-discuss] Flock components python3 migration query on master.
Are you referring to the hard-coding in the spec file?
https://github.com/starlingx/config/blob/master/sysinv/sysinv/centos/sysinv.spec#L60
which gets used when its built
https://github.com/starlingx/config/blob/master/sysinv/sysinv/centos/sysinv.spec#L79
Al
________________________________
From: Kalvala, Haridhar <haridhar.kalvala at intel.com<mailto:haridhar.kalvala at intel.com>>
Sent: Tuesday, January 5, 2021 2:49 AM
To: Bailey, Henry Albert (Al) <Al.Bailey at windriver.com<mailto:Al.Bailey at windriver.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io> <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Wold, Saul <Saul.Wold at windriver.com<mailto:Saul.Wold at windriver.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>
Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com<mailto:sanjay.k.mukherjee at intel.com>>; Kumar, Sharath <sharath.kumar at intel.com<mailto:sharath.kumar at intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com<mailto:gopalkrishna.bhat at intel.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Kalvala, Haridhar <haridhar.kalvala at intel.com<mailto:haridhar.kalvala at intel.com>>; Wensley, Barton <Barton.Wensley at windriver.com<mailto:Barton.Wensley at windriver.com>>
Subject: RE: [Starlingx-discuss] Flock components python3 migration query on master.
[Please note this e-mail is from an EXTERNAL e-mail address]
Hello All,
In continuation to issue below, After migrating(as per below changing specs and cfg files) few flock components and built/pushed horizon container. But did not observe presence of migration components in python3.6 path of controller (/usr/lib/python3.6/site-packages….). So, I proceed with building all of packages and iso with changes.
Deployment of ISO failed because sysinv couldn’t find migrated flock components ( fmclient, cgtsclient etc) , since sysinv is built and search in python2.7 path while these components are migrated to python3.6. Any thoughts, on how to do and solve partial (few components which are also inter dependent) migration to pyhton3.6 on master branch ?
Thank you,
Haridhar Kalvala
From: Kalvala, Haridhar <haridhar.kalvala at intel.com<mailto:haridhar.kalvala at intel.com>>
Sent: Tuesday, December 15, 2020 11:28 AM
To: Bailey, Henry Albert (Al) <Al.Bailey at windriver.com<mailto:Al.Bailey at windriver.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Wensley, Barton <Barton.Wensley at windriver.com<mailto:Barton.Wensley at windriver.com>>; Wold, Saul <Saul.Wold at windriver.com<mailto:Saul.Wold at windriver.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>
Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com<mailto:sanjay.k.mukherjee at intel.com>>; Kumar, Sharath <sharath.kumar at intel.com<mailto:sharath.kumar at intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com<mailto:gopalkrishna.bhat at intel.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>
Subject: Re: [Starlingx-discuss] Flock components python3 migration query on master.
Importance: High
Hi Henry,
Thank you for response.
I tried the same approach(spec and setup configuration file changes) on master branch for few flock components related with fault containerization.
After changing few requires to python3, I observe below issues. Which I am not clear how to solve.
Below are the few more blocker we are facing:
* Starlingx-dashboard :
* can’t migrate python-pbr to python3-pbr.
* Cephclient is throwing errors on trying to migrate to py3.
* Fmclient :
* Nfv components migration is a issue.
So I am looking for some inputs here who have done it earlier.
Sample error log:
INFO: pip is looking at multiple versions of horizon to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of distributedcloud-client to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of coverage to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cgtsclient to determine which version is compatible with other requirements. This could take a while.
^[[91mERROR: Cannot install pylint==1.4.0, pylint==1.4.1, pylint==1.4.2, pylint==1.4.3, pylint==1.4.4 and pylint==1.4.5 because these package versions have conflicting dependencies.
^[[0m
The conflict is caused by:
pylint 1.4.5 depends on logilab-common>=0.53.0
pylint 1.4.4 depends on logilab-common>=0.53.0
pylint 1.4.3 depends on logilab-common>=0.53.0
pylint 1.4.2 depends on logilab-common>=0.53.0
pylint 1.4.1 depends on logilab-common>=0.53.0
pylint 1.4.0 depends on logilab-common>=0.53.0
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
^[[91mERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
^[[0m^[[91mWARNING: You are using pip version 20.3.1; however, version 20.3.2 is available.
You should consider upgrading via the '/var/lib/openstack/bin/python -m pip install --upgrade pip' command.
^[[0mThe command '/bin/sh -c /opt/loci/scripts/install.sh' returned a non-zero code: 1
Command (docker) failed, attempt 1 of 1.
Max command attempts reached. Aborting...
Thank you,
Haridhar Kalvala
From: Bailey, Henry Albert (Al) <Al.Bailey at windriver.com<mailto:Al.Bailey at windriver.com>>
Sent: Tuesday, December 15, 2020 4:25 AM
To: Kalvala, Haridhar <haridhar.kalvala at intel.com<mailto:haridhar.kalvala at intel.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Wensley, Barton <Barton.Wensley at windriver.com<mailto:Barton.Wensley at windriver.com>>; Wold, Saul <Saul.Wold at windriver.com<mailto:Saul.Wold at windriver.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>
Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com<mailto:sanjay.k.mukherjee at intel.com>>; Kumar, Sharath <sharath.kumar at intel.com<mailto:sharath.kumar at intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com<mailto:gopalkrishna.bhat at intel.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>
Subject: Re: [Starlingx-discuss] Flock components python3 migration query on master.
The containers typically install the python components they need using wheels
If the flock components are updated in their setup.cfg to be built universal wheels (which are py2 and py3 compatible) then the wheels being generated from the spec should be able to be included.
That can be done in master, assuming the code runs in both py2 and py3 without needing major changes.
Al
________________________________
From: Kalvala, Haridhar <haridhar.kalvala at intel.com<mailto:haridhar.kalvala at intel.com>>
Sent: Monday, December 14, 2020 6:12 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io> <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Wensley, Barton <Barton.Wensley at windriver.com<mailto:Barton.Wensley at windriver.com>>; Wold, Saul <Saul.Wold at windriver.com<mailto:Saul.Wold at windriver.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>
Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com<mailto:sanjay.k.mukherjee at intel.com>>; Kumar, Sharath <sharath.kumar at intel.com<mailto:sharath.kumar at intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com<mailto:gopalkrishna.bhat at intel.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>
Subject: Re: [Starlingx-discuss] Flock components python3 migration query on master.
[Please note this e-mail is from an EXTERNAL e-mail address]
Reminder for below request.
Thank You,
Haridhar Kalvala
From: Kalvala, Haridhar
Sent: Thursday, December 10, 2020 10:18 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Wensley, Barton <Barton.Wensley at windriver.com<mailto:Barton.Wensley at windriver.com>>; Saul Wold <Saul.Wold at windriver.com<mailto:Saul.Wold at windriver.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>
Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com<mailto:sanjay.k.mukherjee at intel.com>>; Kumar, Sharath <sharath.kumar at intel.com<mailto:sharath.kumar at intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com<mailto:gopalkrishna.bhat at intel.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>
Subject: Flock components python3 migration query on master.
Hello All,
We are working on FM containerization (FM panel in open stack dashboard and backend pod service(fault)).
As part of this activity, we are doing stx-horizon docker build. Which will include components from flock (for fm calls). As a reference example, please refer to this patch<https://review.opendev.org/c/starlingx/upstream/+/661423/4/openstack/python-horizon/centos/stx-horizon.stable_docker_image>.
The front panel would be calling flock components fmclient, cgtsclient, distributedcloud etc.
These components are python2 build. For FM panel to work we need migration of these components (seeing lot of dependency issues) from python2 to python3 on master branch.
Yesterday(9-Dec-2020) in non-openstack distro meeting, the discussion was flock python2 dependent components cannot be migrated in master branch and will done in centos8 migration branch/stream.
So, I request for clarity/decision from community on this blocker for FM containerization.
Earlier in stx-3.0(stein’s openstack) as in above link, these component were part of stx-horizon docker build. But these were removed(as per patch below) as part of stx-4.0 migration (stein -->train --> Ussuri), as latest openstack only supports python3.
Patch details:
commit b8b897a028cda3936f17c5ac6b73af146c5de96b (tag: v4.0.0.rc0)
Author: yuchengde <yu.chengde at 99cloud.net<mailto:yu.chengde at 99cloud.net>>
Depends-on: https://review.opendev.org/#/c/731461/
Change-Id: I415ea75eb24d8b1c3151b63efd2e82075e68f26e
Signed-off-by: Yu Chengde yu.chengde at 99cloud.com<mailto:yu.chengde at 99cloud.com>
Thank you,
Haridhar Kalvala
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20210106/2d3e4e93/attachment-0001.html>
More information about the Starlingx-discuss
mailing list