Hi Don Penney,
Thank you, I will try only (will revert existing modifications) with these changes for the components. But setup.py doesn't need similar changes (Ex: cgts-client/setup.py) ?
Thank you, Haridhar Kalvala
From: Penney, Don Don.Penney@windriver.com Sent: Tuesday, January 5, 2021 8:33 PM To: Kalvala, Haridhar haridhar.kalvala@intel.com; Bailey, Henry Albert (Al) Al.Bailey@windriver.com; starlingx-discuss@lists.starlingx.io; Wold, Saul Saul.Wold@windriver.com; Jones, Bruce E bruce.e.jones@intel.com; Sun, Austin austin.sun@intel.com Cc: Mukherjee, Sanjay K sanjay.k.mukherjee@intel.com; Kumar, Sharath sharath.kumar@intel.com; Bhat, Gopalkrishna gopalkrishna.bhat@intel.com; Khalil, Ghada Ghada.Khalil@windriver.com; Wensley, Barton Barton.Wensley@windriver.com Subject: RE: [Starlingx-discuss] Flock components python3 migration query on master.
I think Al's original suggestion was to avoid changing the on-host installation of packages. Rather, to ensure that the modules required for the container are building a py3-compatible wheel that can be used for the image. Some of the wheels are already py2.py3 and available for use in a py3 image.
The fm-common module, for example, has a compiled library and is py2-specific: /wheels/fm_core-1.0-cp27-cp27mu-linux_x86_64.whl
So updating the spec to use %py3_build_wheel allows us to build a wheel that should be compatible with a py3 image: $ git diff diff --git a/fm-common/centos/fm-common.spec b/fm-common/centos/fm-common.spec index a5c3618..f8d4ac9 100644 --- a/fm-common/centos/fm-common.spec +++ b/fm-common/centos/fm-common.spec @@ -19,6 +19,8 @@ BuildRequires: python-devel BuildRequires: python-setuptools BuildRequires: python2-pip BuildRequires: python2-wheel +BuildRequires: python3-pip +BuildRequires: python3-wheel Buildrequires: json-c-devel Requires: json-c
@@ -55,6 +57,7 @@ MINOR=`echo $VER | awk -F . '{print $2}'` make MAJOR=$MAJOR MINOR=$MINOR %{?_smp_mflags} %{__python} setup.py build %py2_build_wheel +%py3_build_wheel
%install rm -rf %{buildroot}
With that change, we end up with wheels for both py2 and py3: $ rpm -qpl std/rpmbuild/RPMS/fm-common-wheels-1.0-50.tis.x86_64.rpm /wheels/fm_core-1.0-cp27-cp27mu-linux_x86_64.whl /wheels/fm_core-1.0-cp36-cp36m-linux_x86_64.whl
Additionally, some modules may need an update to their setup.cfg to add py3 support. For example, cgts-client is currently py2-only: https://opendev.org/starlingx/config/src/branch/master/sysinv/cgts-client/cg...
So adding the following would be needed to get the wheel to support both py2 and py3, with a universal wheel: $ git diff diff --git a/sysinv/cgts-client/cgts-client/setup.cfg b/sysinv/cgts-client/cgts-client/setup.cfg index e1ee3e7b..0ca4697f 100644 --- a/sysinv/cgts-client/cgts-client/setup.cfg +++ b/sysinv/cgts-client/cgts-client/setup.cfg @@ -13,6 +13,8 @@ classifier = Programming Language :: Python Programming Language :: Python :: 2 Programming Language :: Python :: 2.7 + Programming Language :: Python :: 3 + Programming Language :: Python :: 3.6
[global] setup-hooks = @@ -33,3 +35,6 @@ autodoc_index_modules = False tag_build = tag_date = 0 tag_svn_revision = 0 + +[wheel] +universal = 1
With this change, I now see the following: $ rpm -qpl std/rpmbuild/RPMS/cgts-client-wheels-1.0-199.tis.x86_64.rpm /wheels/cgtsclient-1.0-py2.py3-none-any.whl
A quick check of our wheel RPMs can show us which don't currently support py3. Most of these would be similar to the cgtsclient change, where the setup.cfg would need to be update to build a universal wheel that supports py2 and py3: $ for f in std/rpmbuild/RPMS/*-wheels* ; do rpm -qpl $f | grep -q -e '-py3-' -e '-py2.py3-' -e cp36 || echo $f; done std/rpmbuild/RPMS/ceph-manager-wheels-1.0-20.tis.x86_64.rpm std/rpmbuild/RPMS/cgts-client-wheels-1.0-198.tis.x86_64.rpm std/rpmbuild/RPMS/controllerconfig-wheels-1.0-222.tis.x86_64.rpm std/rpmbuild/RPMS/distributedcloud-client-wheels-1.0.0-1.tis.22.noarch.rpm std/rpmbuild/RPMS/distributedcloud-wheels-1.0.0-1.tis.215.noarch.rpm std/rpmbuild/RPMS/libvirt-python-wheels-4.7.0-1.tis.5.x86_64.rpm std/rpmbuild/RPMS/nfv-wheels-1.0-163.tis.x86_64.rpm std/rpmbuild/RPMS/nova-api-proxy-wheels-1.0-30.tis.x86_64.rpm std/rpmbuild/RPMS/platform-util-wheels-1.0-52.tis.noarch.rpm std/rpmbuild/RPMS/python-3parclient-wheels-4.2.3-0.tis.3.noarch.rpm std/rpmbuild/RPMS/python-barbicanclient-wheels-4.9.0-1.tis.3.noarch.rpm std/rpmbuild/RPMS/python-cephclient-wheels-13.2.2.0-10.tis.noarch.rpm std/rpmbuild/RPMS/python-django-horizon-wheels-15.1.0-1.tis.44.noarch.rpm std/rpmbuild/RPMS/python-lefthandclient-wheels-2.1.0-0.tis.3.noarch.rpm std/rpmbuild/RPMS/python-ryu-wheels-4.19-0.tis.5.noarch.rpm std/rpmbuild/RPMS/sm-api-wheels-1.0-35.tis.x86_64.rpm std/rpmbuild/RPMS/sm-client-wheels-1.0-24.tis.x86_64.rpm std/rpmbuild/RPMS/sm-tools-wheels-1.0-17.tis.x86_64.rpm std/rpmbuild/RPMS/starlingx-dashboard-wheels-1.0-251.tis.noarch.rpm std/rpmbuild/RPMS/sysinv-wheels-1.0-1857.tis.x86_64.rpm std/rpmbuild/RPMS/vm-topology-wheels-1.0-17.tis.noarch.rpm
From: Kalvala, Haridhar <haridhar.kalvala@intel.commailto:haridhar.kalvala@intel.com> Sent: Tuesday, January 5, 2021 8:40 AM To: Bailey, Henry Albert (Al) <Al.Bailey@windriver.commailto:Al.Bailey@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io; Wold, Saul <Saul.Wold@windriver.commailto:Saul.Wold@windriver.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Sun, Austin <austin.sun@intel.commailto:austin.sun@intel.com> Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.commailto:sanjay.k.mukherjee@intel.com>; Kumar, Sharath <sharath.kumar@intel.commailto:sharath.kumar@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.commailto:gopalkrishna.bhat@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.commailto:Ghada.Khalil@windriver.com>; Wensley, Barton <Barton.Wensley@windriver.commailto:Barton.Wensley@windriver.com> Subject: Re: [Starlingx-discuss] Flock components python3 migration query on master.
[Please note this e-mail is from an EXTERNAL e-mail address] Yes Henry. Also #131/#133 And also when we migrate(as python3.6) fmclient, cgtsclient, distributed cloud etc... deployment fails because sysinv can't find the modules in 2.7.
Thank you, Haridhar Kalvala
From: Bailey, Henry Albert (Al) <Al.Bailey@windriver.commailto:Al.Bailey@windriver.com> Sent: Tuesday, January 5, 2021 6:47 PM To: Kalvala, Haridhar <haridhar.kalvala@intel.commailto:haridhar.kalvala@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io; Wold, Saul <Saul.Wold@windriver.commailto:Saul.Wold@windriver.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Sun, Austin <austin.sun@intel.commailto:austin.sun@intel.com> Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.commailto:sanjay.k.mukherjee@intel.com>; Kumar, Sharath <sharath.kumar@intel.commailto:sharath.kumar@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.commailto:gopalkrishna.bhat@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.commailto:Ghada.Khalil@windriver.com>; Wensley, Barton <Barton.Wensley@windriver.commailto:Barton.Wensley@windriver.com> Subject: Re: [Starlingx-discuss] Flock components python3 migration query on master.
Are you referring to the hard-coding in the spec file? https://github.com/starlingx/config/blob/master/sysinv/sysinv/centos/sysinv....
which gets used when its built https://github.com/starlingx/config/blob/master/sysinv/sysinv/centos/sysinv....
Al ________________________________ From: Kalvala, Haridhar <haridhar.kalvala@intel.commailto:haridhar.kalvala@intel.com> Sent: Tuesday, January 5, 2021 2:49 AM To: Bailey, Henry Albert (Al) <Al.Bailey@windriver.commailto:Al.Bailey@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io>; Wold, Saul <Saul.Wold@windriver.commailto:Saul.Wold@windriver.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Sun, Austin <austin.sun@intel.commailto:austin.sun@intel.com> Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.commailto:sanjay.k.mukherjee@intel.com>; Kumar, Sharath <sharath.kumar@intel.commailto:sharath.kumar@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.commailto:gopalkrishna.bhat@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.commailto:Ghada.Khalil@windriver.com>; Kalvala, Haridhar <haridhar.kalvala@intel.commailto:haridhar.kalvala@intel.com>; Wensley, Barton <Barton.Wensley@windriver.commailto:Barton.Wensley@windriver.com> Subject: RE: [Starlingx-discuss] Flock components python3 migration query on master.
[Please note this e-mail is from an EXTERNAL e-mail address]
Hello All,
In continuation to issue below, After migrating(as per below changing specs and cfg files) few flock components and built/pushed horizon container. But did not observe presence of migration components in python3.6 path of controller (/usr/lib/python3.6/site-packages....). So, I proceed with building all of packages and iso with changes.
Deployment of ISO failed because sysinv couldn't find migrated flock components ( fmclient, cgtsclient etc) , since sysinv is built and search in python2.7 path while these components are migrated to python3.6. Any thoughts, on how to do and solve partial (few components which are also inter dependent) migration to pyhton3.6 on master branch ?
Thank you,
Haridhar Kalvala
From: Kalvala, Haridhar <haridhar.kalvala@intel.commailto:haridhar.kalvala@intel.com> Sent: Tuesday, December 15, 2020 11:28 AM To: Bailey, Henry Albert (Al) <Al.Bailey@windriver.commailto:Al.Bailey@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io; Wensley, Barton <Barton.Wensley@windriver.commailto:Barton.Wensley@windriver.com>; Wold, Saul <Saul.Wold@windriver.commailto:Saul.Wold@windriver.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Sun, Austin <austin.sun@intel.commailto:austin.sun@intel.com> Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.commailto:sanjay.k.mukherjee@intel.com>; Kumar, Sharath <sharath.kumar@intel.commailto:sharath.kumar@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.commailto:gopalkrishna.bhat@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.commailto:Ghada.Khalil@windriver.com> Subject: Re: [Starlingx-discuss] Flock components python3 migration query on master. Importance: High
Hi Henry,
Thank you for response.
I tried the same approach(spec and setup configuration file changes) on master branch for few flock components related with fault containerization.
After changing few requires to python3, I observe below issues. Which I am not clear how to solve.
Below are the few more blocker we are facing:
* Starlingx-dashboard :
* can't migrate python-pbr to python3-pbr. * Cephclient is throwing errors on trying to migrate to py3.
* Fmclient :
* Nfv components migration is a issue.
So I am looking for some inputs here who have done it earlier.
Sample error log:
INFO: pip is looking at multiple versions of horizon to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of distributedcloud-client to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of coverage to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cgtsclient to determine which version is compatible with other requirements. This could take a while.
^[[91mERROR: Cannot install pylint==1.4.0, pylint==1.4.1, pylint==1.4.2, pylint==1.4.3, pylint==1.4.4 and pylint==1.4.5 because these package versions have conflicting dependencies.
^[[0m
The conflict is caused by:
pylint 1.4.5 depends on logilab-common>=0.53.0
pylint 1.4.4 depends on logilab-common>=0.53.0
pylint 1.4.3 depends on logilab-common>=0.53.0
pylint 1.4.2 depends on logilab-common>=0.53.0
pylint 1.4.1 depends on logilab-common>=0.53.0
pylint 1.4.0 depends on logilab-common>=0.53.0
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
^[[91mERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
^[[0m^[[91mWARNING: You are using pip version 20.3.1; however, version 20.3.2 is available.
You should consider upgrading via the '/var/lib/openstack/bin/python -m pip install --upgrade pip' command.
^[[0mThe command '/bin/sh -c /opt/loci/scripts/install.sh' returned a non-zero code: 1
Command (docker) failed, attempt 1 of 1.
Max command attempts reached. Aborting...
Thank you,
Haridhar Kalvala
From: Bailey, Henry Albert (Al) <Al.Bailey@windriver.commailto:Al.Bailey@windriver.com> Sent: Tuesday, December 15, 2020 4:25 AM To: Kalvala, Haridhar <haridhar.kalvala@intel.commailto:haridhar.kalvala@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io; Wensley, Barton <Barton.Wensley@windriver.commailto:Barton.Wensley@windriver.com>; Wold, Saul <Saul.Wold@windriver.commailto:Saul.Wold@windriver.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Sun, Austin <austin.sun@intel.commailto:austin.sun@intel.com> Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.commailto:sanjay.k.mukherjee@intel.com>; Kumar, Sharath <sharath.kumar@intel.commailto:sharath.kumar@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.commailto:gopalkrishna.bhat@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.commailto:Ghada.Khalil@windriver.com> Subject: Re: [Starlingx-discuss] Flock components python3 migration query on master.
The containers typically install the python components they need using wheels
If the flock components are updated in their setup.cfg to be built universal wheels (which are py2 and py3 compatible) then the wheels being generated from the spec should be able to be included.
That can be done in master, assuming the code runs in both py2 and py3 without needing major changes.
Al
________________________________
From: Kalvala, Haridhar <haridhar.kalvala@intel.commailto:haridhar.kalvala@intel.com> Sent: Monday, December 14, 2020 6:12 AM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io>; Wensley, Barton <Barton.Wensley@windriver.commailto:Barton.Wensley@windriver.com>; Wold, Saul <Saul.Wold@windriver.commailto:Saul.Wold@windriver.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Sun, Austin <austin.sun@intel.commailto:austin.sun@intel.com> Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.commailto:sanjay.k.mukherjee@intel.com>; Kumar, Sharath <sharath.kumar@intel.commailto:sharath.kumar@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.commailto:gopalkrishna.bhat@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.commailto:Ghada.Khalil@windriver.com> Subject: Re: [Starlingx-discuss] Flock components python3 migration query on master.
[Please note this e-mail is from an EXTERNAL e-mail address]
Reminder for below request.
Thank You,
Haridhar Kalvala
From: Kalvala, Haridhar Sent: Thursday, December 10, 2020 10:18 AM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io; Wensley, Barton <Barton.Wensley@windriver.commailto:Barton.Wensley@windriver.com>; Saul Wold <Saul.Wold@windriver.commailto:Saul.Wold@windriver.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Sun, Austin <austin.sun@intel.commailto:austin.sun@intel.com> Cc: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.commailto:sanjay.k.mukherjee@intel.com>; Kumar, Sharath <sharath.kumar@intel.commailto:sharath.kumar@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.commailto:gopalkrishna.bhat@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.commailto:Ghada.Khalil@windriver.com> Subject: Flock components python3 migration query on master.
Hello All,
We are working on FM containerization (FM panel in open stack dashboard and backend pod service(fault)).
As part of this activity, we are doing stx-horizon docker build. Which will include components from flock (for fm calls). As a reference example, please refer to this patchhttps://review.opendev.org/c/starlingx/upstream/+/661423/4/openstack/python-horizon/centos/stx-horizon.stable_docker_image.
The front panel would be calling flock components fmclient, cgtsclient, distributedcloud etc.
These components are python2 build. For FM panel to work we need migration of these components (seeing lot of dependency issues) from python2 to python3 on master branch.
Yesterday(9-Dec-2020) in non-openstack distro meeting, the discussion was flock python2 dependent components cannot be migrated in master branch and will done in centos8 migration branch/stream.
So, I request for clarity/decision from community on this blocker for FM containerization.
Earlier in stx-3.0(stein's openstack) as in above link, these component were part of stx-horizon docker build. But these were removed(as per patch below) as part of stx-4.0 migration (stein -->train --> Ussuri), as latest openstack only supports python3.
Patch details:
commit b8b897a028cda3936f17c5ac6b73af146c5de96b (tag: v4.0.0.rc0)
Author: yuchengde <yu.chengde@99cloud.netmailto:yu.chengde@99cloud.net>
Depends-on: https://review.opendev.org/#/c/731461/
Change-Id: I415ea75eb24d8b1c3151b63efd2e82075e68f26e
Signed-off-by: Yu Chengde yu.chengde@99cloud.commailto:yu.chengde@99cloud.com
Thank you,
Haridhar Kalvala