[Starlingx-discuss] Python2 -> Python3 (smartpm and rpm-python)

Penney, Don Don.Penney at windriver.com
Thu Dec 19 19:23:13 UTC 2019


For RPM:

1. The stringToVersion function comes from /usr/lib/python2.7/site-packages/rpmUtils/miscutils.py, which is in the "yum" package in CentOS 7. It is not available in CentOS 8, but the function itself is simple. I should be able to just clone it in patch_functions.py, with a reference to the original source, rather than pulling in another external package.

2. There is a python3-rpm package in CentOS 8 that provides the rpm.labelCompare() function, so we should be able to use that. I tried it out on a stock CentOS 8 system in python3.

3. The patch-agent was originally written using "yum", years ago, before switching to "smartpm" to align with Yocto. I should be able to find that history to use as a reference, and get the patch-agent running with "yum" again (or "dnf"). I'll need to make sure the communication between patch agent and controller stays consistent, for interoperability between versions.


I should be able to work this in the master branch, so that I've got a fully running system to test with.

Cheers,
Don.


-----Original Message-----
From: Sun, Austin [mailto:austin.sun at intel.com] 
Sent: Thursday, December 19, 2019 2:01 AM
To: Sun, Austin; Saul Wold; Penney, Don; starlingx-discuss at lists.starlingx.io
Subject: RE: [Starlingx-discuss] Python2 -> Python3 (smartpm and rpm-python) 

Hi Penny:
    Do we have the plan for below story ( python-smartpm and rpm-python) ?  
    CentOS8 upgrade is on-going ,and since python3 is using for CentOS8 , those 2 package can not be built successfully because they don't support python3. 



[1] https://storyboard.openstack.org/#!/story/2006227
[2] https://storyboard.openstack.org/#!/story/2006228

Thanks.
BR
Austin Sun. 
-----Original Message-----
From: Sun, Austin <austin.sun at intel.com> 
Sent: Tuesday, July 16, 2019 2:39 PM
To: Saul Wold <sgw at linux.intel.com>; Penney, Don <Don.Penney at windriver.com>; starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] Python2 -> Python3

Created Story [1] for python-smartpm and [2] for rpm-python. 


[1] https://storyboard.openstack.org/#!/story/2006227 
[2] https://storyboard.openstack.org/#!/story/2006228 

Thanks.
BR
Austin Sun.

-----Original Message-----
From: Saul Wold [mailto:sgw at linux.intel.com] 
Sent: Friday, July 12, 2019 10:22 AM
To: Sun, Austin <austin.sun at intel.com>; Penney, Don <Don.Penney at windriver.com>; starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] Python2 -> Python3



On 7/11/19 6:16 PM, Sun, Austin wrote:
> Hi Penny:
>      Thanks a lot your info.
>      Story [1] is using to track python2to3 for stx.3.0 .
>      Task 35794 was created for upgrade requests-toolbelt.
>      Task 35795  for replacing rpm_python and Task 35796  for 
> replacing python-smartpm

replacing python-smartpm probably need a story on it's own, it will completely change the patch update process.

Sau!

> 
> [1] https://storyboard.openstack.org/#!/story/2006158
> 
> Thank
> BR
> Austin Sun.
> 
> -----Original Message-----
> From: Penney, Don [mailto:Don.Penney at windriver.com]
> Sent: Friday, July 12, 2019 5:12 AM
> To: Saul Wold <sgw at linux.intel.com>; 
> starlingx-discuss at lists.starlingx.io
> Subject: Re: [Starlingx-discuss] Python2 -> Python3
> 
> I think I can use this module in place of the rpm one:
> https://pypi.org/project/version_utils/
> 
> It looks like this provides an equivalent to rpm.labelCompare that should allow me to drop rpm-python.
> 
> 
> -----Original Message-----
> From: Penney, Don
> Sent: Thursday, July 11, 2019 3:50 PM
> To: 'Saul Wold'; starlingx-discuss at lists.starlingx.io
> Subject: RE: [Starlingx-discuss] Python2 -> Python3
> 
> It probably makes sense for me to look at moving the patching framework from "smart" back to "yum" - and at the same, restructure the code so that package management is a backend. It was originally written with yum many years ago, then moved to "smart" to align with yocto.
> 
> We can also look at upversioning requests-toolbelt - we're on an older version solely because there's never been a reason to update it. That said, the version we're using, 0.5.1, is in pypi as py2.py3, so presumably that's ok?
> 
> I can also look at the current use of the rpm module in patching and look for alternatives.
> 
> -----Original Message-----
> From: Saul Wold [mailto:sgw at linux.intel.com]
> Sent: Thursday, July 11, 2019 3:40 PM
> To: starlingx-discuss at lists.starlingx.io
> Subject: Re: [Starlingx-discuss] Python2 -> Python3
> 
> 
> 
> On 7/10/19 7:03 AM, Sun, Austin wrote:
>> Hi All:
>>       The  logical for here is if package from centos directly ,  we will wait upgrading to CentOS 8.x to compliance with python3
>>     Please filter 'Do not contain centos' in Column N ,  then it will show below 11 packages.
>>     As sync in non-OpenStack distro meeting.
>>       We still can filter out the 4 packages from fedora project and python-cephclient as flock servers .
>> So below 6 packages are coming 3rd party which might be not python2to3 compliance.
>>
>> Package                                               | who is using
>> openvswitch                                       | ovs
>> python-cephfs                                    | ceph
>> python-smartpm                               | standalone package
>> qemu-kvm-ev                                     | mtce-compute
>> requests-toolbelt                               | cgcs-patch-controller
>> rpm-python                                        | cgcs-patch-controller
> 
> Can you identify replacement python3 packages for any of these.
> 
> I know we found out that smartpm is used for the patch process, I know 
> that smartpm is also an older project that does not have any upstream 
> support any further, so that will require a fair amount of work.
> 
> Sau!
> 
>>
>> Thanks.
>> BR
>> Austin Sun.
>>
>> -----Original Message-----
>> From: Sun, Austin [mailto:austin.sun at intel.com]
>> Sent: Wednesday, July 10, 2019 4:03 PM
>> To: Xie, Cindy <cindy.xie at intel.com>; Hu, Yong <yong.hu at intel.com>; 
>> 'starlingx-discuss at lists.starlingx.io' 
>> <starlingx-discuss at lists.starlingx.io>
>> Subject: Re: [Starlingx-discuss] Python2 -> Python3
>>
>> Hi All:
>>        New sheet was updated to [1]. Adding column N (repo_info) to record where the package coming from.
>> There are 11 packages not from centos but including python and may not be compatiable python2 and python3.
>>    Package                                               | who is using
>> openvswitch                                       | ovs
>>    python-aniso8601                             | keystone
>>    python-cephclient                             | ceph
>>    python-cephfs                                    | ceph
>>    python-django-bash-completion   | sysinv
>>    python-smartpm                               | standalone package
>>    python-unittest2                               | sysinv
>>    python-XStatic-jquery-ui                  | stx-gui
>>    qemu-kvm-ev                                     | mtce-compute
>>    requests-toolbelt                               | cgcs-patch-controller
>>    rpm-python                                        | cgcs-patch-controller
>>
>>
>> I will continue check those 11 packages .
>>
>> [1] https://bugs.launchpad.net/starlingx/+bug/1808073
>>
>> Thanks.
>> BR
>> Austin Sun.
>>
>> -----Original Message-----
>> From: Sun, Austin
>> Sent: Thursday, July 4, 2019 11:43 AM
>> To: Xie, Cindy <cindy.xie at intel.com>; Hu, Yong <yong.hu at intel.com>; 
>> starlingx-discuss at lists.starlingx.io
>> Subject: RE: [Starlingx-discuss] Python2 -> Python3
>>
>> Hi Cindy:
>>       Yes. we will do it and update sheet.
>>
>> Thanks.
>> BR
>> Austin Sun.
>>
>> -----Original Message-----
>> From: Xie, Cindy [mailto:cindy.xie at intel.com]
>> Sent: Thursday, July 4, 2019 11:37 AM
>> To: Hu, Yong <yong.hu at intel.com>; 
>> starlingx-discuss at lists.starlingx.io
>> Subject: Re: [Starlingx-discuss] Python2 -> Python3
>>
>> Austin,
>> Can you add one more column in your xls sheet: https://bugs.launchpad.net/starlingx/+bug/1808073/+attachment/5274592/+files/rpm_python_status-stx.2.0.xlsx:
>>
>> In your column "I", for your "3rd party" category, to break down "CentOS package" & "3rd party package", so that we can understand how much we can rely on CentOS 8.0, especially for those "risk" ones.
>>
>> Thanks. - cindy
>>
>> -----Original Message-----
>> From: Yong Hu [mailto:yong.hu at intel.com]
>> Sent: Thursday, July 4, 2019 11:25 AM
>> To: starlingx-discuss at lists.starlingx.io
>> Subject: Re: [Starlingx-discuss] Python2 -> Python3
>>
>> A general question, do we have experience and confidence to ensure 2 python versions (both interpreters and pip libs) can co-exist and each of packages can correctly refer to them??
>>
>> In my view the best solution is to wait for CentOS 8.0 :-)
>>
>>
>> On 03/07/2019 2:55 PM, Dean Troyer wrote:
>>> On 7/3/19 4:07 PM, Saul Wold wrote:
>>>> The current proposal seems to be to completely convert the base
>>>> CentOS7.6 system level python to use python3, this carries a high 
>>>> risk factor as changing out all system-level python code could have 
>>>> a cascade effect on system functionality and additional dependencies.
>>>> While
>>>
>>> Changing the distro/system Python version out from under the rest of 
>>> the distro seems like an enormous time sink, much less a significant 
>>> reliability risk.
>>>
>>>> A better solution would be to build python3 and the associated 
>>>> requirements from the existing RHEL EPEL (Extra Packages for 
>>>> Enterprise Linux) Source RPMs repo and install them into the ISO.
>>>> This version correctly installs in a segregated directory tree.
>>>
>>> We would probably want to run a significant subset of the upstream 
>>> OpenStack testing on this combination as it is not (AFAIK) tested there.
>>>     But this is true of any runtime + distro combination that is not 
>>> in the fairly short list of combinations that upstream OpenStack 
>>> actively tests.
>>>
>>>> Another option would be to delay the actual python2 conversion to 
>>>> StarlingX 4.0, the OpenStack Train release will still support python2.
>>>
>>> One downside to this is it leaves us no margin to defer the change 
>>> again, this is our second chance as it were.  OpenStack U (as of 
>>> now) is likely to drop py2 support as a guarantee across-the-board.
>>>
>>>> There is still work that is needed beyond the conversion of the 
>>>> python code itself to things like RPM specfiles data and other 
>>>> source code (such as, C code that has #includes of python2.7). It's 
>>>> not clear to me how much functional testing with python3 has 
>>>> occurred for the flock beyond what Dean has started with devstack.
>>>
>>> I managed to get the fault services running on py3, sysinv fell over 
>>> during the dbsync in my quick post-PTG trial run.  That is as far as 
>>> I took it.  Anyone who wants to try can pick out the local.conf I 
>>> posted [0]
>>>
>>> dt
>>>
>>> [0] http://paste.openstack.org/show/753844/
>>>
>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>
> 
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> 
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss


More information about the Starlingx-discuss mailing list