[Starlingx-discuss] python-eventlet upgrade issue
Hi, Anyone familiar with python-eventlet? I meet a problem when upgrade python-eventlet from "0.18.4-2" to "0.20.1-2", when do CentOS7.5 upgrade. Eventlet will hook/patch some basic python function, such as "call"/"check_call" from subprocess.py The issue is that, after upgrade to 0.20.1-2 version, these functions don't work as expected. For the pass log (with eventlet 0.18.4), below log is as expected: 2018-08-28 05:38:33.816 21142 INFO sysinv.agent.pci [-] Could not determine DPDK support for NIC (vendor 0x8086 device: 0x100e), defaulting to False But for the fail log(with eventlet 0.20.1), it seems the exception is not handled correctly: 2018-08-28 03:48:21.687 21302 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Command '['query_pci_id', '-v 0x8086', '-d 0x100e']' returned non-zero exit status 1 Is there any code style change with new eventlet? Thanks. Pass log: 2018-08-28 05:38:33.477 21142 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on localhost:5672 2018-08-28 05:38:33.485 21142 INFO sysinv.agent.manager [-] Sysinv Agent audit running inv_get_and_report. 2018-08-28 05:38:33.552 21177 INFO migrate.versioning.api [-] 0 -> 1... 2018-08-28 05:38:33.750 21177 INFO migrate.versioning.api [-] done 2018-08-28 05:38:33.750 21177 INFO migrate.versioning.api [-] 1 -> 2... 2018-08-28 05:38:33.816 21142 INFO sysinv.agent.pci [-] Could not determine DPDK support for NIC (vendor 0x8086 device: 0x100e), defaulting to False 2018-08-28 05:38:34.032 21142 INFO sysinv.agent.pci [-] Could not determine DPDK support for NIC (vendor 0x8086 device: 0x100e), defaulting to False 2018-08-28 05:38:34.033 21142 WARNING sysinv.agent.pci [-] Enabling device enp0s8 to query link speed 2018-08-28 05:38:34.036 21142 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp0s8 (flags: 0x1002) 2018-08-28 05:38:34.036 21142 WARNING sysinv.agent.pci [-] Disabling device enp0s8 after querying link speed Fail log: 2018-08-28 03:48:21.373 21302 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on localhost:5672 2018-08-28 03:48:21.382 21302 INFO sysinv.agent.manager [-] Sysinv Agent audit running inv_get_and_report. 2018-08-28 03:48:21.463 21337 INFO migrate.versioning.api [-] 0 -> 1... 2018-08-28 03:48:21.631 21337 INFO migrate.versioning.api [-] done 2018-08-28 03:48:21.631 21337 INFO migrate.versioning.api [-] 1 -> 2... 2018-08-28 03:48:21.687 21302 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Command '['query_pci_id', '-v 0x8086', '-d 0x100e']' returned non-zero exit status 1 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task Traceback (most recent call last): 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 182, in run_periodic_tasks 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task task(self, context) 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1010, in _agent_audit 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task force_updates=None) 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1028, in agent_audit 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task self.ihost_inv_get_and_report(icontext) 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 575, in ihost_inv_get_and_report 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task pci_net_array = self._ipci_operator.pci_get_net_attrs(inic.pciaddr) 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/pci.py", line 470, in pci_get_net_attrs 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task stdout=fnull, stderr=fnull) 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task raise CalledProcessError(retcode, cmd) 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task CalledProcessError: Command '['query_pci_id', '-v 0x8086', '-d 0x100e']' returned non-zero exit status 1 2018-08-28 03:48:21.687 21302 TRACE sysinv.openstack.common.periodic_task 2018-08-28 03:48:21.929 21337 INFO migrate.versioning.api [-] done Best Regards Shuicheng
On 08/29/2018 08:51 PM, Lin, Shuicheng wrote:
Hi,
Anyone familiar with python-eventlet? I meet a problem when upgrade python-eventlet from “0.18.4-2” to “0.20.1-2”, when do CentOS7.5 upgrade.
Eventlet will hook/patch some basic python function, such as “call”/”check_call” from subprocess.py
The issue is that, after upgrade to 0.20.1-2 version, these functions don’t work as expected.
For the pass log (with eventlet 0.18.4), below log is as expected:
2018-08-28 05:38:33.816 21142 INFO sysinv.agent.pci [-] Could not determine DPDK support for NIC (vendor 0x8086 device: 0x100e), defaulting to False
But for the fail log(with eventlet 0.20.1), it seems the exception is not handled correctly:
2018-08-28 03:48:21.687 21302 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Command '['query_pci_id', '-v 0x8086', '-d 0x100e']' returned non-zero exit status 1
Is there any code style change with new eventlet?
There are a couple odd things going on here. First, you found a bug. The check at http://git.openstack.org/cgit/openstack/stx-config/tree/sysinv/sysinv/sysinv... currently reads if e.returncode == '1' when it should be if e.returncode == 1 This explains why you were getting the "Could not determine DPDK support..." message" before, when really you should have been getting "DPDK does support NIC..." I can't explain why the CalledProcessError was caught by sysinv.openstack.common.periodic_task when it should have been caught (and swallowed) by http://git.openstack.org/cgit/openstack/stx-config/tree/sysinv/sysinv/sysinv... Chris
Hi all, The issue is caused by patch method in the new eventlet. It is still open at eventlet issue list as below link. https://github.com/eventlet/eventlet/issues/413 I don't want to change our code to workaround it. So I decide not to upgrade eventlet src rpm. Feel free to notify me if you have concern on it. Hi Chris, Thanks for your help. I will correct the '1' error if it is not done yet. Best Regards Shuicheng -----Original Message----- From: Chris Friesen [mailto:chris.friesen@windriver.com] Sent: Friday, August 31, 2018 3:33 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] python-eventlet upgrade issue On 08/29/2018 08:51 PM, Lin, Shuicheng wrote:
Hi,
Anyone familiar with python-eventlet? I meet a problem when upgrade python-eventlet from “0.18.4-2” to “0.20.1-2”, when do CentOS7.5 upgrade.
Eventlet will hook/patch some basic python function, such as “call”/”check_call” from subprocess.py
The issue is that, after upgrade to 0.20.1-2 version, these functions don’t work as expected.
For the pass log (with eventlet 0.18.4), below log is as expected:
2018-08-28 05:38:33.816 21142 INFO sysinv.agent.pci [-] Could not determine DPDK support for NIC (vendor 0x8086 device: 0x100e), defaulting to False
But for the fail log(with eventlet 0.20.1), it seems the exception is not handled correctly:
2018-08-28 03:48:21.687 21302 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Command '['query_pci_id', '-v 0x8086', '-d 0x100e']' returned non-zero exit status 1
Is there any code style change with new eventlet?
There are a couple odd things going on here. First, you found a bug. The check at http://git.openstack.org/cgit/openstack/stx-config/tree/sysinv/sysinv/sysinv... currently reads if e.returncode == '1' when it should be if e.returncode == 1 This explains why you were getting the "Could not determine DPDK support..." message" before, when really you should have been getting "DPDK does support NIC..." I can't explain why the CalledProcessError was caught by sysinv.openstack.common.periodic_task when it should have been caught (and swallowed) by http://git.openstack.org/cgit/openstack/stx-config/tree/sysinv/sysinv/sysinv... Chris _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Wed, Aug 29, 2018 at 9:51 PM, Lin, Shuicheng <shuicheng.lin@intel.com> wrote:
Anyone familiar with python-eventlet? I meet a problem when upgrade python-eventlet from “0.18.4-2” to “0.20.1-2”, when do CentOS7.5 upgrade. [...] The issue is that, after upgrade to 0.20.1-2 version, these functions don’t work as expected.
Looking at upstream OpenStack requirements for Pike, eventlet is listed in global-requirements (the minimum required version) as eventlet!=0.18.3,>=0.18.2,!=0.20.1,<0.21.0 # MIT and in upper-contraints.txt (what is actually tests in the gate) as eventlet===0.20.0 In order to be co-installable with OpenStack we must maintain similar versions for the STX services. FWIW, rocky's versions of the above are similar: eventlet!=0.18.3,!=0.20.1,!=0.21.0,!=0.23.0 and eventlet===0.20.0 Also note that OpenStack is preparing[0] to raise the u-c version of eventlet to 0.24.1, which I believe is current master. We should be preparing 0.20.0 for packaging. Even though OpenStack master is mostly ready for 0.24.1 I doubt that Pike is nor will it be made so... Background OpenStack maintains both minimum and maximum version requirements for Python dependencies that must be shared by all OpenStack projects. This is to maintain co-installability since Python can not install multiple versions of a package without resorting to techniques such as virtual environments. There are CI jobs for checking these, and projects are free to be stricter in their reqirements.txt files as long as they fit within the globally defined min/max. StarlingX services (flock) need to be compatible with Pike, although it is not currently enforced. dt [0] http://lists.openstack.org/pipermail/openstack-dev/2018-August/133728.html -- Dean Troyer dtroyer@gmail.com
Thanks Dean. I checked eventlet 0.20.0. It has the same issue. https://github.com/eventlet/eventlet/issues/413 Also I cannot find 0.20.0 in centos repo. So I will keep current 0.18.4 version, which also meet the global-requirements for pike. I will try to go through the packages upgraded with the global-requirements and upper-constraints. Will try to match them if possible. https://github.com/openstack/requirements/blob/stable/pike/global-requiremen... https://github.com/openstack/requirements/blob/stable/pike/upper-constraints... Best Regards Shuicheng -----Original Message----- From: Dean Troyer [mailto:dtroyer@gmail.com] Sent: Thursday, September 6, 2018 12:58 AM To: Lin, Shuicheng <shuicheng.lin@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] python-eventlet upgrade issue On Wed, Aug 29, 2018 at 9:51 PM, Lin, Shuicheng <shuicheng.lin@intel.com> wrote:
Anyone familiar with python-eventlet? I meet a problem when upgrade python-eventlet from “0.18.4-2” to “0.20.1-2”, when do CentOS7.5 upgrade. [...] The issue is that, after upgrade to 0.20.1-2 version, these functions don’t work as expected.
Looking at upstream OpenStack requirements for Pike, eventlet is listed in global-requirements (the minimum required version) as eventlet!=0.18.3,>=0.18.2,!=0.20.1,<0.21.0 # MIT and in upper-contraints.txt (what is actually tests in the gate) as eventlet===0.20.0 In order to be co-installable with OpenStack we must maintain similar versions for the STX services. FWIW, rocky's versions of the above are similar: eventlet!=0.18.3,!=0.20.1,!=0.21.0,!=0.23.0 and eventlet===0.20.0 Also note that OpenStack is preparing[0] to raise the u-c version of eventlet to 0.24.1, which I believe is current master. We should be preparing 0.20.0 for packaging. Even though OpenStack master is mostly ready for 0.24.1 I doubt that Pike is nor will it be made so... Background OpenStack maintains both minimum and maximum version requirements for Python dependencies that must be shared by all OpenStack projects. This is to maintain co-installability since Python can not install multiple versions of a package without resorting to techniques such as virtual environments. There are CI jobs for checking these, and projects are free to be stricter in their reqirements.txt files as long as they fit within the globally defined min/max. StarlingX services (flock) need to be compatible with Pike, although it is not currently enforced. dt [0] http://lists.openstack.org/pipermail/openstack-dev/2018-August/133728.html -- Dean Troyer dtroyer@gmail.com
participants (3)
-
Chris Friesen
-
Dean Troyer
-
Lin, Shuicheng