[Starlingx-discuss] CentOS 7.5 upgrade status update
Hi all, For CentOS7.5 upgrade, now we have 38 out of 42 src rpm upgrade merged to f/centos75 branch. While 3 src rpm (python-eventlet/lighttpd/puppet-haproxy) is under debug due to have deploy issue, and 1 src rpm (openstack-aodh) is still under rebase. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Please help report any issue you find. Thanks. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream. 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Best Regards Shuicheng From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, August 30, 2018 12:08 PM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Notes: StarlingX non-OpenStack Distro meeting, 8/29 Agenda & notes for 8/29 meetings: * Review Team objectives & prioritize (Cindy) Review team goal. Concern about the % patch reduction by 50% and 90% by mid of 2019 and end of 2019. Saul: define a more realistic # after patch review. * CentOS 7.5 upgrade status (Cindy) most of sRPM upgrade patches has been merged into f/centos 7.5 branch; 5 to be merged: 2 patches is under review, 1 in progress. 2 patches causes basic deployment issues: lighttpd (https://review.openstack.org/#/c/596263/) & python-eventlet-0.20.1-2.el7.src.rpm (patch# TBD); plan is to keep the SRPM in 7.4 w/ other sRPM for a test build to GDC and WR. Trending: end of Friday China time for test build; ISO to GDC, build instructions (including pending patches if needs to be cherry-picked), branch-name. Shuicheng will send this out. * OS kernel version upgrade: can we upgrade to another kernel (to be planned). Code freeze for 2018.10: Sep 26th. AR: Shuicheng file a story for OS kernel upgrade into 7.5 kernel. * Out of tree kernel driver is part of the scope? AR: Shuicheng to scope what is the kernel drivers needs to be upgraded it's OK if we cannot hit Sep 26 code freeze date. * Ceph package upgrade (Vivian Zhu is working) - part of storage team. AR: Cindy to invite Vivian into this meeting to provide the status update. recommendation from Brent: very high risk of upgrading Ceph to hit Sep 26th code freeze date. * libvirt and qemu upgrade status (Ghada) Jim has a basic build for libvirt w/ patch rebase. Finishing Qemu trending end of this week. will be off next week. Patch will not be merged before WR internal testing, will be merged before Sep 20th? * non-OpenStack patches review status (Saul) master xls: https://docs.google.com/spreadsheets/d/1nKnkweuxcqvVOoRcpnTYMVUUv1RoAugOWXMj... Intel or gmail address to Bruce if you'd like to get access. Goal: to get 1st pass of analysis results by PTG (Sep 12th) * 2018.10 release content review (Ghada) * Active all storyboard: https://storyboard.openstack.org/#!/story/list?status=active&tags=stx.distro.other&project_group_id=86 * Tag 2018.10: https://storyboard.openstack.org/#!/story/list?status=active&tags=stx.distro.other&tags=stx.2018.10&project_group_id=86 * OS independent topics (Saul) - defered EOM
Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Thanks Shuicheng. We'll run some tests in Wind River. Numan will be priming this activity. He'll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they've moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We'll run some tests in Wind River. Numan will be priming this activity. He'll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Hi Shuicheng, I've completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they've moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We'll run some tests in Wind River. Numan will be priming this activity. He'll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
As previously, we found lighttpd 1.4.49 and 1.4.50 both have runtime issues, there is one help-needed email to Brent see attached. If we have to switch to 1.4.50, then let's prioritize the trouble-shooting together on this. @Jason, if you can help Zhipeng on debugging this issue, that will be very much helpful: https://review.openstack.org/#/c/596263/ thanks. - cindy From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I've completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they've moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We'll run some tests in Wind River. Numan will be priming this activity. He'll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. :) Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I've completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don't have this issue. I used last week's master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let's further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they've moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We'll run some tests in Wind River. Numan will be priming this activity. He'll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/ It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test. Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. :) Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I've completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don't have this issue. I used last week's master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let's further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they've moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We'll run some tests in Wind River. Numan will be priming this activity. He'll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Hi Jason, I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. https://review.openstack.org/#/c/600920/ Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/ It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test. Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. :) Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is: Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid. -Jason From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: RE: CentOS 7.5 upgrade status update Hi Jason, I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. https://review.openstack.org/#/c/600920/ Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/ It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test. Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. :) Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment? I am syncing my code to latest to double confirm it. Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: RE: CentOS 7.5 upgrade status update Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is: Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid. -Jason From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Jason, I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. https://review.openstack.org/#/c/600920/ Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/ It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test. Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. :) Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “ Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment? I am syncing my code to latest to double confirm it. Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is: Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid. -Jason From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Jason, I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. https://review.openstack.org/#/c/600920/ Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/ It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test. Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. :) Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Adding Nimalini From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “ Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment? I am syncing my code to latest to double confirm it. Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is: Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid. -Jason From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Jason, I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. https://review.openstack.org/#/c/600920/ Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/ It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test. Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. :) Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…” -Nimalini From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update Adding Nimalini From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “ Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment? I am syncing my code to latest to double confirm it. Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is: Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid. -Jason From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Jason, I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. https://review.openstack.org/#/c/600920/ Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/ It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test. Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. :) Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Is this the same issue as yesterday’s build? or this is different issue now? From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…” -Nimalini From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update Adding Nimalini From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “ Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment? I am syncing my code to latest to double confirm it. Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is: Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid. -Jason From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Jason, I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. https://review.openstack.org/#/c/600920/ Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/ It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test. Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. :) Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
This is on the same build (yesterday’s build). From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update Is this the same issue as yesterday’s build? or this is different issue now? From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>> Subject: RE: CentOS 7.5 upgrade status update It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…” -Nimalini From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update Adding Nimalini From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “ Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment? I am syncing my code to latest to double confirm it. Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is: Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid. -Jason From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Jason, I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. https://review.openstack.org/#/c/600920/ Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/ It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test. Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. :) Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
But as Jason reported yesterday, the build cannot be installed with error messages: Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid. We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test? Thx. - cindy From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update This is on the same build (yesterday’s build). From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update Is this the same issue as yesterday’s build? or this is different issue now? From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>> Subject: RE: CentOS 7.5 upgrade status update It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…” -Nimalini From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update Adding Nimalini From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “ Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment? I am syncing my code to latest to double confirm it. Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is: Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid. -Jason From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Jason, I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. https://review.openstack.org/#/c/600920/ Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/ It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test. Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. :) Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Hi Cindy, We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.openstack.org/#/c/601204/) but I do not know if that is expected to resolve the issue -J From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update But as Jason reported yesterday, the build cannot be installed with error messages: Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid. We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test? Thx. - cindy From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: CentOS 7.5 upgrade status update This is on the same build (yesterday’s build). From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update Is this the same issue as yesterday’s build? or this is different issue now? From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>> Subject: RE: CentOS 7.5 upgrade status update It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…” -Nimalini From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update Adding Nimalini From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “ Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment? I am syncing my code to latest to double confirm it. Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is: Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid. -Jason From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Jason, I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. https://review.openstack.org/#/c/600920/ Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/ It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test. Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. :) Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue. Is it possible that you try to deploy it in virtual environment? Thx. - cindy From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update Hi Cindy, We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.openstack.org/#/c/601204/) but I do not know if that is expected to resolve the issue -J From: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com<mailto:Nimalini.Rasa@windriver.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: CentOS 7.5 upgrade status update But as Jason reported yesterday, the build cannot be installed with error messages: Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid. We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test? Thx. - cindy From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: CentOS 7.5 upgrade status update This is on the same build (yesterday’s build). From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update Is this the same issue as yesterday’s build? or this is different issue now? From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>> Subject: RE: CentOS 7.5 upgrade status update It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…” -Nimalini From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update Adding Nimalini From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “ Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment? I am syncing my code to latest to double confirm it. Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is: Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid. -Jason From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Jason, I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. https://review.openstack.org/#/c/600920/ Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/ It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test. Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. :) Best Regards Shuicheng From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: RE: CentOS 7.5 upgrade status update Hi Shuicheng, I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered: - The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https://review.openstack.org/#/c/598388/6/openstack/openstack-aodh/centos/me... -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta-modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django-horizon and some magnum packages. I had to pull in https://review.openstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code. After these changes, I was able to produce a .iso -Jason From: McKenna, Jason <Jason.McKenna@windriver.com<mailto:Jason.McKenna@windriver.com>> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm. -Jason From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Waheed, Numan <Numan.Waheed@windriver.com<mailto:Numan.Waheed@windriver.com>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week. Regards, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update Hi all, Most task of CentOS7.5 upgrade in story 2003389<https://storyboard.openstack.org/#!/story/2003389> is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet. Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python-os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd. Besides srpm, we also upgraded 260+ rpm packages to 7.5 version. Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639 2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO. For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389 Feel free to contact me if you have trouble on it. Best Regards Shuicheng
Hi You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any) -Erich On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Cindy, With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7-5.1804.el7.centos.1.x86_64.rpm, and the patches checked in do not apply against either package. Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort. -J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Tue, Sep 11, 2018 at 11:21 AM, McKenna, Jason <Jason.McKenna@windriver.com> wrote:
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
The f/centos75 branch is only periodically rebased on master, it sounds like this is what you need. I will do a test run to see if it is clean soon... (at PTG) dt -- Dean Troyer dtroyer@gmail.com
On Tue, Sep 11, 2018 at 12:24 PM, Dean Troyer <dtroyer@gmail.com> wrote:
The f/centos75 branch is only periodically rebased on master, it sounds like this is what you need. I will do a test run to see if it is clean soon... (at PTG)
The only conflict is in stx-integ in .zuul.yaml, which I can clean up easily enough. We need to do this before merging back in to master anyway...shall I go ahead with the rebase? dt -- Dean Troyer dtroyer@gmail.com
-----Original Message----- From: Dean Troyer <dtroyer@gmail.com> Sent: September 11, 2018 2:07 PM To: McKenna, Jason <Jason.McKenna@windriver.com> Cc: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
On Tue, Sep 11, 2018 at 12:24 PM, Dean Troyer <dtroyer@gmail.com> wrote:
The f/centos75 branch is only periodically rebased on master, it sounds like this is what you need. I will do a test run to see if it is clean soon... (at PTG)
The only conflict is in stx-integ in .zuul.yaml, which I can clean up easily enough.
We need to do this before merging back in to master anyway...shall I go ahead with the rebase?
Hi Dean, It doesn't seem like the issue is a rebase. It's that code in some repos (stx-integ, for example) was updated to build against new versions of source rpms, but the .lst files currently (or at least as of when I sync'd) checked in in stx-tools call out old or different versions to be downloaded and put into the build environment. Most of the out-of-sync files can be downloaded manually, but some of them aren't accepting the checked in patches.
dt
--
Dean Troyer dtroyer@gmail.com
Hi, Jason, Did you use the manifest Shuicheng provided to get the exact commitID in each repo? I am not sure if this is going to make difference, but we were trying to avoid other code changes (in mainline) to impact the build stability. Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 2:12 AM To: Dean Troyer <dtroyer@gmail.com> Cc: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Dean Troyer <dtroyer@gmail.com> Sent: September 11, 2018 2:07 PM To: McKenna, Jason <Jason.McKenna@windriver.com> Cc: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
On Tue, Sep 11, 2018 at 12:24 PM, Dean Troyer <dtroyer@gmail.com> wrote:
The f/centos75 branch is only periodically rebased on master, it sounds like this is what you need. I will do a test run to see if it is clean soon... (at PTG)
The only conflict is in stx-integ in .zuul.yaml, which I can clean up easily enough.
We need to do this before merging back in to master anyway...shall I go ahead with the rebase?
Hi Dean, It doesn't seem like the issue is a rebase. It's that code in some repos (stx-integ, for example) was updated to build against new versions of source rpms, but the .lst files currently (or at least as of when I sync'd) checked in in stx-tools call out old or different versions to be downloaded and put into the build environment. Most of the out-of-sync files can be downloaded manually, but some of them aren't accepting the checked in patches.
dt
--
Dean Troyer dtroyer@gmail.com
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5-68.el7.... For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos-release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos-release-7-5... [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7-5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos-release-7... If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks. Best Regards Shuicheng -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7-5.1804.el7.centos.1.x86_64.rpm, and the patches checked in do not apply against either package. Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort. -J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5- 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257 Please help to do a sanity test based on this new ISO from Shuicheng on VE. @Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week. Thanks all for the support. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5- 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discu ss
Hi Jason, I have not synced my local code. So you could use the same manifest file. You just need sync the latest code for stx-integ. And update mirror with the latest lst in stx-tools. Here is the patch list you need apply: https://review.openstack.org/596855 for python-horizon https://review.openstack.org/599892 for build-iso https://review.openstack.org/602238 for update-pxe-network-installer Best Regards Shuicheng -----Original Message----- From: Xie, Cindy Sent: Thursday, September 13, 2018 2:44 PM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257 Please help to do a sanity test based on this new ISO from Shuicheng on VE. @Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week. Thanks all for the support. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5- 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discu ss
Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5- 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discu ss
Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Hi Cindy, Adding more clarification of the test cases that are failed: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted. FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know. Thanks, Nimalini -----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Nimalini, Thanks for the report. Just to ensure, so the below 2 tests were passed in mainline build, correct? Thx. - cindy -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Adding more clarification of the test cases that are failed: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted. FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know. Thanks, Nimalini -----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Yes, the failed test cases were passing on mainline build. -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Friday, September 14, 2018 8:32 PM To: Rasa, Nimalini; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Nimalini, Thanks for the report. Just to ensure, so the below 2 tests were passed in mainline build, correct? Thx. - cindy -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Adding more clarification of the test cases that are failed: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted. FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know. Thanks, Nimalini -----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"? For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ " Best Regards Shuicheng -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Adding more clarification of the test cases that are failed: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted. FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know. Thanks, Nimalini -----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M pages. For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel Thanks, Nimalini -----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"? For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ " Best Regards Shuicheng -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Adding more clarification of the test cases that are failed: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted. FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know. Thanks, Nimalini -----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
On 09/17/2018 09:21 AM, Rasa, Nimalini wrote:
Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M pages.
For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel
This could be a difference in HW that is being tested. Nimalini and Shuicheng, can you both share your test hardware? But, regardless of hardware it seems that something was not signed correctly or possibly rebuilt. Shuicheng, if you don't have the e1000e hardware, you could still try a modprobe of the kernel-module. Sau!
Thanks, Nimalini
-----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"?
For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ "
Best Regards Shuicheng
-----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy, Adding more clarification of the test cases that are failed:
FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted.
FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module
Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know.
Thanks, Nimalini
-----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
Here is the last test update, I am reporting the failed test cases only here:
From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose)
From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576]
LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0
EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848
FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state)
FAIL 20180914 07:57:34 test_kernel_module_signatures LOG
[2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output:
Thanks, Nimalini
-----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back.
The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818
Thanks. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX.
-Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi, Following testing was done for testing CentOS7.5 on a custom build created by Jason: Load: SW_VERSION="18.08" BUILD_TARGET="Unknown" BUILD_TYPE="Informal" BUILD_ID="n/a" JOB="n/a" BUILD_BY="jmckenna" BUILD_NUMBER="n/a" BUILD_HOST="yow-cgts1-lx" BUILD_DATE="2018-09-13 10:42:03 -0400" BUILD_DIR="/" WRS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root" WRS_GIT_BRANCH="HEAD" CGCS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root/stx" CGCS_GIT_BRANCH="HEAD" Sanity - 23 Test Cases Nightly Regression - 46 Test Cases (1 Failure for Kernel module Signature – tested and reproduced on three different labs) Nova Regression - Executed on three labs (AIO-SX, Standard and Storage Lab) - Labs covered Mellanox, Niantic and Fortville NICs - AIO-SX: o Test Cases Executed: 209 o Passed: 204 o Failed: 5 (3 known issues, 1 Ping issue, 1 timing issue that passed on re-execution) - Standard Lab: o Test Cases Executed: 373 o Passed: 346 o Failed: 27 (6 known issues, 1 ping issue, 20 passed on re-execution) - Storage: o Test Cases Executed: 374 o Passed: 367 o Failed: 7 (6 issues also seen in STX branch without CentOS7.5, 1 test case passed on re-execution) Failed test cases were re-executed and majority of them passed on re-execution. The remaining failed test cases are also failing on STX build without CentOS7.5 update. Hence, the failures are not caused by CentOS7.5 update. The main issue seen so far is only about Kernel Signature. Another issue seen is no Inventory tab available in Horizon. However, that does not seem to be related to CentOS update. Thanks, Numan. -----Original Message----- From: Rasa, Nimalini Sent: September-17-18 12:21 PM To: Lin, Shuicheng; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M pages. For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel Thanks, Nimalini -----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"? For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ " Best Regards Shuicheng -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Adding more clarification of the test cases that are failed: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted. FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know. Thanks, Nimalini -----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Numan, thank you, my take away from this is that the upgrade is looking pretty good. Is there a bug opened for the kernel signature issue? brucej -----Original Message----- From: Waheed, Numan [mailto:Numan.Waheed@windriver.com] Sent: Monday, September 17, 2018 1:59 PM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi, Following testing was done for testing CentOS7.5 on a custom build created by Jason: Load: SW_VERSION="18.08" BUILD_TARGET="Unknown" BUILD_TYPE="Informal" BUILD_ID="n/a" JOB="n/a" BUILD_BY="jmckenna" BUILD_NUMBER="n/a" BUILD_HOST="yow-cgts1-lx" BUILD_DATE="2018-09-13 10:42:03 -0400" BUILD_DIR="/" WRS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root" WRS_GIT_BRANCH="HEAD" CGCS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root/stx" CGCS_GIT_BRANCH="HEAD" Sanity - 23 Test Cases Nightly Regression - 46 Test Cases (1 Failure for Kernel module Signature – tested and reproduced on three different labs) Nova Regression - Executed on three labs (AIO-SX, Standard and Storage Lab) - Labs covered Mellanox, Niantic and Fortville NICs - AIO-SX: o Test Cases Executed: 209 o Passed: 204 o Failed: 5 (3 known issues, 1 Ping issue, 1 timing issue that passed on re-execution) - Standard Lab: o Test Cases Executed: 373 o Passed: 346 o Failed: 27 (6 known issues, 1 ping issue, 20 passed on re-execution) - Storage: o Test Cases Executed: 374 o Passed: 367 o Failed: 7 (6 issues also seen in STX branch without CentOS7.5, 1 test case passed on re-execution) Failed test cases were re-executed and majority of them passed on re-execution. The remaining failed test cases are also failing on STX build without CentOS7.5 update. Hence, the failures are not caused by CentOS7.5 update. The main issue seen so far is only about Kernel Signature. Another issue seen is no Inventory tab available in Horizon. However, that does not seem to be related to CentOS update. Thanks, Numan. -----Original Message----- From: Rasa, Nimalini Sent: September-17-18 12:21 PM To: Lin, Shuicheng; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M pages. For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel Thanks, Nimalini -----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"? For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ " Best Regards Shuicheng -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Adding more clarification of the test cases that are failed: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted. FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know. Thanks, Nimalini -----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
The case " test_kernel_module_signatures" seems passing on Shuicheng's system. We will provide the HW config to compare. But Numan, please open up a Launchpad ticket so that we can track there. My read for Numan's test report is also good. I think we are OK to merge back to mainline unless we hear objections? Thx. - cindy -----Original Message----- From: Jones, Bruce E Sent: Tuesday, September 18, 2018 5:55 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Numan, thank you, my take away from this is that the upgrade is looking pretty good. Is there a bug opened for the kernel signature issue? brucej -----Original Message----- From: Waheed, Numan [mailto:Numan.Waheed@windriver.com] Sent: Monday, September 17, 2018 1:59 PM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi, Following testing was done for testing CentOS7.5 on a custom build created by Jason: Load: SW_VERSION="18.08" BUILD_TARGET="Unknown" BUILD_TYPE="Informal" BUILD_ID="n/a" JOB="n/a" BUILD_BY="jmckenna" BUILD_NUMBER="n/a" BUILD_HOST="yow-cgts1-lx" BUILD_DATE="2018-09-13 10:42:03 -0400" BUILD_DIR="/" WRS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root" WRS_GIT_BRANCH="HEAD" CGCS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root/stx" CGCS_GIT_BRANCH="HEAD" Sanity - 23 Test Cases Nightly Regression - 46 Test Cases (1 Failure for Kernel module Signature – tested and reproduced on three different labs) Nova Regression - Executed on three labs (AIO-SX, Standard and Storage Lab) - Labs covered Mellanox, Niantic and Fortville NICs - AIO-SX: o Test Cases Executed: 209 o Passed: 204 o Failed: 5 (3 known issues, 1 Ping issue, 1 timing issue that passed on re-execution) - Standard Lab: o Test Cases Executed: 373 o Passed: 346 o Failed: 27 (6 known issues, 1 ping issue, 20 passed on re-execution) - Storage: o Test Cases Executed: 374 o Passed: 367 o Failed: 7 (6 issues also seen in STX branch without CentOS7.5, 1 test case passed on re-execution) Failed test cases were re-executed and majority of them passed on re-execution. The remaining failed test cases are also failing on STX build without CentOS7.5 update. Hence, the failures are not caused by CentOS7.5 update. The main issue seen so far is only about Kernel Signature. Another issue seen is no Inventory tab available in Horizon. However, that does not seem to be related to CentOS update. Thanks, Numan. -----Original Message----- From: Rasa, Nimalini Sent: September-17-18 12:21 PM To: Lin, Shuicheng; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M pages. For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel Thanks, Nimalini -----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"? For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ " Best Regards Shuicheng -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Adding more clarification of the test cases that are failed: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted. FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know. Thanks, Nimalini -----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
My read for Numan's test report is also good. I think we are OK to merge back to mainline unless we hear objections?
Suggest we discuss this at the project team call Wednesday morning but yes, it's looking good. brucej -----Original Message----- From: Xie, Cindy Sent: Monday, September 17, 2018 4:25 PM To: Jones, Bruce E <bruce.e.jones@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update The case " test_kernel_module_signatures" seems passing on Shuicheng's system. We will provide the HW config to compare. But Numan, please open up a Launchpad ticket so that we can track there. My read for Numan's test report is also good. I think we are OK to merge back to mainline unless we hear objections? Thx. - cindy -----Original Message----- From: Jones, Bruce E Sent: Tuesday, September 18, 2018 5:55 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Numan, thank you, my take away from this is that the upgrade is looking pretty good. Is there a bug opened for the kernel signature issue? brucej -----Original Message----- From: Waheed, Numan [mailto:Numan.Waheed@windriver.com] Sent: Monday, September 17, 2018 1:59 PM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi, Following testing was done for testing CentOS7.5 on a custom build created by Jason: Load: SW_VERSION="18.08" BUILD_TARGET="Unknown" BUILD_TYPE="Informal" BUILD_ID="n/a" JOB="n/a" BUILD_BY="jmckenna" BUILD_NUMBER="n/a" BUILD_HOST="yow-cgts1-lx" BUILD_DATE="2018-09-13 10:42:03 -0400" BUILD_DIR="/" WRS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root" WRS_GIT_BRANCH="HEAD" CGCS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root/stx" CGCS_GIT_BRANCH="HEAD" Sanity - 23 Test Cases Nightly Regression - 46 Test Cases (1 Failure for Kernel module Signature – tested and reproduced on three different labs) Nova Regression - Executed on three labs (AIO-SX, Standard and Storage Lab) - Labs covered Mellanox, Niantic and Fortville NICs - AIO-SX: o Test Cases Executed: 209 o Passed: 204 o Failed: 5 (3 known issues, 1 Ping issue, 1 timing issue that passed on re-execution) - Standard Lab: o Test Cases Executed: 373 o Passed: 346 o Failed: 27 (6 known issues, 1 ping issue, 20 passed on re-execution) - Storage: o Test Cases Executed: 374 o Passed: 367 o Failed: 7 (6 issues also seen in STX branch without CentOS7.5, 1 test case passed on re-execution) Failed test cases were re-executed and majority of them passed on re-execution. The remaining failed test cases are also failing on STX build without CentOS7.5 update. Hence, the failures are not caused by CentOS7.5 update. The main issue seen so far is only about Kernel Signature. Another issue seen is no Inventory tab available in Horizon. However, that does not seem to be related to CentOS update. Thanks, Numan. -----Original Message----- From: Rasa, Nimalini Sent: September-17-18 12:21 PM To: Lin, Shuicheng; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M pages. For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel Thanks, Nimalini -----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"? For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ " Best Regards Shuicheng -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Adding more clarification of the test cases that are failed: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted. FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know. Thanks, Nimalini -----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Yes, let's discuss that. I did send out the agenda including a decision required for merge back. -----Original Message----- From: Jones, Bruce E Sent: Tuesday, September 18, 2018 7:26 AM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
My read for Numan's test report is also good. I think we are OK to merge back to mainline unless we hear objections?
Suggest we discuss this at the project team call Wednesday morning but yes, it's looking good. brucej -----Original Message----- From: Xie, Cindy Sent: Monday, September 17, 2018 4:25 PM To: Jones, Bruce E <bruce.e.jones@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update The case " test_kernel_module_signatures" seems passing on Shuicheng's system. We will provide the HW config to compare. But Numan, please open up a Launchpad ticket so that we can track there. My read for Numan's test report is also good. I think we are OK to merge back to mainline unless we hear objections? Thx. - cindy -----Original Message----- From: Jones, Bruce E Sent: Tuesday, September 18, 2018 5:55 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Numan, thank you, my take away from this is that the upgrade is looking pretty good. Is there a bug opened for the kernel signature issue? brucej -----Original Message----- From: Waheed, Numan [mailto:Numan.Waheed@windriver.com] Sent: Monday, September 17, 2018 1:59 PM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi, Following testing was done for testing CentOS7.5 on a custom build created by Jason: Load: SW_VERSION="18.08" BUILD_TARGET="Unknown" BUILD_TYPE="Informal" BUILD_ID="n/a" JOB="n/a" BUILD_BY="jmckenna" BUILD_NUMBER="n/a" BUILD_HOST="yow-cgts1-lx" BUILD_DATE="2018-09-13 10:42:03 -0400" BUILD_DIR="/" WRS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root" WRS_GIT_BRANCH="HEAD" CGCS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root/stx" CGCS_GIT_BRANCH="HEAD" Sanity - 23 Test Cases Nightly Regression - 46 Test Cases (1 Failure for Kernel module Signature – tested and reproduced on three different labs) Nova Regression - Executed on three labs (AIO-SX, Standard and Storage Lab) - Labs covered Mellanox, Niantic and Fortville NICs - AIO-SX: o Test Cases Executed: 209 o Passed: 204 o Failed: 5 (3 known issues, 1 Ping issue, 1 timing issue that passed on re-execution) - Standard Lab: o Test Cases Executed: 373 o Passed: 346 o Failed: 27 (6 known issues, 1 ping issue, 20 passed on re-execution) - Storage: o Test Cases Executed: 374 o Passed: 367 o Failed: 7 (6 issues also seen in STX branch without CentOS7.5, 1 test case passed on re-execution) Failed test cases were re-executed and majority of them passed on re-execution. The remaining failed test cases are also failing on STX build without CentOS7.5 update. Hence, the failures are not caused by CentOS7.5 update. The main issue seen so far is only about Kernel Signature. Another issue seen is no Inventory tab available in Horizon. However, that does not seem to be related to CentOS update. Thanks, Numan. -----Original Message----- From: Rasa, Nimalini Sent: September-17-18 12:21 PM To: Lin, Shuicheng; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M pages. For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel Thanks, Nimalini -----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"? For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ " Best Regards Shuicheng -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Adding more clarification of the test cases that are failed: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted. FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know. Thanks, Nimalini -----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 09/17/2018 01:58 PM, Waheed, Numan wrote:
Hi,
Following testing was done for testing CentOS7.5 on a custom build created by Jason:
Load: SW_VERSION="18.08" BUILD_TARGET="Unknown" BUILD_TYPE="Informal" BUILD_ID="n/a"
JOB="n/a" BUILD_BY="jmckenna" BUILD_NUMBER="n/a" BUILD_HOST="yow-cgts1-lx" BUILD_DATE="2018-09-13 10:42:03 -0400"
BUILD_DIR="/" WRS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root" WRS_GIT_BRANCH="HEAD" CGCS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root/stx" CGCS_GIT_BRANCH="HEAD"
Sanity - 23 Test Cases
Nightly Regression - 46 Test Cases (1 Failure for Kernel module Signature – tested and reproduced on three different labs)
So, I checked with Shuicheng and he does have the e1000e signed module loading correctly, I wonder if the problem is related to your build of the kernel and signed modules. Can you confirm the signature is correct by getting the e1000e.ko to another machine and then running the kernel's extract-module-sig.pl (found in the kernel/scripts directory) extract-module-sig.pl -n e1000e.ko You should get output something like this: Read 431181 bytes from module file Found magic number at 431181 Found X.509 cert identifier Found 386 bytes of signature [01808b1c7c983aae1a50ffa4422a6a52] Found 20 bytes of key identifier [ff1d56e0f0f42602f2a4fc5cb641d1f4] Found 31 bytes of signer's name [CentOS Linux kernel signing key] CentOS Linux kernel signing key Could this be related to creating (or not creating) a new installer? Sau!
Nova Regression - Executed on three labs (AIO-SX, Standard and Storage Lab) - Labs covered Mellanox, Niantic and Fortville NICs - AIO-SX: o Test Cases Executed: 209 o Passed: 204 o Failed: 5 (3 known issues, 1 Ping issue, 1 timing issue that passed on re-execution)
- Standard Lab: o Test Cases Executed: 373 o Passed: 346 o Failed: 27 (6 known issues, 1 ping issue, 20 passed on re-execution)
- Storage: o Test Cases Executed: 374 o Passed: 367 o Failed: 7 (6 issues also seen in STX branch without CentOS7.5, 1 test case passed on re-execution)
Failed test cases were re-executed and majority of them passed on re-execution. The remaining failed test cases are also failing on STX build without CentOS7.5 update. Hence, the failures are not caused by CentOS7.5 update.
The main issue seen so far is only about Kernel Signature. Another issue seen is no Inventory tab available in Horizon. However, that does not seem to be related to CentOS update.
Thanks,
Numan.
-----Original Message----- From: Rasa, Nimalini Sent: September-17-18 12:21 PM To: Lin, Shuicheng; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M pages.
For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel
Thanks, Nimalini
-----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"?
For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ "
Best Regards Shuicheng
-----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy, Adding more clarification of the test cases that are failed:
FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted.
FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module
Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know.
Thanks, Nimalini
-----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
Here is the last test update, I am reporting the failed test cases only here:
From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose)
From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576]
LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0
EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848
FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state)
FAIL 20180914 07:57:34 test_kernel_module_signatures LOG
[2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output:
Thanks, Nimalini
-----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back.
The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818
Thanks. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX.
-Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Nimalini, For the kernel signature issue, could you help confirm whether the kernel version is 3.10.0-862.11.6 or not? You could check it with "uname -a" or search it in the kern.log file. Thanks. Best Regards Shuicheng -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 18, 2018 12:21 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M pages. For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel Thanks, Nimalini -----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"? For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ " Best Regards Shuicheng -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Adding more clarification of the test cases that are failed: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted. FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know. Thanks, Nimalini -----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Hi Shuicheng, It does have the correct version: controller-0:~$ uname -a Linux controller-0 3.10.0-862.11.6.el7.36.tis.x86_64 #1 SMP PREEMPT Thu Sep 13 10:48:38 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux Thanks, Nimalini -----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Monday, September 17, 2018 9:16 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Nimalini, For the kernel signature issue, could you help confirm whether the kernel version is 3.10.0-862.11.6 or not? You could check it with "uname -a" or search it in the kern.log file. Thanks. Best Regards Shuicheng -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 18, 2018 12:21 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M pages. For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel Thanks, Nimalini -----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"? For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ " Best Regards Shuicheng -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Adding more clarification of the test cases that are failed: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted. FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know. Thanks, Nimalini -----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Hi folks, I do not think the kernel signature item is a real issue, but an artifact of how we built the test image. The warning reported is basically saying that the driver was compiled against a different build of the kernel than is currently running. This is typical for intermediate builds where a kernel was compiled, the driver was compiled, and the kernel compiled again (which was the case when I was attempting to get the CentOS 7.5 test load built as the contents in the repos were changing). I will clean the kernel driver packages the next time I perform a build and we can verify that the error disappears. -J For anyone interested in technical details, the kernel regenerates a key pair every time it is built, and compiles one half into the kernel and puts the other half in the kernel-devel package. Packages (like kernel drivers) building against the kernel-devel package will sign themselves with the key from the package. Upon loading a driver, the kernel will validate that the signature on the driver matches the half of the key which the kernel has access to. A mismatched key indicates that the kernel running and kernel-devel package which the driver was built against came from 2 different builds of the kernel, and marks the kernel tainted as a warning.
-----Original Message----- From: Rasa, Nimalini Sent: September 18, 2018 9:50 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng, It does have the correct version: controller-0:~$ uname -a Linux controller-0 3.10.0-862.11.6.el7.36.tis.x86_64 #1 SMP PREEMPT Thu Sep 13 10:48:38 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Thanks, Nimalini
-----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Monday, September 17, 2018 9:16 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Nimalini, For the kernel signature issue, could you help confirm whether the kernel version is 3.10.0-862.11.6 or not? You could check it with "uname -a" or search it in the kern.log file. Thanks.
Best Regards Shuicheng
-----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 18, 2018 12:21 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M pages.
For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel
Thanks, Nimalini
-----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"?
For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ "
Best Regards Shuicheng
-----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy, Adding more clarification of the test cases that are failed:
FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted.
FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module
Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know.
Thanks, Nimalini
Jason, So if I am understand this correct the test_kernel_module_signatures failure listed below is a not an issue due to the way WR built the kernel and modules? Why was this build differently by WR than you would normally build an image for testing? Wouldn't you always have this signature problem then? I am just trying to understand what differences in the way WR builds and the way that is documented, just as we had the issue with the installer setup. Nimalini: Given this information, do we have any blockers for the 7.5 upgrade? Thanks Sau! On 09/18/2018 07:27 AM, McKenna, Jason wrote:
Hi folks,
I do not think the kernel signature item is a real issue, but an artifact of how we built the test image.
The warning reported is basically saying that the driver was compiled against a different build of the kernel than is currently running. This is typical for intermediate builds where a kernel was compiled, the driver was compiled, and the kernel compiled again (which was the case when I was attempting to get the CentOS 7.5 test load built as the contents in the repos were changing). I will clean the kernel driver packages the next time I perform a build and we can verify that the error disappears.
-J
For anyone interested in technical details, the kernel regenerates a key pair every time it is built, and compiles one half into the kernel and puts the other half in the kernel-devel package. Packages (like kernel drivers) building against the kernel-devel package will sign themselves with the key from the package. Upon loading a driver, the kernel will validate that the signature on the driver matches the half of the key which the kernel has access to. A mismatched key indicates that the kernel running and kernel-devel package which the driver was built against came from 2 different builds of the kernel, and marks the kernel tainted as a warning.
-----Original Message----- From: Rasa, Nimalini Sent: September 18, 2018 9:50 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng, It does have the correct version: controller-0:~$ uname -a Linux controller-0 3.10.0-862.11.6.el7.36.tis.x86_64 #1 SMP PREEMPT Thu Sep 13 10:48:38 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Thanks, Nimalini
-----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Monday, September 17, 2018 9:16 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Nimalini, For the kernel signature issue, could you help confirm whether the kernel version is 3.10.0-862.11.6 or not? You could check it with "uname -a" or search it in the kern.log file. Thanks.
Best Regards Shuicheng
-----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 18, 2018 12:21 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M pages.
For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel
Thanks, Nimalini
-----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"?
For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ "
Best Regards Shuicheng
-----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy, Adding more clarification of the test cases that are failed:
FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted.
FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module
Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know.
Thanks, Nimalini
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
We don't have any blockers now. -----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Tuesday, September 18, 2018 12:08 PM To: McKenna, Jason; Rasa, Nimalini; Lin, Shuicheng; Xie, Cindy; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update Jason, So if I am understand this correct the test_kernel_module_signatures failure listed below is a not an issue due to the way WR built the kernel and modules? Why was this build differently by WR than you would normally build an image for testing? Wouldn't you always have this signature problem then? I am just trying to understand what differences in the way WR builds and the way that is documented, just as we had the issue with the installer setup. Nimalini: Given this information, do we have any blockers for the 7.5 upgrade? Thanks Sau! On 09/18/2018 07:27 AM, McKenna, Jason wrote:
Hi folks,
I do not think the kernel signature item is a real issue, but an artifact of how we built the test image.
The warning reported is basically saying that the driver was compiled against a different build of the kernel than is currently running. This is typical for intermediate builds where a kernel was compiled, the driver was compiled, and the kernel compiled again (which was the case when I was attempting to get the CentOS 7.5 test load built as the contents in the repos were changing). I will clean the kernel driver packages the next time I perform a build and we can verify that the error disappears.
-J
For anyone interested in technical details, the kernel regenerates a key pair every time it is built, and compiles one half into the kernel and puts the other half in the kernel-devel package. Packages (like kernel drivers) building against the kernel-devel package will sign themselves with the key from the package. Upon loading a driver, the kernel will validate that the signature on the driver matches the half of the key which the kernel has access to. A mismatched key indicates that the kernel running and kernel-devel package which the driver was built against came from 2 different builds of the kernel, and marks the kernel tainted as a warning.
-----Original Message----- From: Rasa, Nimalini Sent: September 18, 2018 9:50 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng, It does have the correct version: controller-0:~$ uname -a Linux controller-0 3.10.0-862.11.6.el7.36.tis.x86_64 #1 SMP PREEMPT Thu Sep 13 10:48:38 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Thanks, Nimalini
-----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Monday, September 17, 2018 9:16 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Nimalini, For the kernel signature issue, could you help confirm whether the kernel version is 3.10.0-862.11.6 or not? You could check it with "uname -a" or search it in the kern.log file. Thanks.
Best Regards Shuicheng
-----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 18, 2018 12:21 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M pages.
For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel
Thanks, Nimalini
-----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"?
For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ "
Best Regards Shuicheng
-----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy, Adding more clarification of the test cases that are failed:
FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted.
FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module
Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know.
Thanks, Nimalini
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Saul, please see inline
-----Original Message----- From: Rasa, Nimalini Sent: September 18, 2018 12:59 PM To: Saul Wold <sgw@linux.intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
We don't have any blockers now.
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Tuesday, September 18, 2018 12:08 PM To: McKenna, Jason; Rasa, Nimalini; Lin, Shuicheng; Xie, Cindy; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Jason,
So if I am understand this correct the test_kernel_module_signatures failure listed below is a not an issue due to the way WR built the kernel and modules?
Why was this build differently by WR than you would normally build an image for testing? Wouldn't you always have this signature problem then?
In this instance, after I had done a build, I was examining the commits and there was a head-scratcher in one of the build tools. I manually issued a rebuild of the kernel package to better observe the flow of the tool, which regenerated the signing keys in the kernel package. The tainted kernel notice would not normally appear.
I am just trying to understand what differences in the way WR builds and the way that is documented, just as we had the issue with the installer setup.
That's touching on a bigger issue, I think. As you are aware there was no "documented" way to build the CentOS 7.5 branch. The email thread from last week discusses this in more detail, but involved the sending of manifest files through email, specifying some commits to cherry-pick from master into a build, using the revision of some gits specified in the manifest, but overriding the manifest and using the tip of the f/centos75 branch for other gits, etc. Because the build was hand managed, it opened the door to deviations like manually recompiling a package. So that was the immediate cause. I understand that for policy reasons binary artifacts cannot be exchanged, but we should come up with a general purpose better way to align our builds to reduce the chance of this kind of thing happening.
Nimalini: Given this information, do we have any blockers for the 7.5 upgrade?
Thanks Sau!
Hi folks,
I do not think the kernel signature item is a real issue, but an artifact of how we built the test image.
The warning reported is basically saying that the driver was compiled against a different build of the kernel than is currently running. This is typical for intermediate builds where a kernel was compiled, the driver was compiled, and the kernel compiled again (which was the case when I was attempting to get the CentOS 7.5 test load built as the contents in the repos were changing). I will clean the kernel driver packages the next time I
On 09/18/2018 07:27 AM, McKenna, Jason wrote: perform a build and we can verify that the error disappears.
-J
For anyone interested in technical details, the kernel regenerates a key pair
every time it is built, and compiles one half into the kernel and puts the other half in the kernel-devel package. Packages (like kernel drivers) building against the kernel-devel package will sign themselves with the key from the package. Upon loading a driver, the kernel will validate that the signature on the driver matches the half of the key which the kernel has access to. A mismatched key indicates that the kernel running and kernel-devel package which the driver was built against came from 2 different builds of the kernel, and marks the kernel tainted as a warning.
-----Original Message----- From: Rasa, Nimalini Sent: September 18, 2018 9:50 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason
Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng, It does have the correct version: controller-0:~$ uname -a Linux controller-0 3.10.0-862.11.6.el7.36.tis.x86_64 #1 SMP PREEMPT Thu Sep 13 10:48:38 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Thanks, Nimalini
-----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Monday, September 17, 2018 9:16 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Nimalini, For the kernel signature issue, could you help confirm whether the kernel version is 3.10.0-862.11.6 or not? You could check it with "uname -a" or search it in the kern.log file. Thanks.
Best Regards Shuicheng
-----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 18, 2018 12:21 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng, In this test case, basically trying to allocate double the amount of 2M
<Jason.McKenna@windriver.com>; pages.
For the kernel signature, I tried it on 3 different hardware labs and got the following: controller-0:~$ cat /proc/sys/kernel/tainted 12288 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel
Thanks, Nimalini
-----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"?
For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ "
Best Regards Shuicheng
-----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy, Adding more clarification of the test cases that are failed:
FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted.
FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module
Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know.
Thanks, Nimalini
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
See inline KWY+> On 2018-09-18, 3:19 PM, "McKenna, Jason" <Jason.McKenna@windriver.com> wrote: Hi Saul, please see inline > -----Original Message----- > From: Rasa, Nimalini > Sent: September 18, 2018 12:59 PM > To: Saul Wold <sgw@linux.intel.com>; McKenna, Jason > <Jason.McKenna@windriver.com>; Lin, Shuicheng > <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Martinez > Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan > <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io > Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update > > We don't have any blockers now. > > -----Original Message----- > From: Saul Wold [mailto:sgw@linux.intel.com] > Sent: Tuesday, September 18, 2018 12:08 PM > To: McKenna, Jason; Rasa, Nimalini; Lin, Shuicheng; Xie, Cindy; Martinez > Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io > Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update > > > Jason, > > So if I am understand this correct the test_kernel_module_signatures failure > listed below is a not an issue due to the way WR built the kernel and > modules? > > Why was this build differently by WR than you would normally build an image > for testing? Wouldn't you always have this signature problem then? In this instance, after I had done a build, I was examining the commits and there was a head-scratcher in one of the build tools. I manually issued a rebuild of the kernel package to better observe the flow of the tool, which regenerated the signing keys in the kernel package. The tainted kernel notice would not normally appear. > > I am just trying to understand what differences in the way WR builds and the > way that is documented, just as we had the issue with the installer setup. That's touching on a bigger issue, I think. As you are aware there was no "documented" way to build the CentOS 7.5 branch. The email thread from last week discusses this in more detail, but involved the sending of manifest files through email, specifying some commits to cherry-pick from master into a build, using the revision of some gits specified in the manifest, but overriding the manifest and using the tip of the f/centos75 branch for other gits, etc. Because the build was hand managed, it opened the door to deviations like manually recompiling a package. So that was the immediate cause. I understand that for policy reasons binary artifacts cannot be exchanged, but we should come up with a general purpose better way to align our builds to reduce the chance of this kind of thing happening. KWY+> Let's get back to first principles. For the build to be reproducible, all the code needs to be in the branch. Otherwise, it is a designer build which is not acceptable for formal branch testing. > > Nimalini: Given this information, do we have any blockers for the 7.5 > upgrade? > > Thanks > Sau! > > > On 09/18/2018 07:27 AM, McKenna, Jason wrote: > > Hi folks, > > > > I do not think the kernel signature item is a real issue, but an artifact of how > we built the test image. > > > > The warning reported is basically saying that the driver was compiled > against a different build of the kernel than is currently running. This is typical > for intermediate builds where a kernel was compiled, the driver was > compiled, and the kernel compiled again (which was the case when I was > attempting to get the CentOS 7.5 test load built as the contents in the repos > were changing). I will clean the kernel driver packages the next time I > perform a build and we can verify that the error disappears. > > > > -J > > > > For anyone interested in technical details, the kernel regenerates a key pair > every time it is built, and compiles one half into the kernel and puts the other > half in the kernel-devel package. Packages (like kernel drivers) building > against the kernel-devel package will sign themselves with the key from the > package. Upon loading a driver, the kernel will validate that the signature on > the driver matches the half of the key which the kernel has access to. A > mismatched key indicates that the kernel running and kernel-devel package > which the driver was built against came from 2 different builds of the kernel, > and marks the kernel tainted as a warning. > > > >> -----Original Message----- > >> From: Rasa, Nimalini > >> Sent: September 18, 2018 9:50 AM > >> To: Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy > >> <cindy.xie@intel.com>; McKenna, Jason > <Jason.McKenna@windriver.com>; > >> Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, > Numan > >> <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io > >> Cc: Khalil, Ghada <Ghada.Khalil@windriver.com> > >> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update > >> > >> Hi Shuicheng, > >> It does have the correct version: > >> controller-0:~$ uname -a > >> Linux controller-0 3.10.0-862.11.6.el7.36.tis.x86_64 #1 SMP PREEMPT > >> Thu Sep > >> 13 10:48:38 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux > >> > >> Thanks, > >> Nimalini > >> > >> -----Original Message----- > >> From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] > >> Sent: Monday, September 17, 2018 9:16 PM > >> To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, > >> Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io > >> Cc: Khalil, Ghada > >> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update > >> > >> Hi Nimalini, > >> For the kernel signature issue, could you help confirm whether the > >> kernel version is 3.10.0-862.11.6 or not? > >> You could check it with "uname -a" or search it in the kern.log file. > >> Thanks. > >> > >> Best Regards > >> Shuicheng > >> > >> > >> -----Original Message----- > >> From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] > >> Sent: Tuesday, September 18, 2018 12:21 AM > >> To: Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy > >> <cindy.xie@intel.com>; McKenna, Jason > <Jason.McKenna@windriver.com>; > >> Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, > Numan > >> <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io > >> Cc: Khalil, Ghada <Ghada.Khalil@windriver.com> > >> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update > >> > >> Hi Shuicheng, > >> In this test case, basically trying to allocate double the amount of 2M > pages. > >> > >> For the kernel signature, I tried it on 3 different hardware labs and > >> got the > >> following: > >> controller-0:~$ cat /proc/sys/kernel/tainted > >> 12288 > >> controller-0:~$ grep --color=never -i "module verification failed" > >> /var/log/kern.log > >> 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: > >> module verification failed: signature and/or required key missing - > >> tainting kernel > >> 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: > >> module verification failed: signature and/or required key missing - > >> tainting kernel > >> > >> > >> Thanks, > >> Nimalini > >> > >> -----Original Message----- > >> From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] > >> Sent: Friday, September 14, 2018 9:13 PM > >> To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, > >> Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io > >> Cc: Khalil, Ghada > >> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update > >> > >> Hi Nimalini, > >> Could you share me the detail reproduce step for " > >> test_schedule_vm_mempage_config"? > >> > >> For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in > >> my virtual and bare metal environment, and both of them show 4096. > >> Then what should I do next? > >> " > >> controller-0:~$ cat /proc/sys/kernel/tainted > >> 4096 > >> controller-0:~$ grep --color=never -i "module verification failed" > >> /var/log/kern.log controller-0:~$ " > >> > >> Best Regards > >> Shuicheng > >> > >> -----Original Message----- > >> From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] > >> Sent: Saturday, September 15, 2018 3:02 AM > >> To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason > >> <Jason.McKenna@windriver.com>; Martinez Monroy, Elio > >> <elio.martinez.monroy@intel.com>; Waheed, Numan > >> <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io > >> Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng > >> <shuicheng.lin@intel.com> > >> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update > >> > >> Hi Cindy, > >> Adding more clarification of the test cases that are failed: > >> > >> FAIL 20180914 06:54:06 > >> test_schedule_vm_mempage_config[1048576] > >> Test case is trying to over allocate 2M pages, expected to reject > >> the cli, in this case it's accepted. > >> > >> FAIL 20180914 07:57:34 test_kernel_module_signatures > >> Test case is doing the following: > >> - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If > >> not, do following > >> steps: > >> - 'grep --color=never -i "module verification failed" > >> /var/log/kern.log' to find out failed modules > >> - 'modinfo <failed_module> | grep --color=never -E > >> "sig|filename" to display signing info for each module > >> > >> > >> Hope it clarifies bit more as to what the test cases it trying to do. > >> If you have any questions please let me know. > >> > >> Thanks, > >> Nimalini > > _______________________________________________ > > Starlingx-discuss mailing list > > Starlingx-discuss@lists.starlingx.io > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss > >
Hi Shuicheng, The "test_schedule_vm_mempage_config" passed on re-run, it seems like a timing issue where the VM memory has not been released or has not been updated by sysinv-agent audit after the host is locked...fyi Thanks, Nimalini -----Original Message----- From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 14, 2018 9:13 PM To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Nimalini, Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"? For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next? " controller-0:~$ cat /proc/sys/kernel/tainted 4096 controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ " Best Regards Shuicheng -----Original Message----- From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Saturday, September 15, 2018 3:02 AM To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Adding more clarification of the test cases that are failed: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted. FAIL 20180914 07:57:34 test_kernel_module_signatures Test case is doing the following: - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps: - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know. Thanks, Nimalini -----Original Message----- From: Rasa, Nimalini Sent: Friday, September 14, 2018 12:46 PM To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Here is the last test update, I am reporting the failed test cases only here: From Sanity: FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose) From Nightly: FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576] LOG: ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1' [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output: +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True | | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True | +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+ controller-1:~$ [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-1:~$ [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420'] [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0] [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0'] [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1] [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None'] [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4 [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0' [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output: +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | Memory: Usable Total (MiB) | 7540 | | Platform (MiB) | 8000 | | Available (MiB) | 2420 | | Huge Pages Configured | True | | vSwitch Huge Pages: Size (MiB) | 1024 | | Total | 1 | | Available | 0 | | VM Pages (4K): Total | 20992 | | VM Huge Pages (2M): Total | 3258 | | Total Pending | 2420 | | Available | 1210 | | VM Huge Pages (1G): Total | 0 | | Available | 0 | | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 | | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea | | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 | | created_at | 2018-09-14T02:30:56.591922+00:00 | | updated_at | 2018-09-14T06:48:29.882522+00:00 | +--------------------------------+--------------------------------------+ controller-1:~$ [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output: 0 EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD: [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0' [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output: Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848 FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small] This test failed b/c of the above test failure (Alarm was present, compute was in locked state) FAIL 20180914 07:57:34 test_kernel_module_signatures LOG [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 1: Check for unassigned kernel modules on controller-0 [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted' [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output: 12288 controller-0:~$ [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 2: Check kern.log for modules with failed verification [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log' [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output: 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel controller-0:~$ [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output: 0 controller-0:~$ [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures:: ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e'] [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command... [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"' [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output: filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko signer: CentOS Linux kernel signing key sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC sig_hashalgo: sha256 controller-0:~$ [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?' [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output: Thanks, Nimalini -----Original Message----- From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Thursday, September 13, 2018 10:05 AM To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back. The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818 Thanks. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 9:35 PM To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Hi Cindy, Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX. -Jason
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: September 13, 2018 2:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week.
Thanks all for the support. - cindy
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5 - 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss
Will try to have it today! Thanks -----Original Message----- From: Xie, Cindy Sent: Thursday, September 13, 2018 1:44 AM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update Elio, We will provide you an updated ISO later today with the following: - kernel upgrade to 3.10.0-862.11.6 - symbol issue fixed: https://bugs.launchpad.net/starlingx/+bug/1792257 Please help to do a sanity test based on this new ISO from Shuicheng on VE. @Jason, Please re-generate a build using today's manifest with above fix. Shuicheng shall be able to provide manifest with exact commitID. @Numan, please abort your current testing w/ old ISO once you get a new one from Jason and continue your sanity and regression on bare-metal. I am not sure how long it takes, but please take priority as we want merge back to mainline early next week. Thanks all for the support. - cindy -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Thursday, September 13, 2018 4:24 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
-----Original Message----- From: Lin, Shuicheng <shuicheng.lin@intel.com> Sent: September 11, 2018 8:44 PM To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Here is the dhcp src rpm we are using: [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source dhcp http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5- 68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src rpm for build, and 1 rpm for the mock environment. And it has the "slightly difference" in the names as you mentioned. Both of them could be found in the centos repo with the exact name. ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source centos- release-7-5.1804.1.el7.centos http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos- release-7-5.1804.1.el7.centos.src.rpm [slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url centos-release-7- 5.1804.el7.centos.x86_64 http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos- release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them from the repo. I have try my best to keep my local mirror the same as the lst. But it is still possible some package missed to be added to the repo lst. Please feel free to notify me or correct the error directly. Thanks.
With the latest commits to stx-tools I am now able to download the rpms and produce a build. I'll let you know how testing goes
Best Regards Shuicheng
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Wednesday, September 12, 2018 12:21 AM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) I am unable to produce a build. There seems to be several source rpm files missing from the currently pushed version of the .lst files (or the .lst files pull down the wrong version). Most of the missing files I can download manually, however the checked in patches for dhcp and centos-release packages do not apply against the version of the source RPM which I can manually find on the Internet. For example, the version of centos-release called out in the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the version called out in the checked in code base is the slightly different centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches checked in do not apply against either package.
Until the centos75 branch of the lst files pulls down the correct rpms, I will be unable to continue the validation effort.
-J
-----Original Message----- From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com> Sent: September 11, 2018 11:17 AM To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
Shuicheng saw the symbol table error issue as well but it recovers in 8s, he is still working on debugging it. But 601204 is just kernel upgrade but not intended to solve this issues. Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 9:50 PM To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini <Nimalini.Rasa@w indriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discus s@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up against the symbol table error. Our system hangs after this message. I suspect a kernel issue. I see there is an open code review to refactor some kernel patches to a rebased kernel (https://review.open stack.org/#/c/601204/) but I do not know if that is expected to resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com> Sent: September 11, 2018 9:32 AM To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan <Numa n.Waheed@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs .liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed with error messages:
Installation failed. ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in Intel’s HW. Thus Shuicheng created a new build and attached a manifest in this email thread. Did WindRiver has a chance to create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:25 PM To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@win driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> ; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com] Sent: Tuesday, September 11, 2018 9:22 AM To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io' Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com] Sent: Tuesday, September 11, 2018 9:06 PM To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@w indriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-disc uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com> Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the following , it just hangs there: “Error: no symbol table. Press any key to continue…”
-Nimalini
From: Waheed, Numan Sent: Tuesday, September 11, 2018 8:50 AM To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; Rasa, Nimalini Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: September-11-18 2:17 AM To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, Numan Subject: RE: CentOS 7.5 upgrade status update
Hi Jason, Just finished deploy image with latest code, I could config controller successfully in bare metal environment. Manifest is created for you reference. There are two patches I add manually: https://review.openstack.org/596855 for python-horizon from master https://review.openstack.org/599892 for build-iso
There is an issue found when reboot controller node. After the OS selection menu, it will show below error. And it will auto continue boot after around 8 seconds. I will check the issue later. “ Error: no symbol table. Press any key to continue… “
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, September 11, 2018 9:35 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS <zhip engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'sta rlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx .io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@w indriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I just tried my last Friday’s image at bare metal machine. There is no issue to install controller node. Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:13 AM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng .lin@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 'starlin gx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> ; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windr iver.com> Subject: RE: CentOS 7.5 upgrade status update
Status update. The build from the centos75 branch does not install on our hardware. The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: September 8, 2018 9:56 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.M cKenna@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
https://review.openstack.org/#/c/600920/
Thanks! Zhipeng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: 2018年9月8日 21:14 To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, I have fixed the error in the openstack-aodh. Patch is updated: https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one. Sorry for the trouble caused, and thanks for your test.
Best Regards Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 8, 2018 9:23 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <star lingx-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason, Thanks very much for such detail feedback. I added some comments inline, and will check the issue in my environment also. Feel free to submit patch if you have to fix them. J
Best Regards Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Saturday, September 8, 2018 3:47 AM To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada <Ghad a.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlin gx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed @windriver.com> Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code. We have not yet run run-time tests, but here are the build-time issues I encountered:
- The .lst file contains lighttpd-1.4.49, but this is no longer available for download, and the download_mirrors.sh step fails. It appears than CentOS has updated to lighttpd-1.4.50. I manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst - Even though the .lst file specifies lighttpd-1.4.49, the code still references 1.4.45 - At build-pkgs time (the build-srpms step), the meta-patches to openstack-aodh do not apply, as two of them are malformed. https: //review.openstack.org/#/c/598388/6/openstack/openstack- aodh/centos/meta_patches/0001-Modify-service-files-and-create- expirer-cron-script.patch -- In the new version, on line 190, the patch states that the old code covered 6 lines, and the new code covers 9 lines (-283,6 +286,9). But there are only two lines inserted by the patch. There is also a new blank line in the patch at line 194. Removing this blank line and editing the hunk description to only insert 2 lines resolves the issue for this hunk (there are other similar issues in this patch, and in 0001-meta- modify-aodh-api.patch) [slin14]: Let me double check it. - At binary build time, I had build failures on python-django- horizon and some magnum packages. I had to pull in https://review.op enstack.org/#/c/596855/2 (from the master branch, but not in the centos75 branch of stx-upstream) to get past this. [slin14]: It is due to I have not synced latest master code, so I don’t have this issue. I used last week’s master code. - After building all packages, I attempted to build the ISO. The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and should be added to the .lst files. [slin14]: This package is already in the lst. - Furthermore, the tpm2-tools package was requesting version tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7). I hacked the spec file for tpm2-tools and rebuilt it to require the version mentioned, but there is some rework to be done there. [slin14]: I have the same issue also. Let’s further check it to find the root cause. I use rebuild tpm2-tools/tss2 to workaround it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com> Sent: September 7, 2018 10:08 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starling x-discuss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has been changed from 1.4.45 to 1.4.49. The lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or CentOS servers, as they’ve moved to 1.4.50. I am attempting to locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: September 7, 2018 9:36 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; 'starlingx-discuss@list s.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c indy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will be priming this activity. He’ll share the results with the team sometime next week.
Regards, Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Friday, September 07, 2018 1:32 AM To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, Cindy Subject: CentOS 7.5 upgrade status update
Hi all, Most task of CentOS7.5 upgrade in story 2003389 is done. Please go ahead to have a try and verify it. I expect there should be some issue due to we just did limited deploy test yet.
Here is the detail status update of this week. For the total 44 srpm we plan to upgrade: 38 already merged in feature branch. 3 kept as original version. Anaconda upgrade is not needed. python- os-vif has higher version than 7.5. python-eventlet kept due to new src rpm itself have known issue. 2 under review with code ready. Puppet-haproxy and openstack-aodh. 1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction: 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx- upstream/stx-root. Cherry-pick below un-merged patch: For puppet-haproxy: https://review.openstack.org/600369 https://review.openstack.org/600359 For openstack-aodh: https://review.openstack.org/598388 https://review.openstack.org/600591 For build-iso: https://review.openstack.org/599892 For build-pkgs: https://review.openstack.org/599901 https://review.openstack.org/600639
2. Run mirror script to get the upgraded src rpm and related rpm. 3. Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story: https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards Shuicheng _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discu ss
participants (13)
-
Cordoba Malibran, Erich
-
Dean Troyer
-
Jones, Bruce E
-
Khalil, Ghada
-
Lin, Shuicheng
-
Liu, ZhipengS
-
Martinez Monroy, Elio
-
McKenna, Jason
-
Rasa, Nimalini
-
Saul Wold
-
Waheed, Numan
-
Xie, Cindy
-
Young, Ken