[Starlingx-discuss] CentOS 7.5 upgrade status update

Rasa, Nimalini Nimalini.Rasa at windriver.com
Fri Sep 14 16:46:04 UTC 2018


Hi Cindy,

Here is the last test update, I am reporting the failed test cases only  here:

From Sanity:
FAIL       20180914 04:32:46            test_evacuate_vms  --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose)

From Nightly:
FAIL       20180914 06:54:06            test_schedule_vm_mempage_config[1048576]
LOG:
====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08>
[2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4
[2018-09-14 06:55:20,035] 419  DEBUG MainThread ssh.exec_cmd:: Executing command...
[2018-09-14 06:55:20,036] 262  DEBUG MainThread ssh.send    :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1'
[2018-09-14 06:55:21,883] 382  DEBUG MainThread ssh.expect  :: Output: 
+-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+
| processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G |
+-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+
| 0         | 7540           | 8000              | 2420           | True                     | 1024            | 1           | 0           | 20992       | 3258           | 1210           | None             | 0              | 0              | None             | True         |
| 1         | 13770          | 2000              | 12746          | True                     | 1024            | 1           | 0           | 0           | 6373           | 6373           | None             | 0              | 0              | None             | True         |
+-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+
controller-1:~$ 
[2018-09-14 06:55:21,884] 262  DEBUG MainThread ssh.send    :: Send 'echo $?'
[2018-09-14 06:55:21,986] 382  DEBUG MainThread ssh.expect  :: Output: 
0
controller-1:~$ 
[2018-09-14 06:55:21,987] 366  DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0]
[2018-09-14 06:55:21,987] 498  DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420']
[2018-09-14 06:55:21,987] 366  DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0]
[2018-09-14 06:55:21,988] 498  DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0']
[2018-09-14 06:55:21,988] 366  DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0]
[2018-09-14 06:55:21,988] 498  DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None']
[2018-09-14 06:55:21,988] 366  DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1]
[2018-09-14 06:55:21,989] 498  DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746']
[2018-09-14 06:55:21,989] 366  DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1]
[2018-09-14 06:55:21,989] 498  DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0']
[2018-09-14 06:55:21,989] 366  DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1]
[2018-09-14 06:55:21,990] 498  DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None']
[2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4
[2018-09-14 06:55:21,990] 419  DEBUG MainThread ssh.exec_cmd:: Executing command...
[2018-09-14 06:55:21,990] 262  DEBUG MainThread ssh.send    :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0'
[2018-09-14 06:55:24,362] 382  DEBUG MainThread ssh.expect  :: Output: 
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| Memory: Usable Total (MiB)     | 7540                                 |
|         Platform     (MiB)     | 8000                                 |
|         Available    (MiB)     | 2420                                 |
| Huge Pages Configured          | True                                 |
| vSwitch Huge Pages: Size (MiB) | 1024                                 |
|                     Total      | 1                                    |
|                     Available  | 0                                    |
| VM  Pages (4K): Total          | 20992                                |
| VM  Huge Pages (2M): Total     | 3258                                 |
|                 Total Pending  | 2420                                 |
|                 Available      | 1210                                 |
| VM  Huge Pages (1G): Total     | 0                                    |
|                 Available      | 0                                    |
| uuid                           | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 |
| ihost_uuid                     | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea |
| inode_uuid                     | 96430065-9a83-4e33-ab78-c7de77cf8bc9 |
| created_at                     | 2018-09-14T02:30:56.591922+00:00     |
| updated_at                     | 2018-09-14T06:48:29.882522+00:00     |
+--------------------------------+--------------------------------------+
controller-1:~$ 
[2018-09-14 06:55:24,362] 262  DEBUG MainThread ssh.send    :: Send 'echo $?'
[2018-09-14 06:55:24,465] 382  DEBUG MainThread ssh.expect  :: Output: 
0

EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD:
[2018-09-14 12:37:33,741] 262  DEBUG MainThread ssh.send    :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0'
[2018-09-14 12:37:36,023] 382  DEBUG MainThread ssh.expect  :: Output: 
Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848


FAIL       20180914 07:00:37            test_schedule_vm_mempage_config[small]
	This test failed b/c of the above test failure (Alarm was present, compute was in locked state)


FAIL       20180914 07:57:34            test_kernel_module_signatures
LOG

[2018-09-14 07:57:37,946] 24   INFO  MainThread test_kernel_modules.test_kernel_module_signatures:: 
====================== Test Step 1: Check for unassigned kernel modules on controller-0
[2018-09-14 07:57:37,947] 419  DEBUG MainThread ssh.exec_cmd:: Executing command...
[2018-09-14 07:57:37,947] 262  DEBUG MainThread ssh.send    :: Send 'cat /proc/sys/kernel/tainted'
[2018-09-14 07:57:38,050] 382  DEBUG MainThread ssh.expect  :: Output: 
12288
controller-0:~$ 
[2018-09-14 07:57:38,050] 262  DEBUG MainThread ssh.send    :: Send 'echo $?'
[2018-09-14 07:57:38,152] 382  DEBUG MainThread ssh.expect  :: Output: 
0
controller-0:~$ 
[2018-09-14 07:57:38,153] 32   ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info
[2018-09-14 07:57:38,153] 34   INFO  MainThread test_kernel_modules.test_kernel_module_signatures:: 
====================== Test Step 2: Check kern.log for modules with failed verification
[2018-09-14 07:57:38,153] 419  DEBUG MainThread ssh.exec_cmd:: Executing command...
[2018-09-14 07:57:38,153] 262  DEBUG MainThread ssh.send    :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log'
[2018-09-14 07:57:38,265] 382  DEBUG MainThread ssh.expect  :: Output: 
2018-09-13T19:09:51.212 localhost kernel: notice [    6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel
2018-09-13T19:51:42.174 controller-0 kernel: notice [    6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel
2018-09-14T03:33:29.054 controller-0 kernel: notice [    6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel
controller-0:~$ 
[2018-09-14 07:57:38,265] 262  DEBUG MainThread ssh.send    :: Send 'echo $?'
[2018-09-14 07:57:38,367] 382  DEBUG MainThread ssh.expect  :: Output: 
0
controller-0:~$ 
[2018-09-14 07:57:38,368] 43   INFO  MainThread test_kernel_modules.test_kernel_module_signatures:: 
====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e']
[2018-09-14 07:57:38,368] 419  DEBUG MainThread ssh.exec_cmd:: Executing command...
[2018-09-14 07:57:38,368] 262  DEBUG MainThread ssh.send    :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"'
[2018-09-14 07:57:38,474] 382  DEBUG MainThread ssh.expect  :: Output: 
filename:       /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko
signer:         CentOS Linux kernel signing key
sig_key:        58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC
sig_hashalgo:   sha256
controller-0:~$ 
[2018-09-14 07:57:38,475] 262  DEBUG MainThread ssh.send    :: Send 'echo $?'
[2018-09-14 07:57:38,577] 382  DEBUG MainThread ssh.expect  :: Output:



Thanks,
Nimalini

-----Original Message-----
From: Xie, Cindy [mailto:cindy.xie at intel.com] 
Sent: Thursday, September 13, 2018 10:05 AM
To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss at lists.starlingx.io
Cc: Khalil, Ghada; Lin, Shuicheng
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update

Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back.

The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818

Thanks. - cindy

-----Original Message-----
From: McKenna, Jason [mailto:Jason.McKenna at windriver.com]
Sent: Thursday, September 13, 2018 9:35 PM
To: Xie, Cindy <cindy.xie at intel.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>; Waheed, Numan <Numan.Waheed at windriver.com>; Rasa, Nimalini <Nimalini.Rasa at windriver.com>; starlingx-discuss at lists.starlingx.io
Cc: Khalil, Ghada <Ghada.Khalil at windriver.com>; Lin, Shuicheng <shuicheng.lin at intel.com>
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update

Hi Cindy,

Will do.  In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite.  The results (attached) align with the 7.4 based master branch of StarlingX.

-Jason

> -----Original Message-----
> From: Xie, Cindy <cindy.xie at intel.com>
> Sent: September 13, 2018 2:44 AM
> To: Martinez Monroy, Elio <elio.martinez.monroy at intel.com>; McKenna, 
> Jason <Jason.McKenna at windriver.com>; Waheed, Numan 
> <Numan.Waheed at windriver.com>; Rasa, Nimalini 
> <Nimalini.Rasa at windriver.com>; starlingx-discuss at lists.starlingx.io
> Cc: Khalil, Ghada <Ghada.Khalil at windriver.com>; Lin, Shuicheng 
> <shuicheng.lin at intel.com>
> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
> 
> Elio,
> We will provide you an updated ISO later today with the following:
> - kernel upgrade to 3.10.0-862.11.6
> - symbol issue fixed: 
> https://bugs.launchpad.net/starlingx/+bug/1792257
> 
> Please help to do a sanity test based on this new ISO from Shuicheng on VE.
> 
> @Jason,
> Please re-generate a build using today's manifest with above fix. 
> Shuicheng shall be able to provide manifest with exact commitID. 
> @Numan, please abort your current testing w/ old ISO once you get a 
> new one from Jason and continue your sanity and regression on 
> bare-metal. I am not sure how long it takes, but please take priority 
> as we want merge back to mainline early next week.
> 
> Thanks all for the support.
> - cindy
> 
> -----Original Message-----
> From: McKenna, Jason [mailto:Jason.McKenna at windriver.com]
> Sent: Thursday, September 13, 2018 4:24 AM
> To: Lin, Shuicheng <shuicheng.lin at intel.com>; Cordoba Malibran, Erich 
> <erich.cordoba.malibran at intel.com>; Waheed, Numan 
> <Numan.Waheed at windriver.com>; Rasa, Nimalini 
> <Nimalini.Rasa at windriver.com>; starlingx-discuss at lists.starlingx.io;
> Liu, ZhipengS <zhipengs.liu at intel.com>; Khalil, Ghada 
> <Ghada.Khalil at windriver.com>; Xie, Cindy <cindy.xie at intel.com>
> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
> 
> > -----Original Message-----
> > From: Lin, Shuicheng <shuicheng.lin at intel.com>
> > Sent: September 11, 2018 8:44 PM
> > To: McKenna, Jason <Jason.McKenna at windriver.com>; Cordoba Malibran, 
> > Erich <erich.cordoba.malibran at intel.com>; Waheed, Numan 
> > <Numan.Waheed at windriver.com>; Rasa, Nimalini 
> > <Nimalini.Rasa at windriver.com>; starlingx-discuss at lists.starlingx.io;
> > Liu, ZhipengS <zhipengs.liu at intel.com>; Khalil, Ghada 
> > <Ghada.Khalil at windriver.com>; Xie, Cindy <cindy.xie at intel.com>
> > Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
> >
> > Hi Jason,
> > Here is the dhcp src rpm we are using:
> > [slin14 at f998ec953c85 /]$ sudo -E yumdownloader -C --url --source 
> > dhcp
> > http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5
> > -
> > 68.el7.centos.1.src.rpm
> >
> > For centos-release, there are two packages in the mirror lst, 1 src 
> > rpm for build, and 1 rpm for the mock environment. And it has the 
> > "slightly difference" in the names as you mentioned.
> > Both of them could be found in the centos repo with the exact name.
> > ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm
> > ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm
> > [slin14 at f998ec953c85 /]$ sudo -E yumdownloader -C --url --source
> > centos- release-7-5.1804.1.el7.centos
> > http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos-
> > release-7-5.1804.1.el7.centos.src.rpm
> > [slin14 at f998ec953c85 /]$ sudo -E yumdownloader -C --url
> > centos-release-7-
> > 5.1804.el7.centos.x86_64
> > http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos-
> > release-7-5.1804.el7.centos.x86_64.rpm
> >
> > If there is still some other rpm missing, you could try to find them 
> > from the repo.
> > I have try my best to keep my local mirror the same as the lst. But 
> > it is still possible some package missed to be added to the repo lst.
> > Please feel free to notify me or correct the error directly.
> > Thanks.
> 
> With the latest commits to stx-tools I am now able to download the 
> rpms and produce a build.  I'll let you know how testing goes
> 
> >
> > Best Regards
> > Shuicheng
> >
> >
> > -----Original Message-----
> > From: McKenna, Jason [mailto:Jason.McKenna at windriver.com]
> > Sent: Wednesday, September 12, 2018 12:21 AM
> > To: Cordoba Malibran, Erich <erich.cordoba.malibran at intel.com>;
> > Waheed, Numan <Numan.Waheed at windriver.com>; Rasa, Nimalini 
> > <Nimalini.Rasa at windriver.com>; starlingx-discuss at lists.starlingx.io;
> > Liu, ZhipengS <zhipengs.liu at intel.com>; Khalil, Ghada 
> > <Ghada.Khalil at windriver.com>; Lin, Shuicheng 
> > <shuicheng.lin at intel.com>; Xie, Cindy <cindy.xie at intel.com>
> > Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
> >
> > Hi Cindy,
> >
> > With the latest centos75 branch (and the cherry-picks you mentioned) 
> > I am unable to produce a build.  There seems to be several source 
> > rpm files missing from the currently pushed version of the .lst 
> > files (or the .lst files pull down the wrong version).  Most of the 
> > missing files I can download manually, however the checked in 
> > patches for dhcp and centos-release packages do not apply against 
> > the version of the source RPM which I can manually find on the 
> > Internet.  For example, the version of centos-release called out in 
> > the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the 
> > version called out in the checked in code base is the slightly 
> > different
> > centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
> checked in do not apply against either package.
> >
> > Until the centos75 branch of the lst files pulls down the correct 
> > rpms, I will be unable to continue the validation effort.
> >
> > -J
> >
> > > -----Original Message-----
> > > From: Cordoba Malibran, Erich <erich.cordoba.malibran at intel.com>
> > > Sent: September 11, 2018 11:17 AM
> > > To: Waheed, Numan <Numan.Waheed at windriver.com>; Rasa, Nimalini 
> > > <Nimalini.Rasa at windriver.com>; 
> > > starlingx-discuss at lists.starlingx.io;
> > > Liu, ZhipengS <zhipengs.liu at intel.com>; Khalil, Ghada 
> > > <Ghada.Khalil at windriver.com>; Lin, Shuicheng 
> > > <shuicheng.lin at intel.com>; McKenna, Jason 
> > > <Jason.McKenna at windriver.com>; Xie, Cindy <cindy.xie at intel.com>
> > > Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
> > >
> > > Hi
> > >
> > > You can add "debug loglevel=8" to the kernel boot option to get a 
> > > more detailed output of the crash (if any)
> > >
> > > -Erich
> > >
> > > On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
> > > > Shuicheng saw the symbol table error issue as well but it 
> > > > recovers in 8s, he is still working on debugging it. But 601204 
> > > > is just kernel upgrade but not intended to solve this issues.
> > > > Are you running bare metal? I am wondering if this is HW config and driver issue.
> > > >
> > > > Is it possible that you try to deploy it in virtual environment?
> > > >
> > > > Thx. - cindy
> > > >
> > > > From: McKenna, Jason [mailto:Jason.McKenna at windriver.com]
> > > > Sent: Tuesday, September 11, 2018 9:50 PM
> > > > To: Xie, Cindy <cindy.xie at intel.com>; Rasa, Nimalini 
> > > > <Nimalini.Rasa at w indriver.com>; Waheed, Numan 
> > > > <Numan.Waheed at windriver.com>; Lin, Shuicheng 
> > > > <shuicheng.lin at intel.com>; Liu, ZhipengS <zhipengs.liu at inte 
> > > > l.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>; 
> > > > 'starlingx-discus s at lists.starlingx.io'
> > > > <starlingx-discuss at lists.starlingx.io>
> > > > Subject: RE: CentOS 7.5 upgrade status update
> > > >
> > > > Hi Cindy,
> > > >
> > > > We got past the installation device error, but are coming up 
> > > > against the symbol table error.  Our system hangs after this 
> > > > message.  I suspect a kernel issue.  I see there is an open code 
> > > > review to refactor some kernel patches to a rebased kernel 
> > > > (https://review.open
> > > > stack.org/#/c/601204/) but I do not know if that is expected to 
> > > > resolve the issue
> > > >
> > > > -J
> > > >
> > > > From: Xie, Cindy <cindy.xie at intel.com>
> > > > Sent: September 11, 2018 9:32 AM
> > > > To: Rasa, Nimalini <Nimalini.Rasa at windriver.com>; Waheed, Numan
> > > <Numa
> > > > n.Waheed at windriver.com>; Lin, Shuicheng
> <shuicheng.lin at intel.com>;
> > > > McKenna, Jason <Jason.McKenna at windriver.com>; Liu, ZhipengS
> > > <zhipengs
> > > > .liu at intel.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>; 
> > > > 'starlin gx-discuss at lists.starlingx.io'
> > > > <starlingx-discuss at lists.starlingx.io>
> > > > Subject: RE: CentOS 7.5 upgrade status update
> > > >
> > > > But as Jason reported yesterday, the build cannot be installed 
> > > > with error messages:
> > > >
> > > > Installation failed.
> > > > ERROR: Specified installation (sda) or boot (sda) device is invalid.
> > > >
> > > > We cannot debug without more information, the ISO installs in 
> > > > Intel’s HW. Thus Shuicheng created a new build and attached a 
> > > > manifest in this email thread. Did WindRiver has a chance to 
> > > > create a new ISO using today’s manifest file and re-run the test?
> > > >
> > > > Thx. - cindy
> > > >
> > > > From: Rasa, Nimalini [mailto:Nimalini.Rasa at windriver.com]
> > > > Sent: Tuesday, September 11, 2018 9:25 PM
> > > > To: Xie, Cindy <cindy.xie at intel.com>; Waheed, Numan
> > > <Numan.Waheed at win
> > > > driver.com>; Lin, Shuicheng <shuicheng.lin at intel.com>; McKenna, 
> > > > Jason <Jason.McKenna at windriver.com>; Liu, ZhipengS
> > > <zhipengs.liu at intel.com>
> > > > ; Khalil, Ghada <Ghada.Khalil at windriver.com>; 
> > > > 'starlingx-discuss at list s.starlingx.io'
> > > > <starlingx-discuss at lists.starlingx.io>
> > > > Subject: RE: CentOS 7.5 upgrade status update
> > > >
> > > > This is on the same build (yesterday’s build).
> > > >
> > > > From: Xie, Cindy [mailto:cindy.xie at intel.com]
> > > > Sent: Tuesday, September 11, 2018 9:22 AM
> > > > To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, 
> > > > Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss at lists.starlingx.io'
> > > > Subject: RE: CentOS 7.5 upgrade status update
> > > >
> > > > Is this the same issue as yesterday’s build? or this is 
> > > > different issue now?
> > > >
> > > > From: Rasa, Nimalini [mailto:Nimalini.Rasa at windriver.com]
> > > > Sent: Tuesday, September 11, 2018 9:06 PM
> > > > To: Waheed, Numan <Numan.Waheed at windriver.com>; Lin,
> Shuicheng
> > > <shuic
> > > > heng.lin at intel.com>; McKenna, Jason
> > <Jason.McKenna at windriver.com>;
> > > > Liu, ZhipengS <zhipengs.liu at intel.com>; Khalil, Ghada 
> > > > <Ghada.Khalil at w indriver.com>; 
> > > > 'starlingx-discuss at lists.starlingx.io' <starlingx-disc 
> > > > uss at lists.starlingx.io>; Xie, Cindy <cindy.xie at intel.com>
> > > > Subject: RE: CentOS 7.5 upgrade status update
> > > >
> > > > It is not booting in my case. After the  following , it just 
> > > > hangs
> > > > there:
> > > > “Error: no symbol table.
> > > > Press any key to continue…”
> > > >
> > > > -Nimalini
> > > >
> > > > From: Waheed, Numan
> > > > Sent: Tuesday, September 11, 2018 8:50 AM
> > > > To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, 
> > > > Ghada; 'st arlingx-discuss at lists.starlingx.io'; Xie, Cindy; 
> > > > Rasa, Nimalini
> > > > Subject: RE: CentOS 7.5 upgrade status update
> > > >
> > > > Adding Nimalini
> > > >
> > > > From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
> > > > Sent: September-11-18 2:17 AM
> > > > To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 
> > > > 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, 
> > > > Numan
> > > > Subject: RE: CentOS 7.5 upgrade status update
> > > >
> > > > Hi Jason,
> > > > Just finished deploy image with latest code, I could config 
> > > > controller successfully in bare metal environment.
> > > > Manifest is created for you reference. There are two patches I 
> > > > add
> > > > manually:
> > > > https://review.openstack.org/596855            for 
> > > > python-horizon from master
> > > > https://review.openstack.org/599892            for build-iso
> > > >
> > > >
> > > > There is an issue found when reboot controller node.
> > > > After the OS selection menu, it will show below error. And it 
> > > > will auto continue boot after around 8 seconds. I will check the 
> > > > issue later.
> > > > “
> > > > Error: no symbol table.
> > > > Press any key to continue…
> > > > “
> > > >
> > > > Best Regards
> > > > Shuicheng
> > > >
> > > > From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
> > > > Sent: Tuesday, September 11, 2018 9:35 AM
> > > > To: McKenna, Jason <Jason.McKenna at windriver.com>; Liu, ZhipengS
> > > <zhip
> > > > engs.liu at intel.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>; 
> > > > 'sta rlingx-discuss at lists.starlingx.io'
> > > > <starlingx-discuss at lists.starlingx
> > > > .io>; Xie, Cindy <cindy.xie at intel.com>; Waheed, Numan
> > > <Numan.Waheed at w
> > > > indriver.com>
> > > > Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status 
> > > > update
> > > >
> > > > Hi Jason,
> > > > I just tried my last Friday’s image at bare metal machine. There 
> > > > is no issue to install controller node.
> > > > Will you help check what cause the install error in your environment?
> > > >
> > > > I am syncing my code to latest to double confirm it.
> > > >
> > > > Best Regards
> > > > Shuicheng
> > > >
> > > > From: McKenna, Jason [mailto:Jason.McKenna at windriver.com]
> > > > Sent: Tuesday, September 11, 2018 12:13 AM
> > > > To: Liu, ZhipengS <zhipengs.liu at intel.com>; Lin, Shuicheng 
> > > > <shuicheng .lin at intel.com>; Khalil, Ghada 
> > > > <Ghada.Khalil at windriver.com>; 'starlin 
> > > > gx-discuss at lists.starlingx.io'
> > > > <starlingx-discuss at lists.starlingx.io>
> > > > ; Xie, Cindy <cindy.xie at intel.com>; Waheed, Numan
> > > <Numan.Waheed at windr
> > > > iver.com>
> > > > Subject: RE: CentOS 7.5 upgrade status update
> > > >
> > > > Status update.  The build from the centos75 branch does not 
> > > > install on our hardware.  The error message we are seeing is:
> > > >
> > > > Installation failed.
> > > >
> > > > ERROR: Specified installation (sda) or boot (sda) device is invalid.
> > > >
> > > > -Jason
> > > >
> > > > From: Liu, ZhipengS <zhipengs.liu at intel.com>
> > > > Sent: September 8, 2018 9:56 AM
> > > > To: Lin, Shuicheng <shuicheng.lin at intel.com>; McKenna, Jason 
> > > > <Jason.M cKenna at windriver.com>; Khalil, Ghada 
> > > > <Ghada.Khalil at windriver.com>; 's 
> > > > tarlingx-discuss at lists.starlingx.io'
> > > > <starlingx-discuss at lists.starlin gx.io>; Xie, Cindy 
> > > > <cindy.xie at intel.com>; Waheed, Numan
> > > <Numan.Waheed
> > > > @windriver.com>
> > > > Subject: RE: CentOS 7.5 upgrade status update
> > > >
> > > > Hi Jason,
> > > >
> > > > I have fixed the issue you mentioned below.
> > > > -        Furthermore, the tpm2-tools package was requesting version
> > > > tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7).  
> > > > I hacked the spec file for tpm2-tools and rebuilt it to require 
> > > > the version mentioned, but there is some rework to be done there.
> > > > [slin14]: I have the same issue also. Let’s further check it to 
> > > > find the root cause. I use rebuild tpm2-tools/tss2 to workaround 
> > > > it without any code.
> > > >
> > > > https://review.openstack.org/#/c/600920/
> > > >
> > > > Thanks!
> > > > Zhipeng
> > > > From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
> > > > Sent: 2018年9月8日 21:14
> > > > To: McKenna, Jason <Jason.McKenna at windriver.com>; Khalil, Ghada
> > > <Ghad
> > > > a.Khalil at windriver.com>; 'starlingx-discuss at lists.starlingx.io'
> > > > <star lingx-discuss at lists.starlingx.io>; Xie, Cindy 
> > > > <cindy.xie at intel.com>; Waheed, Numan
> > <Numan.Waheed at windriver.com>
> > > > Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status 
> > > > update
> > > >
> > > > Hi Jason,
> > > > I have fixed the error in the openstack-aodh. Patch is updated:
> > > > https://review.openstack.org/#/c/598388/
> > > >
> > > > It is due to my local version is not the latest one.
> > > > Sorry for the trouble caused, and thanks for your test.
> > > >
> > > > Best Regards
> > > > Shuicheng
> > > >
> > > > From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
> > > > Sent: Saturday, September 8, 2018 9:23 AM
> > > > To: McKenna, Jason <Jason.McKenna at windriver.com>; Khalil, Ghada
> > > <Ghad
> > > > a.Khalil at windriver.com>; 'starlingx-discuss at lists.starlingx.io'
> > > > <star lingx-discuss at lists.starlingx.io>; Xie, Cindy 
> > > > <cindy.xie at intel.com>; Waheed, Numan
> > <Numan.Waheed at windriver.com>
> > > > Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status 
> > > > update
> > > >
> > > > Hi Jason,
> > > > Thanks very much for such detail feedback.
> > > > I added some comments inline, and will check the issue in my 
> > > > environment also.
> > > > Feel free to submit patch if you have to fix them. J
> > > >
> > > >
> > > > Best Regards
> > > > Shuicheng
> > > >
> > > > From: McKenna, Jason [mailto:Jason.McKenna at windriver.com]
> > > > Sent: Saturday, September 8, 2018 3:47 AM
> > > > To: McKenna, Jason <Jason.McKenna at windriver.com>; Khalil, Ghada
> > > <Ghad
> > > > a.Khalil at windriver.com>; Lin, Shuicheng 
> > > > <shuicheng.lin at intel.com>; 's tarlingx-discuss at lists.starlingx.io'
> > > > <starlingx-discuss at lists.starlin gx.io>; Xie, Cindy 
> > > > <cindy.xie at intel.com>; Waheed, Numan
> > > <Numan.Waheed
> > > > @windriver.com>
> > > > Subject: RE: CentOS 7.5 upgrade status update
> > > >
> > > > Hi Shuicheng,
> > > >
> > > > I’ve completed a build of the code.  We have not yet run 
> > > > run-time tests, but here are the build-time issues I encountered:
> > > >
> > > > -        The .lst file contains lighttpd-1.4.49, but this is no
> > > > longer available for download, and the download_mirrors.sh step fails.
> > > > It appears than CentOS has updated to lighttpd-1.4.50.  I 
> > > > manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst
> > > > -        Even though the .lst file specifies lighttpd-1.4.49, the
> > > > code still references 1.4.45
> > > > -        At build-pkgs time (the build-srpms step), the meta-patches
> > > > to openstack-aodh do not apply, as two of them are malformed.  https:
> > > > //review.openstack.org/#/c/598388/6/openstack/openstack-
> > > > aodh/centos/meta_patches/0001-Modify-service-files-and-create-
> > > > expirer-cron-script.patch -- In the new version, on line 190, 
> > > > the patch states that the old code covered 6 lines, and the new 
> > > > code covers 9 lines (-283,6 +286,9).  But there are only two 
> > > > lines inserted by the patch.  There is also a new blank line in 
> > > > the patch at line 194.  Removing this blank line and editing the 
> > > > hunk description to only insert 2 lines resolves the issue for 
> > > > this hunk (there are other similar issues in this patch, and in
> > > > 0001-meta-
> > > > modify-aodh-api.patch)
> > > > [slin14]: Let me double check it.
> > > > -        At binary build time, I had build failures on python-django-
> > > > horizon and some magnum packages.  I had to pull in 
> > > > https://review.op
> > > > enstack.org/#/c/596855/2 (from the master branch, but not in the
> > > > centos75 branch of stx-upstream) to get past this.
> > > > [slin14]: It is due to I have not synced latest master code, so 
> > > > I don’t have this issue. I used last week’s master code.
> > > > -        After building all packages, I attempted to build the ISO.
> > > > The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and 
> > > > should be added to the .lst files.
> > > > [slin14]: This package is already in the lst.
> > > > -        Furthermore, the tpm2-tools package was requesting version
> > > > tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7).  
> > > > I hacked the spec file for tpm2-tools and rebuilt it to require 
> > > > the version mentioned, but there is some rework to be done there.
> > > > [slin14]: I have the same issue also. Let’s further check it to 
> > > > find the root cause. I use rebuild tpm2-tools/tss2 to workaround 
> > > > it without any code.
> > > >
> > > > After these changes, I was able to produce a .iso
> > > >
> > > > -Jason
> > > >
> > > > From: McKenna, Jason <Jason.McKenna at windriver.com>
> > > > Sent: September 7, 2018 10:08 AM
> > > > To: Khalil, Ghada <Ghada.Khalil at windriver.com>; Lin, Shuicheng 
> > > > <shuic heng.lin at intel.com>; 'starlingx-discuss at lists.starlingx.io'
> > > > <starling x-discuss at lists.starlingx.io>; Xie, Cindy 
> > > > <cindy.xie at intel.com>; Waheed, Numan
> > <Numan.Waheed at windriver.com>
> > > > Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status 
> > > > update
> > > >
> > > > Hi Shuicheng,
> > > >
> > > > As a download_mirrors.sh time issue, the version of lighttpd has 
> > > > been changed from 1.4.45 to 1.4.49.  The 
> > > > lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or 
> > > > CentOS servers, as they’ve moved to 1.4.50.  I am attempting to 
> > > > locate an archived version of the 1.4.49 .src rpm.
> > > >
> > > > -Jason
> > > >
> > > > From: Khalil, Ghada <Ghada.Khalil at windriver.com>
> > > > Sent: September 7, 2018 9:36 AM
> > > > To: Lin, Shuicheng <shuicheng.lin at intel.com>; 
> > > > 'starlingx-discuss at list s.starlingx.io'
> > > > <starlingx-discuss at lists.starlingx.io>; Xie, Cindy <c 
> > > > indy.xie at intel.com>; Waheed, Numan
> > <Numan.Waheed at windriver.com>
> > > > Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status 
> > > > update
> > > >
> > > > Thanks Shuicheng. We’ll run some tests in Wind River. Numan will 
> > > > be priming this activity. He’ll share the results with the team 
> > > > sometime next week.
> > > >
> > > > Regards,
> > > > Ghada
> > > >
> > > > From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
> > > > Sent: Friday, September 07, 2018 1:32 AM
> > > > To: 'starlingx-discuss at lists.starlingx.io'; Khalil, Ghada; Xie, 
> > > > Cindy
> > > > Subject: CentOS 7.5 upgrade status update
> > > >
> > > > Hi all,
> > > > Most task of CentOS7.5 upgrade in story 2003389 is done. Please 
> > > > go ahead to have a try and verify it.
> > > > I expect there should be some issue due to we just did limited 
> > > > deploy test yet.
> > > >
> > > > Here is the detail status update of this week.
> > > > For the total 44 srpm we plan to upgrade:
> > > > 38 already merged in feature branch.
> > > > 3 kept as original version.  Anaconda upgrade is not needed.
> > > > python- os-vif has higher version than 7.5. python-eventlet kept 
> > > > due to new src rpm itself have known issue.
> > > > 2 under review with code ready. Puppet-haproxy and openstack-aodh.
> > > > 1 under debug. Lighttpd.
> > > >
> > > > Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
> > > >
> > > > Here is the build instruction:
> > > > 1.      Switch to f/centos75 branch for stx-tools/stx-integ/stx-
> > > > upstream/stx-root.
> > > > Cherry-pick below un-merged patch:
> > > > For puppet-haproxy:
> > > > https://review.openstack.org/600369
> > > > https://review.openstack.org/600359
> > > > For openstack-aodh:
> > > > https://review.openstack.org/598388
> > > > https://review.openstack.org/600591
> > > > For build-iso:
> > > > https://review.openstack.org/599892
> > > > For build-pkgs:
> > > > https://review.openstack.org/599901
> > > > https://review.openstack.org/600639
> > > >
> > > > 2.      Run mirror script to get the upgraded src rpm and related
> > > > rpm.
> > > > 3.      Try to build packages and ISO.
> > > >
> > > > For the detail status of CentOS7.5 upgrade, please check below story:
> > > > https://storyboard.openstack.org/#!/story/2003389
> > > >
> > > > Feel free to contact me if you have trouble on it.
> > > >
> > > > Best Regards
> > > > Shuicheng
> > > > _______________________________________________
> > > > Starlingx-discuss mailing list
> > > > Starlingx-discuss at lists.starlingx.io
> > > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis
> > > > cu
> > > > ss


More information about the Starlingx-discuss mailing list