Re: [Starlingx-discuss] CentOS 7.5 upgrade status update

17 Sep 2018

      Yes, let's discuss that. I did send out the agenda including a decision required for merge back.

-----Original Message-----
From: Jones, Bruce E 
Sent: Tuesday, September 18, 2018 7:26 AM
To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
...
My read for Numan's test report is also good. I think we are OK to merge back to mainline unless we hear objections?
Suggest we discuss this at the project team call Wednesday morning but yes, it's looking good.

       brucej

-----Original Message-----
From: Xie, Cindy
Sent: Monday, September 17, 2018 4:25 PM
To: Jones, Bruce E <bruce.e.jones@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update

The case " test_kernel_module_signatures" seems passing on Shuicheng's system. We will provide the HW config to compare. But Numan, please open up a Launchpad ticket so that we can track there.

My read for Numan's test report is also good. I think we are OK to merge back to mainline unless we hear objections?

Thx. - cindy

-----Original Message-----
From: Jones, Bruce E
Sent: Tuesday, September 18, 2018 5:55 AM
To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update

Numan, thank you, my take away from this is that the upgrade is looking pretty good.

Is there a bug opened for the kernel signature issue?

     brucej

-----Original Message-----
From: Waheed, Numan [mailto:Numan.Waheed@windriver.com]
Sent: Monday, September 17, 2018 1:59 PM
To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update

Hi,

Following testing was done for testing CentOS7.5 on a custom build created by Jason:

Load:
SW_VERSION="18.08"
BUILD_TARGET="Unknown"
BUILD_TYPE="Informal"
BUILD_ID="n/a"

JOB="n/a"
BUILD_BY="jmckenna"
BUILD_NUMBER="n/a"
BUILD_HOST="yow-cgts1-lx"
BUILD_DATE="2018-09-13 10:42:03 -0400"

BUILD_DIR="/"
WRS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root"
WRS_GIT_BRANCH="HEAD"
CGCS_SRC_DIR="/localdisk/designer/jmckenna/starlingx-centos75/cgcs-root/stx"
CGCS_GIT_BRANCH="HEAD"

Sanity
-	23 Test Cases

Nightly Regression
-	46 Test Cases (1 Failure for Kernel module Signature – tested and reproduced on three different labs)

Nova Regression
-	Executed on three labs (AIO-SX, Standard and Storage Lab)
-	Labs covered Mellanox, Niantic and Fortville NICs
-	AIO-SX: 
	o	Test Cases Executed: 209
	o	Passed: 204
	o	Failed: 5	(3 known issues, 1 Ping issue, 1 timing issue that passed on re-execution)

-	Standard Lab:
	o	Test Cases Executed: 373
	o	Passed: 346
	o	Failed: 27	(6 known issues, 1 ping issue, 20 passed on re-execution)

-	Storage:
	o	Test Cases Executed: 374
	o	Passed: 367
	o	Failed: 7	(6 issues also seen in STX branch without CentOS7.5, 1 test case passed on re-execution)

Failed test cases were re-executed and majority of them passed on re-execution. The remaining failed test cases are also failing on STX build without CentOS7.5 update. Hence, the failures are not caused by CentOS7.5 update.

The main issue seen so far is only about Kernel Signature. Another issue seen is no Inventory tab available in Horizon. However, that does not seem to be related to CentOS update.

Thanks,

Numan.

-----Original Message-----
From: Rasa, Nimalini
Sent: September-17-18 12:21 PM
To: Lin, Shuicheng; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io
Cc: Khalil, Ghada
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update

Hi Shuicheng,
In this test case, basically trying to allocate double the amount of 2M pages.

For the kernel signature, I tried it on 3 different hardware labs and got the following:
controller-0:~$ cat /proc/sys/kernel/tainted
12288
controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log 
2018-09-14T19:55:12.495 localhost kernel: notice [    8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel
2018-09-14T20:49:52.937 controller-0 kernel: notice [    9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel

Thanks,
Nimalini

-----Original Message-----
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com]
Sent: Friday, September 14, 2018 9:13 PM
To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io
Cc: Khalil, Ghada
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update

Hi Nimalini,
Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"? 

For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next?
"
controller-0:~$ cat /proc/sys/kernel/tainted
4096
controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ "

Best Regards
Shuicheng

-----Original Message-----
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com]
Sent: Saturday, September 15, 2018 3:02 AM
To: Xie, Cindy <cindy.xie@intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; starlingx-discuss@lists.starlingx.io
Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update

Hi Cindy,
Adding more clarification of the test cases that are failed:

FAIL       20180914 06:54:06            test_schedule_vm_mempage_config[1048576]
  Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted.

FAIL       20180914 07:57:34            test_kernel_module_signatures
Test case is doing the following:
        - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps:
        - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules
        - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module

Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know.

Thanks,
Nimalini

-----Original Message-----
From: Rasa, Nimalini
Sent: Friday, September 14, 2018 12:46 PM
To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss@lists.starlingx.io
Cc: Khalil, Ghada; Lin, Shuicheng
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update

Hi Cindy,

Here is the last test update, I am reporting the failed test cases only  here:

From Sanity:
FAIL       20180914 04:32:46            test_evacuate_vms  --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose)

From Nightly:
FAIL       20180914 06:54:06            test_schedule_vm_mempage_config[1048576]

LOG:
====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08>
[2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4
[2018-09-14 06:55:20,035] 419  DEBUG MainThread ssh.exec_cmd:: Executing command...
[2018-09-14 06:55:20,036] 262  DEBUG MainThread ssh.send    :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1'
[2018-09-14 06:55:21,883] 382  DEBUG MainThread ssh.expect  :: Output: 
+-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+
| processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) | 
| hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail
| | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M |
| vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G |
+-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+
| 0         | 7540           | 8000              | 2420           | True                     | 1024            | 1           | 0           | 20992       | 3258           | 1210           | None             | 0              | 0              | None             | True         |
| 1         | 13770          | 2000              | 12746          | True                     | 1024            | 1           | 0           | 0           | 6373           | 6373           | None             | 0              | 0              | None             | True         |
+-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+
controller-1:~$ 
[2018-09-14 06:55:21,884] 262  DEBUG MainThread ssh.send    :: Send 'echo $?'
[2018-09-14 06:55:21,986] 382  DEBUG MainThread ssh.expect  :: Output: 
0
controller-1:~$
[2018-09-14 06:55:21,987] 366  DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0]
[2018-09-14 06:55:21,987] 498  DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420']
[2018-09-14 06:55:21,987] 366  DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0]
[2018-09-14 06:55:21,988] 498  DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0']
[2018-09-14 06:55:21,988] 366  DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0]
[2018-09-14 06:55:21,988] 498  DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None']
[2018-09-14 06:55:21,988] 366  DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1]
[2018-09-14 06:55:21,989] 498  DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746']
[2018-09-14 06:55:21,989] 366  DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1]
[2018-09-14 06:55:21,989] 498  DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0']
[2018-09-14 06:55:21,989] 366  DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1]
[2018-09-14 06:55:21,990] 498  DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None']
[2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4
[2018-09-14 06:55:21,990] 419  DEBUG MainThread ssh.exec_cmd:: Executing command...
[2018-09-14 06:55:21,990] 262  DEBUG MainThread ssh.send    :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0'
[2018-09-14 06:55:24,362] 382  DEBUG MainThread ssh.expect  :: Output: 
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| Memory: Usable Total (MiB)     | 7540                                 |
|         Platform     (MiB)     | 8000                                 |
|         Available    (MiB)     | 2420                                 |
| Huge Pages Configured          | True                                 |
| vSwitch Huge Pages: Size (MiB) | 1024                                 |
|                     Total      | 1                                    |
|                     Available  | 0                                    |
| VM  Pages (4K): Total          | 20992                                |
| VM  Huge Pages (2M): Total     | 3258                                 |
|                 Total Pending  | 2420                                 |
|                 Available      | 1210                                 |
| VM  Huge Pages (1G): Total     | 0                                    |
|                 Available      | 0                                    |
| uuid                           | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 |
| ihost_uuid                     | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea |
| inode_uuid                     | 96430065-9a83-4e33-ab78-c7de77cf8bc9 |
| created_at                     | 2018-09-14T02:30:56.591922+00:00     |
| updated_at                     | 2018-09-14T06:48:29.882522+00:00     |
+--------------------------------+--------------------------------------+
controller-1:~$ 
[2018-09-14 06:55:24,362] 262  DEBUG MainThread ssh.send    :: Send 'echo $?'
[2018-09-14 06:55:24,465] 382  DEBUG MainThread ssh.expect  :: Output: 
0

EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD:
[2018-09-14 12:37:33,741] 262  DEBUG MainThread ssh.send    :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0'
[2018-09-14 12:37:36,023] 382  DEBUG MainThread ssh.expect  :: Output: 
Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848

FAIL       20180914 07:00:37            test_schedule_vm_mempage_config[small]
	This test failed b/c of the above test failure (Alarm was present, compute was in locked state)

FAIL       20180914 07:57:34            test_kernel_module_signatures
LOG

[2018-09-14 07:57:37,946] 24   INFO  MainThread test_kernel_modules.test_kernel_module_signatures:: 
====================== Test Step 1: Check for unassigned kernel modules on controller-0
[2018-09-14 07:57:37,947] 419  DEBUG MainThread ssh.exec_cmd:: Executing command...
[2018-09-14 07:57:37,947] 262  DEBUG MainThread ssh.send    :: Send 'cat /proc/sys/kernel/tainted'
[2018-09-14 07:57:38,050] 382  DEBUG MainThread ssh.expect  :: Output: 
12288
controller-0:~$ 
[2018-09-14 07:57:38,050] 262  DEBUG MainThread ssh.send    :: Send 'echo $?'
[2018-09-14 07:57:38,152] 382  DEBUG MainThread ssh.expect  :: Output: 
0
controller-0:~$ 
[2018-09-14 07:57:38,153] 32   ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info
[2018-09-14 07:57:38,153] 34   INFO  MainThread test_kernel_modules.test_kernel_module_signatures:: 
====================== Test Step 2: Check kern.log for modules with failed verification
[2018-09-14 07:57:38,153] 419  DEBUG MainThread ssh.exec_cmd:: Executing command...
[2018-09-14 07:57:38,153] 262  DEBUG MainThread ssh.send    :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log'
[2018-09-14 07:57:38,265] 382  DEBUG MainThread ssh.expect  :: Output: 
2018-09-13T19:09:51.212 localhost kernel: notice [    6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel
2018-09-13T19:51:42.174 controller-0 kernel: notice [    6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel
2018-09-14T03:33:29.054 controller-0 kernel: notice [    6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel
controller-0:~$ 
[2018-09-14 07:57:38,265] 262  DEBUG MainThread ssh.send    :: Send 'echo $?'
[2018-09-14 07:57:38,367] 382  DEBUG MainThread ssh.expect  :: Output: 
0
controller-0:~$ 
[2018-09-14 07:57:38,368] 43   INFO  MainThread test_kernel_modules.test_kernel_module_signatures:: 
====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e']
[2018-09-14 07:57:38,368] 419  DEBUG MainThread ssh.exec_cmd:: Executing command...
[2018-09-14 07:57:38,368] 262  DEBUG MainThread ssh.send    :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"'
[2018-09-14 07:57:38,474] 382  DEBUG MainThread ssh.expect  :: Output: 
filename:       /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko
signer:         CentOS Linux kernel signing key
sig_key:        58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC
sig_hashalgo:   sha256
controller-0:~$ 
[2018-09-14 07:57:38,475] 262  DEBUG MainThread ssh.send    :: Send 'echo $?'
[2018-09-14 07:57:38,577] 382  DEBUG MainThread ssh.expect  :: Output:

Thanks,
Nimalini

-----Original Message-----
From: Xie, Cindy [mailto:cindy.xie@intel.com]
Sent: Thursday, September 13, 2018 10:05 AM
To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss@lists.starlingx.io
Cc: Khalil, Ghada; Lin, Shuicheng
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update

Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back.

The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818

Thanks. - cindy

-----Original Message-----
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com]
Sent: Thursday, September 13, 2018 9:35 PM
To: Xie, Cindy <cindy.xie@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini <Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io
Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update

Hi Cindy,

Will do.  In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite.  The results (attached) align with the 7.4 based master branch of StarlingX.

-Jason
...
-----Original Message-----
From: Xie, Cindy <cindy.xie@intel.com>
Sent: September 13, 2018 2:44 AM
To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; McKenna, 
Jason <Jason.McKenna@windriver.com>; Waheed, Numan 
<Numan.Waheed@windriver.com>; Rasa, Nimalini 
<Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io
Cc: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng 
<shuicheng.lin@intel.com>
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Elio,
We will provide you an updated ISO later today with the following:
- kernel upgrade to 3.10.0-862.11.6
- symbol issue fixed: 
https://bugs.launchpad.net/starlingx/+bug/1792257
Please help to do a sanity test based on this new ISO from Shuicheng on VE.
@Jason,
Please re-generate a build using today's manifest with above fix. 
Shuicheng shall be able to provide manifest with exact commitID. 
@Numan, please abort your current testing w/ old ISO once you get a 
new one from Jason and continue your sanity and regression on 
bare-metal. I am not sure how long it takes, but please take priority 
as we want merge back to mainline early next week.
Thanks all for the support.
- cindy
-----Original Message-----
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com]
Sent: Thursday, September 13, 2018 4:24 AM
To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich 
<erich.cordoba.malibran@intel.com>; Waheed, Numan 
<Numan.Waheed@windriver.com>; Rasa, Nimalini 
<Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io;
Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada 
<Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com>
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
...
-----Original Message-----
From: Lin, Shuicheng <shuicheng.lin@intel.com>
Sent: September 11, 2018 8:44 PM
To: McKenna, Jason <Jason.McKenna@windriver.com>; Cordoba Malibran, 
Erich <erich.cordoba.malibran@intel.com>; Waheed, Numan 
<Numan.Waheed@windriver.com>; Rasa, Nimalini 
<Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io;
Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada 
<Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com>
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Jason,
Here is the dhcp src rpm we are using:
[slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source 
dhcp
http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5
-
68.el7.centos.1.src.rpm
For centos-release, there are two packages in the mirror lst, 1 src 
rpm for build, and 1 rpm for the mock environment. And it has the 
"slightly difference" in the names as you mentioned.
Both of them could be found in the centos repo with the exact name.
./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm
./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm
[slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url --source
centos- release-7-5.1804.1.el7.centos
http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos-
release-7-5.1804.1.el7.centos.src.rpm
[slin14@f998ec953c85 /]$ sudo -E yumdownloader -C --url
centos-release-7-
5.1804.el7.centos.x86_64
http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos-
release-7-5.1804.el7.centos.x86_64.rpm
If there is still some other rpm missing, you could try to find them 
from the repo.
I have try my best to keep my local mirror the same as the lst. But 
it is still possible some package missed to be added to the repo lst.
Please feel free to notify me or correct the error directly.
Thanks.
With the latest commits to stx-tools I am now able to download the 
rpms and produce a build.  I'll let you know how testing goes
...
Best Regards
Shuicheng
-----Original Message-----
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com]
Sent: Wednesday, September 12, 2018 12:21 AM
To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>;
Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini 
<Nimalini.Rasa@windriver.com>; starlingx-discuss@lists.starlingx.io;
Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada 
<Ghada.Khalil@windriver.com>; Lin, Shuicheng 
<shuicheng.lin@intel.com>; Xie, Cindy <cindy.xie@intel.com>
Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi Cindy,
With the latest centos75 branch (and the cherry-picks you mentioned) 
I am unable to produce a build.  There seems to be several source 
rpm files missing from the currently pushed version of the .lst 
files (or the .lst files pull down the wrong version).  Most of the 
missing files I can download manually, however the checked in 
patches for dhcp and centos-release packages do not apply against 
the version of the source RPM which I can manually find on the 
Internet.  For example, the version of centos-release called out in 
the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the 
version called out in the checked in code base is the slightly 
different
centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
checked in do not apply against either package.
...
Until the centos75 branch of the lst files pulls down the correct 
rpms, I will be unable to continue the validation effort.
-J
...
-----Original Message-----
From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>
Sent: September 11, 2018 11:17 AM
To: Waheed, Numan <Numan.Waheed@windriver.com>; Rasa, Nimalini 
<Nimalini.Rasa@windriver.com>; 
starlingx-discuss@lists.starlingx.io;
Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada 
<Ghada.Khalil@windriver.com>; Lin, Shuicheng 
<shuicheng.lin@intel.com>; McKenna, Jason 
<Jason.McKenna@windriver.com>; Xie, Cindy <cindy.xie@intel.com>
Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
Hi
You can add "debug loglevel=8" to the kernel boot option to get a 
more detailed output of the crash (if any)
-Erich
On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
...
Shuicheng saw the symbol table error issue as well but it 
recovers in 8s, he is still working on debugging it. But 601204 
is just kernel upgrade but not intended to solve this issues.
Are you running bare metal? I am wondering if this is HW config and driver issue.
Is it possible that you try to deploy it in virtual environment?
Thx. - cindy
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com]
Sent: Tuesday, September 11, 2018 9:50 PM
To: Xie, Cindy <cindy.xie@intel.com>; Rasa, Nimalini 
<Nimalini.Rasa@w indriver.com>; Waheed, Numan 
<Numan.Waheed@windriver.com>; Lin, Shuicheng 
<shuicheng.lin@intel.com>; Liu, ZhipengS <zhipengs.liu@inte 
l.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 
'starlingx-discus s@lists.starlingx.io'
<starlingx-discuss@lists.starlingx.io>
Subject: RE: CentOS 7.5 upgrade status update
Hi Cindy,
We got past the installation device error, but are coming up 
against the symbol table error.  Our system hangs after this 
message.  I suspect a kernel issue.  I see there is an open code 
review to refactor some kernel patches to a rebased kernel 
(https://review.open
stack.org/#/c/601204/) but I do not know if that is expected to 
resolve the issue
-J
From: Xie, Cindy <cindy.xie@intel.com>
Sent: September 11, 2018 9:32 AM
To: Rasa, Nimalini <Nimalini.Rasa@windriver.com>; Waheed, Numan
<Numa
n.Waheed@windriver.com>; Lin, Shuicheng
<shuicheng.lin@intel.com>;
...
...
...
McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS
<zhipengs
.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 
'starlin gx-discuss@lists.starlingx.io'
<starlingx-discuss@lists.starlingx.io>
Subject: RE: CentOS 7.5 upgrade status update
But as Jason reported yesterday, the build cannot be installed 
with error messages:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
We cannot debug without more information, the ISO installs in 
Intel’s HW. Thus Shuicheng created a new build and attached a 
manifest in this email thread. Did WindRiver has a chance to 
create a new ISO using today’s manifest file and re-run the test?
Thx. - cindy
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com]
Sent: Tuesday, September 11, 2018 9:25 PM
To: Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan
<Numan.Waheed@win
driver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, 
Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS
<zhipengs.liu@intel.com>
; Khalil, Ghada <Ghada.Khalil@windriver.com>; 
'starlingx-discuss@list s.starlingx.io'
<starlingx-discuss@lists.starlingx.io>
Subject: RE: CentOS 7.5 upgrade status update
This is on the same build (yesterday’s build).
From: Xie, Cindy [mailto:cindy.xie@intel.com]
Sent: Tuesday, September 11, 2018 9:22 AM
To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna, 
Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss@lists.starlingx.io'
Subject: RE: CentOS 7.5 upgrade status update
Is this the same issue as yesterday’s build? or this is 
different issue now?
From: Rasa, Nimalini [mailto:Nimalini.Rasa@windriver.com]
Sent: Tuesday, September 11, 2018 9:06 PM
To: Waheed, Numan <Numan.Waheed@windriver.com>; Lin,
Shuicheng
<shuic
heng.lin@intel.com>; McKenna, Jason
<Jason.McKenna@windriver.com>;
Liu, ZhipengS <zhipengs.liu@intel.com>; Khalil, Ghada 
<Ghada.Khalil@w indriver.com>; 
'starlingx-discuss@lists.starlingx.io' <starlingx-disc 
uss@lists.starlingx.io>; Xie, Cindy <cindy.xie@intel.com>
Subject: RE: CentOS 7.5 upgrade status update
It is not booting in my case. After the  following , it just 
hangs
there:
“Error: no symbol table.
Press any key to continue…”
-Nimalini
From: Waheed, Numan
Sent: Tuesday, September 11, 2018 8:50 AM
To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil, 
Ghada; 'st arlingx-discuss@lists.starlingx.io'; Xie, Cindy; 
Rasa, Nimalini
Subject: RE: CentOS 7.5 upgrade status update
Adding Nimalini
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com]
Sent: September-11-18 2:17 AM
To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada; 
'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed, 
Numan
Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
Just finished deploy image with latest code, I could config 
controller successfully in bare metal environment.
Manifest is created for you reference. There are two patches I 
add
manually:
https://review.openstack.org/596855            for 
python-horizon from master
https://review.openstack.org/599892            for build-iso
There is an issue found when reboot controller node.
After the OS selection menu, it will show below error. And it 
will auto continue boot after around 8 seconds. I will check the 
issue later.
“
Error: no symbol table.
Press any key to continue…
“
Best Regards
Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com]
Sent: Tuesday, September 11, 2018 9:35 AM
To: McKenna, Jason <Jason.McKenna@windriver.com>; Liu, ZhipengS
<zhip
engs.liu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; 
'sta rlingx-discuss@lists.starlingx.io'
<starlingx-discuss@lists.starlingx
.io>; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan
<Numan.Waheed@w
indriver.com>
Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status 
update
Hi Jason,
I just tried my last Friday’s image at bare metal machine. There 
is no issue to install controller node.
Will you help check what cause the install error in your environment?
I am syncing my code to latest to double confirm it.
Best Regards
Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com]
Sent: Tuesday, September 11, 2018 12:13 AM
To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng 
<shuicheng .lin@intel.com>; Khalil, Ghada 
<Ghada.Khalil@windriver.com>; 'starlin 
gx-discuss@lists.starlingx.io'
<starlingx-discuss@lists.starlingx.io>
; Xie, Cindy <cindy.xie@intel.com>; Waheed, Numan
<Numan.Waheed@windr
iver.com>
Subject: RE: CentOS 7.5 upgrade status update
Status update.  The build from the centos75 branch does not 
install on our hardware.  The error message we are seeing is:
Installation failed.
ERROR: Specified installation (sda) or boot (sda) device is invalid.
-Jason
From: Liu, ZhipengS <zhipengs.liu@intel.com>
Sent: September 8, 2018 9:56 AM
To: Lin, Shuicheng <shuicheng.lin@intel.com>; McKenna, Jason 
<Jason.M cKenna@windriver.com>; Khalil, Ghada 
<Ghada.Khalil@windriver.com>; 's 
tarlingx-discuss@lists.starlingx.io'
<starlingx-discuss@lists.starlin gx.io>; Xie, Cindy 
<cindy.xie@intel.com>; Waheed, Numan
<Numan.Waheed
@windriver.com>
Subject: RE: CentOS 7.5 upgrade status update
Hi Jason,
I have fixed the issue you mentioned below.
-        Furthermore, the tpm2-tools package was requesting version
tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7).  
I hacked the spec file for tpm2-tools and rebuilt it to require 
the version mentioned, but there is some rework to be done there.
[slin14]: I have the same issue also. Let’s further check it to 
find the root cause. I use rebuild tpm2-tools/tss2 to workaround 
it without any code.
https://review.openstack.org/#/c/600920/
Thanks!
Zhipeng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com]
Sent: 2018年9月8日 21:14
To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada
<Ghad
a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io'
<star lingx-discuss@lists.starlingx.io>; Xie, Cindy 
<cindy.xie@intel.com>; Waheed, Numan
<Numan.Waheed@windriver.com>
Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status 
update
Hi Jason,
I have fixed the error in the openstack-aodh. Patch is updated:
https://review.openstack.org/#/c/598388/
It is due to my local version is not the latest one.
Sorry for the trouble caused, and thanks for your test.
Best Regards
Shuicheng
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com]
Sent: Saturday, September 8, 2018 9:23 AM
To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada
<Ghad
a.Khalil@windriver.com>; 'starlingx-discuss@lists.starlingx.io'
<star lingx-discuss@lists.starlingx.io>; Xie, Cindy 
<cindy.xie@intel.com>; Waheed, Numan
<Numan.Waheed@windriver.com>
Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status 
update
Hi Jason,
Thanks very much for such detail feedback.
I added some comments inline, and will check the issue in my 
environment also.
Feel free to submit patch if you have to fix them. J
Best Regards
Shuicheng
From: McKenna, Jason [mailto:Jason.McKenna@windriver.com]
Sent: Saturday, September 8, 2018 3:47 AM
To: McKenna, Jason <Jason.McKenna@windriver.com>; Khalil, Ghada
<Ghad
a.Khalil@windriver.com>; Lin, Shuicheng 
<shuicheng.lin@intel.com>; 's tarlingx-discuss@lists.starlingx.io'
<starlingx-discuss@lists.starlin gx.io>; Xie, Cindy 
<cindy.xie@intel.com>; Waheed, Numan
<Numan.Waheed
@windriver.com>
Subject: RE: CentOS 7.5 upgrade status update
Hi Shuicheng,
I’ve completed a build of the code.  We have not yet run 
run-time tests, but here are the build-time issues I encountered:
-        The .lst file contains lighttpd-1.4.49, but this is no
longer available for download, and the download_mirrors.sh step fails.
It appears than CentOS has updated to lighttpd-1.4.50.  I 
manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst
-        Even though the .lst file specifies lighttpd-1.4.49, the
code still references 1.4.45
-        At build-pkgs time (the build-srpms step), the meta-patches
to openstack-aodh do not apply, as two of them are malformed.  https:
//review.openstack.org/#/c/598388/6/openstack/openstack-
aodh/centos/meta_patches/0001-Modify-service-files-and-create-
expirer-cron-script.patch -- In the new version, on line 190, 
the patch states that the old code covered 6 lines, and the new 
code covers 9 lines (-283,6 +286,9).  But there are only two 
lines inserted by the patch.  There is also a new blank line in 
the patch at line 194.  Removing this blank line and editing the 
hunk description to only insert 2 lines resolves the issue for 
this hunk (there are other similar issues in this patch, and in
0001-meta-
modify-aodh-api.patch)
[slin14]: Let me double check it.
-        At binary build time, I had build failures on python-django-
horizon and some magnum packages.  I had to pull in 
https://review.op
enstack.org/#/c/596855/2 (from the master branch, but not in the
centos75 branch of stx-upstream) to get past this.
[slin14]: It is due to I have not synced latest master code, so 
I don’t have this issue. I used last week’s master code.
-        After building all packages, I attempted to build the ISO.
The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and 
should be added to the .lst files.
[slin14]: This package is already in the lst.
-        Furthermore, the tpm2-tools package was requesting version
tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7).  
I hacked the spec file for tpm2-tools and rebuilt it to require 
the version mentioned, but there is some rework to be done there.
[slin14]: I have the same issue also. Let’s further check it to 
find the root cause. I use rebuild tpm2-tools/tss2 to workaround 
it without any code.
After these changes, I was able to produce a .iso
-Jason
From: McKenna, Jason <Jason.McKenna@windriver.com>
Sent: September 7, 2018 10:08 AM
To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng 
<shuic heng.lin@intel.com>; 'starlingx-discuss@lists.starlingx.io'
<starling x-discuss@lists.starlingx.io>; Xie, Cindy 
<cindy.xie@intel.com>; Waheed, Numan
<Numan.Waheed@windriver.com>
Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status 
update
Hi Shuicheng,
As a download_mirrors.sh time issue, the version of lighttpd has 
been changed from 1.4.45 to 1.4.49.  The 
lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or 
CentOS servers, as they’ve moved to 1.4.50.  I am attempting to 
locate an archived version of the 1.4.49 .src rpm.
-Jason
From: Khalil, Ghada <Ghada.Khalil@windriver.com>
Sent: September 7, 2018 9:36 AM
To: Lin, Shuicheng <shuicheng.lin@intel.com>; 
'starlingx-discuss@list s.starlingx.io'
<starlingx-discuss@lists.starlingx.io>; Xie, Cindy <c 
indy.xie@intel.com>; Waheed, Numan
<Numan.Waheed@windriver.com>
Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status 
update
Thanks Shuicheng. We’ll run some tests in Wind River. Numan will 
be priming this activity. He’ll share the results with the team 
sometime next week.
Regards,
Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com]
Sent: Friday, September 07, 2018 1:32 AM
To: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Xie, 
Cindy
Subject: CentOS 7.5 upgrade status update
Hi all,
Most task of CentOS7.5 upgrade in story 2003389 is done. Please 
go ahead to have a try and verify it.
I expect there should be some issue due to we just did limited 
deploy test yet.
Here is the detail status update of this week.
For the total 44 srpm we plan to upgrade:
38 already merged in feature branch.
3 kept as original version.  Anaconda upgrade is not needed.
python- os-vif has higher version than 7.5. python-eventlet kept 
due to new src rpm itself have known issue.
2 under review with code ready. Puppet-haproxy and openstack-aodh.
1 under debug. Lighttpd.
Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
Here is the build instruction:
1.      Switch to f/centos75 branch for stx-tools/stx-integ/stx-
upstream/stx-root.
Cherry-pick below un-merged patch:
For puppet-haproxy:
https://review.openstack.org/600369
https://review.openstack.org/600359
For openstack-aodh:
https://review.openstack.org/598388
https://review.openstack.org/600591
For build-iso:
https://review.openstack.org/599892
For build-pkgs:
https://review.openstack.org/599901
https://review.openstack.org/600639
2.      Run mirror script to get the upgraded src rpm and related
rpm.
3.      Try to build packages and ISO.
For the detail status of CentOS7.5 upgrade, please check below story:
https://storyboard.openstack.org/#!/story/2003389
Feel free to contact me if you have trouble on it.
Best Regards
Shuicheng
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis
cu
ss

Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss