[Starlingx-discuss] CentOS 7.5 upgrade status update
Saul Wold
sgw at linux.intel.com
Mon Sep 17 19:58:48 UTC 2018
On 09/17/2018 09:21 AM, Rasa, Nimalini wrote:
> Hi Shuicheng,
> In this test case, basically trying to allocate double the amount of 2M pages.
>
> For the kernel signature, I tried it on 3 different hardware labs and got the following:
> controller-0:~$ cat /proc/sys/kernel/tainted
> 12288
> controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log
> 2018-09-14T19:55:12.495 localhost kernel: notice [ 8.016748] e1000e: module verification failed: signature and/or required key missing - tainting kernel
> 2018-09-14T20:49:52.937 controller-0 kernel: notice [ 9.517476] e1000e: module verification failed: signature and/or required key missing - tainting kernel
>
This could be a difference in HW that is being tested.
Nimalini and Shuicheng, can you both share your test hardware?
But, regardless of hardware it seems that something was not signed
correctly or possibly rebuilt. Shuicheng, if you don't have the e1000e
hardware, you could still try a modprobe of the kernel-module.
Sau!
>
> Thanks,
> Nimalini
>
> -----Original Message-----
> From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
> Sent: Friday, September 14, 2018 9:13 PM
> To: Rasa, Nimalini; Xie, Cindy; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss at lists.starlingx.io
> Cc: Khalil, Ghada
> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
>
> Hi Nimalini,
> Could you share me the detail reproduce step for " test_schedule_vm_mempage_config"?
>
> For the 2nd failure, I tried "'cat /proc/sys/kernel/tainted'" both in my virtual and bare metal environment, and both of them show 4096. Then what should I do next?
> "
> controller-0:~$ cat /proc/sys/kernel/tainted
> 4096
> controller-0:~$ grep --color=never -i "module verification failed" /var/log/kern.log controller-0:~$ "
>
> Best Regards
> Shuicheng
>
> -----Original Message-----
> From: Rasa, Nimalini [mailto:Nimalini.Rasa at windriver.com]
> Sent: Saturday, September 15, 2018 3:02 AM
> To: Xie, Cindy <cindy.xie at intel.com>; McKenna, Jason <Jason.McKenna at windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>; Waheed, Numan <Numan.Waheed at windriver.com>; starlingx-discuss at lists.starlingx.io
> Cc: Khalil, Ghada <Ghada.Khalil at windriver.com>; Lin, Shuicheng <shuicheng.lin at intel.com>
> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
>
> Hi Cindy,
> Adding more clarification of the test cases that are failed:
>
> FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576]
> Test case is trying to over allocate 2M pages, expected to reject the cli, in this case it's accepted.
>
> FAIL 20180914 07:57:34 test_kernel_module_signatures
> Test case is doing the following:
> - 'cat /proc/sys/kernel/tainted', ensure value is 4096. If not, do following steps:
> - 'grep --color=never -i "module verification failed" /var/log/kern.log' to find out failed modules
> - 'modinfo <failed_module> | grep --color=never -E "sig|filename" to display signing info for each module
>
>
> Hope it clarifies bit more as to what the test cases it trying to do. If you have any questions please let me know.
>
> Thanks,
> Nimalini
>
> -----Original Message-----
> From: Rasa, Nimalini
> Sent: Friday, September 14, 2018 12:46 PM
> To: 'Xie, Cindy'; McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; starlingx-discuss at lists.starlingx.io
> Cc: Khalil, Ghada; Lin, Shuicheng
> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
>
> Hi Cindy,
>
> Here is the last test update, I am reporting the failed test cases only here:
>
> From Sanity:
> FAIL 20180914 04:32:46 test_evacuate_vms --> Ping Vms failed and this is seen intermittently on STX load as well (can be ignored for this testing puprpose)
>
> From Nightly:
> FAIL 20180914 06:54:06 test_schedule_vm_mempage_config[1048576]
>
> LOG:
> ====================== Setup Step 6: (module) Execute modify function: <function add_1g_and_4k_pages.<locals>._modify at 0x7fc023a03d08>
> [2018-09-14 06:55:20,035] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4
> [2018-09-14 06:55:20,035] 419 DEBUG MainThread ssh.exec_cmd:: Executing command...
> [2018-09-14 06:55:20,036] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-list --nowrap compute-1'
> [2018-09-14 06:55:21,883] 382 DEBUG MainThread ssh.expect :: Output:
> +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+
> | processor | mem_total(MiB) | mem_platform(MiB) | mem_avail(MiB) |
> | hugepages(hp)_configured | vs_hp_size(MiB) | vs_hp_total | vs_hp_avail
> | | vm_total_4K | vm_hp_total_2M | vm_hp_avail_2M | vm_hp_pending_2M |
> | vm_hp_total_1G | vm_hp_avail_1G | vm_hp_pending_1G | vm_hp_use_1G |
> +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+
> | 0 | 7540 | 8000 | 2420 | True | 1024 | 1 | 0 | 20992 | 3258 | 1210 | None | 0 | 0 | None | True |
> | 1 | 13770 | 2000 | 12746 | True | 1024 | 1 | 0 | 0 | 6373 | 6373 | None | 0 | 0 | None | True |
> +-----------+----------------+-------------------+----------------+--------------------------+-----------------+-------------+-------------+-------------+----------------+----------------+------------------+----------------+----------------+------------------+--------------+
> controller-1:~$
> [2018-09-14 06:55:21,884] 262 DEBUG MainThread ssh.send :: Send 'echo $?'
> [2018-09-14 06:55:21,986] 382 DEBUG MainThread ssh.expect :: Output:
> 0
> controller-1:~$
> [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0]
> [2018-09-14 06:55:21,987] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['2420']
> [2018-09-14 06:55:21,987] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0]
> [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0']
> [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 0: [0]
> [2018-09-14 06:55:21,988] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None']
> [2018-09-14 06:55:21,988] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1]
> [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching mem_avail(MiB) value(s): ['12746']
> [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1]
> [2018-09-14 06:55:21,989] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_total_1G value(s): ['0']
> [2018-09-14 06:55:21,989] 366 DEBUG MainThread table_parser.__get_row_indexes_string:: row index list for processor: 1: [1]
> [2018-09-14 06:55:21,990] 498 DEBUG MainThread table_parser.get_values:: Returning matching vm_hp_pending_1G value(s): ['None']
> [2018-09-14 06:55:21,990] 1300 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for ip_1_4
> [2018-09-14 06:55:21,990] 419 DEBUG MainThread ssh.exec_cmd:: Executing command...
> [2018-09-14 06:55:21,990] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 2420 compute-1 0'
> [2018-09-14 06:55:24,362] 382 DEBUG MainThread ssh.expect :: Output:
> +--------------------------------+--------------------------------------+
> | Property | Value |
> +--------------------------------+--------------------------------------+
> | Memory: Usable Total (MiB) | 7540 |
> | Platform (MiB) | 8000 |
> | Available (MiB) | 2420 |
> | Huge Pages Configured | True |
> | vSwitch Huge Pages: Size (MiB) | 1024 |
> | Total | 1 |
> | Available | 0 |
> | VM Pages (4K): Total | 20992 |
> | VM Huge Pages (2M): Total | 3258 |
> | Total Pending | 2420 |
> | Available | 1210 |
> | VM Huge Pages (1G): Total | 0 |
> | Available | 0 |
> | uuid | 8f317d7a-3bcd-4b2d-bd1e-704429d0e0c5 |
> | ihost_uuid | 4189d5c3-0c85-49b9-9fc9-1d54540c54ea |
> | inode_uuid | 96430065-9a83-4e33-ab78-c7de77cf8bc9 |
> | created_at | 2018-09-14T02:30:56.591922+00:00 |
> | updated_at | 2018-09-14T06:48:29.882522+00:00 |
> +--------------------------------+--------------------------------------+
> controller-1:~$
> [2018-09-14 06:55:24,362] 262 DEBUG MainThread ssh.send :: Send 'echo $?'
> [2018-09-14 06:55:24,465] 382 DEBUG MainThread ssh.expect :: Output:
> 0
>
> EXPECTED TO REJECT: HERE IS AN EXAMPLE FROM PASSED LOG ON DIFF LOAD:
> [2018-09-14 12:37:33,741] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne host-memory-modify -2M 53818 compute-1 0'
> [2018-09-14 12:37:36,023] 382 DEBUG MainThread ssh.expect :: Output:
> Processor 0:No available space for 2M huge page allocation, max 2M pages: 26848
>
>
> FAIL 20180914 07:00:37 test_schedule_vm_mempage_config[small]
> This test failed b/c of the above test failure (Alarm was present, compute was in locked state)
>
>
> FAIL 20180914 07:57:34 test_kernel_module_signatures
> LOG
>
> [2018-09-14 07:57:37,946] 24 INFO MainThread test_kernel_modules.test_kernel_module_signatures::
> ====================== Test Step 1: Check for unassigned kernel modules on controller-0
> [2018-09-14 07:57:37,947] 419 DEBUG MainThread ssh.exec_cmd:: Executing command...
> [2018-09-14 07:57:37,947] 262 DEBUG MainThread ssh.send :: Send 'cat /proc/sys/kernel/tainted'
> [2018-09-14 07:57:38,050] 382 DEBUG MainThread ssh.expect :: Output:
> 12288
> controller-0:~$
> [2018-09-14 07:57:38,050] 262 DEBUG MainThread ssh.send :: Send 'echo $?'
> [2018-09-14 07:57:38,152] 382 DEBUG MainThread ssh.expect :: Output:
> 0
> controller-0:~$
> [2018-09-14 07:57:38,153] 32 ERROR MainThread test_kernel_modules.test_kernel_module_signatures:: Kernel module verification(s) failed on controller-0. Collecting more info
> [2018-09-14 07:57:38,153] 34 INFO MainThread test_kernel_modules.test_kernel_module_signatures::
> ====================== Test Step 2: Check kern.log for modules with failed verification
> [2018-09-14 07:57:38,153] 419 DEBUG MainThread ssh.exec_cmd:: Executing command...
> [2018-09-14 07:57:38,153] 262 DEBUG MainThread ssh.send :: Send 'grep --color=never -i "module verification failed" /var/log/kern.log'
> [2018-09-14 07:57:38,265] 382 DEBUG MainThread ssh.expect :: Output:
> 2018-09-13T19:09:51.212 localhost kernel: notice [ 6.271466] e1000e: module verification failed: signature and/or required key missing - tainting kernel
> 2018-09-13T19:51:42.174 controller-0 kernel: notice [ 6.133741] e1000e: module verification failed: signature and/or required key missing - tainting kernel
> 2018-09-14T03:33:29.054 controller-0 kernel: notice [ 6.183120] e1000e: module verification failed: signature and/or required key missing - tainting kernel
> controller-0:~$
> [2018-09-14 07:57:38,265] 262 DEBUG MainThread ssh.send :: Send 'echo $?'
> [2018-09-14 07:57:38,367] 382 DEBUG MainThread ssh.expect :: Output:
> 0
> controller-0:~$
> [2018-09-14 07:57:38,368] 43 INFO MainThread test_kernel_modules.test_kernel_module_signatures::
> ====================== Test Step 3: Display signing info for controller-0 failed kernel modules: ['e1000e']
> [2018-09-14 07:57:38,368] 419 DEBUG MainThread ssh.exec_cmd:: Executing command...
> [2018-09-14 07:57:38,368] 262 DEBUG MainThread ssh.send :: Send 'modinfo e1000e | grep --color=never -E "sig|filename"'
> [2018-09-14 07:57:38,474] 382 DEBUG MainThread ssh.expect :: Output:
> filename: /lib/modules/3.10.0-862.11.6.el7.36.tis.x86_64/weak-updates/e1000e/e1000e.ko
> signer: CentOS Linux kernel signing key
> sig_key: 58:7B:E7:83:22:7A:67:CA:49:54:D2:3B:57:AF:99:45:D3:AC:99:AC
> sig_hashalgo: sha256
> controller-0:~$
> [2018-09-14 07:57:38,475] 262 DEBUG MainThread ssh.send :: Send 'echo $?'
> [2018-09-14 07:57:38,577] 382 DEBUG MainThread ssh.expect :: Output:
>
>
>
> Thanks,
> Nimalini
>
> -----Original Message-----
> From: Xie, Cindy [mailto:cindy.xie at intel.com]
> Sent: Thursday, September 13, 2018 10:05 AM
> To: McKenna, Jason; Martinez Monroy, Elio; Waheed, Numan; Rasa, Nimalini; starlingx-discuss at lists.starlingx.io
> Cc: Khalil, Ghada; Lin, Shuicheng
> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
>
> Thanks Jason for the update. Appreciate you can have a new build generated according to Shuicheng's email earlier. @Nimalini, please report any finding back.
>
> The failure test cases in your both reports, is this the bug link? https://bugs.launchpad.net/starlingx/+bug/1791818
>
> Thanks. - cindy
>
> -----Original Message-----
> From: McKenna, Jason [mailto:Jason.McKenna at windriver.com]
> Sent: Thursday, September 13, 2018 9:35 PM
> To: Xie, Cindy <cindy.xie at intel.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>; Waheed, Numan <Numan.Waheed at windriver.com>; Rasa, Nimalini <Nimalini.Rasa at windriver.com>; starlingx-discuss at lists.starlingx.io
> Cc: Khalil, Ghada <Ghada.Khalil at windriver.com>; Lin, Shuicheng <shuicheng.lin at intel.com>
> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
>
> Hi Cindy,
>
> Will do. In the meantime, Nimalini (one of our validation gurus) was able to boot a build which I had done a couple of days ago, and perform sanity and our nightly regression suite. The results (attached) align with the 7.4 based master branch of StarlingX.
>
> -Jason
>
>> -----Original Message-----
>> From: Xie, Cindy <cindy.xie at intel.com>
>> Sent: September 13, 2018 2:44 AM
>> To: Martinez Monroy, Elio <elio.martinez.monroy at intel.com>; McKenna,
>> Jason <Jason.McKenna at windriver.com>; Waheed, Numan
>> <Numan.Waheed at windriver.com>; Rasa, Nimalini
>> <Nimalini.Rasa at windriver.com>; starlingx-discuss at lists.starlingx.io
>> Cc: Khalil, Ghada <Ghada.Khalil at windriver.com>; Lin, Shuicheng
>> <shuicheng.lin at intel.com>
>> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
>>
>> Elio,
>> We will provide you an updated ISO later today with the following:
>> - kernel upgrade to 3.10.0-862.11.6
>> - symbol issue fixed:
>> https://bugs.launchpad.net/starlingx/+bug/1792257
>>
>> Please help to do a sanity test based on this new ISO from Shuicheng on VE.
>>
>> @Jason,
>> Please re-generate a build using today's manifest with above fix.
>> Shuicheng shall be able to provide manifest with exact commitID.
>> @Numan, please abort your current testing w/ old ISO once you get a
>> new one from Jason and continue your sanity and regression on
>> bare-metal. I am not sure how long it takes, but please take priority
>> as we want merge back to mainline early next week.
>>
>> Thanks all for the support.
>> - cindy
>>
>> -----Original Message-----
>> From: McKenna, Jason [mailto:Jason.McKenna at windriver.com]
>> Sent: Thursday, September 13, 2018 4:24 AM
>> To: Lin, Shuicheng <shuicheng.lin at intel.com>; Cordoba Malibran, Erich
>> <erich.cordoba.malibran at intel.com>; Waheed, Numan
>> <Numan.Waheed at windriver.com>; Rasa, Nimalini
>> <Nimalini.Rasa at windriver.com>; starlingx-discuss at lists.starlingx.io;
>> Liu, ZhipengS <zhipengs.liu at intel.com>; Khalil, Ghada
>> <Ghada.Khalil at windriver.com>; Xie, Cindy <cindy.xie at intel.com>
>> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
>>
>>> -----Original Message-----
>>> From: Lin, Shuicheng <shuicheng.lin at intel.com>
>>> Sent: September 11, 2018 8:44 PM
>>> To: McKenna, Jason <Jason.McKenna at windriver.com>; Cordoba Malibran,
>>> Erich <erich.cordoba.malibran at intel.com>; Waheed, Numan
>>> <Numan.Waheed at windriver.com>; Rasa, Nimalini
>>> <Nimalini.Rasa at windriver.com>; starlingx-discuss at lists.starlingx.io;
>>> Liu, ZhipengS <zhipengs.liu at intel.com>; Khalil, Ghada
>>> <Ghada.Khalil at windriver.com>; Xie, Cindy <cindy.xie at intel.com>
>>> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
>>>
>>> Hi Jason,
>>> Here is the dhcp src rpm we are using:
>>> [slin14 at f998ec953c85 /]$ sudo -E yumdownloader -C --url --source
>>> dhcp
>>> http://vault.centos.org/7.5.1804/updates/Source/SPackages/dhcp-4.2.5
>>> -
>>> 68.el7.centos.1.src.rpm
>>>
>>> For centos-release, there are two packages in the mirror lst, 1 src
>>> rpm for build, and 1 rpm for the mock environment. And it has the
>>> "slightly difference" in the names as you mentioned.
>>> Both of them could be found in the centos repo with the exact name.
>>> ./rpms_centos.lst:135:centos-release-7-5.1804.1.el7.centos.src.rpm
>>> ./rpms_centos.lst:136:centos-release-7-5.1804.el7.centos.x86_64.rpm
>>> [slin14 at f998ec953c85 /]$ sudo -E yumdownloader -C --url --source
>>> centos- release-7-5.1804.1.el7.centos
>>> http://vault.centos.org/7.5.1804/updates/Source/SPackages/centos-
>>> release-7-5.1804.1.el7.centos.src.rpm
>>> [slin14 at f998ec953c85 /]$ sudo -E yumdownloader -C --url
>>> centos-release-7-
>>> 5.1804.el7.centos.x86_64
>>> http://mirror.centos.org/centos/7.5.1804/os/x86_64/Packages/centos-
>>> release-7-5.1804.el7.centos.x86_64.rpm
>>>
>>> If there is still some other rpm missing, you could try to find them
>>> from the repo.
>>> I have try my best to keep my local mirror the same as the lst. But
>>> it is still possible some package missed to be added to the repo lst.
>>> Please feel free to notify me or correct the error directly.
>>> Thanks.
>>
>> With the latest commits to stx-tools I am now able to download the
>> rpms and produce a build. I'll let you know how testing goes
>>
>>>
>>> Best Regards
>>> Shuicheng
>>>
>>>
>>> -----Original Message-----
>>> From: McKenna, Jason [mailto:Jason.McKenna at windriver.com]
>>> Sent: Wednesday, September 12, 2018 12:21 AM
>>> To: Cordoba Malibran, Erich <erich.cordoba.malibran at intel.com>;
>>> Waheed, Numan <Numan.Waheed at windriver.com>; Rasa, Nimalini
>>> <Nimalini.Rasa at windriver.com>; starlingx-discuss at lists.starlingx.io;
>>> Liu, ZhipengS <zhipengs.liu at intel.com>; Khalil, Ghada
>>> <Ghada.Khalil at windriver.com>; Lin, Shuicheng
>>> <shuicheng.lin at intel.com>; Xie, Cindy <cindy.xie at intel.com>
>>> Subject: RE: [Starlingx-discuss] CentOS 7.5 upgrade status update
>>>
>>> Hi Cindy,
>>>
>>> With the latest centos75 branch (and the cherry-picks you mentioned)
>>> I am unable to produce a build. There seems to be several source
>>> rpm files missing from the currently pushed version of the .lst
>>> files (or the .lst files pull down the wrong version). Most of the
>>> missing files I can download manually, however the checked in
>>> patches for dhcp and centos-release packages do not apply against
>>> the version of the source RPM which I can manually find on the
>>> Internet. For example, the version of centos-release called out in
>>> the .lst is centos-release-7-5.1804.el7.centos.x86_64.rpm, the
>>> version called out in the checked in code base is the slightly
>>> different
>>> centos-release-7- 5.1804.el7.centos.1.x86_64.rpm, and the patches
>> checked in do not apply against either package.
>>>
>>> Until the centos75 branch of the lst files pulls down the correct
>>> rpms, I will be unable to continue the validation effort.
>>>
>>> -J
>>>
>>>> -----Original Message-----
>>>> From: Cordoba Malibran, Erich <erich.cordoba.malibran at intel.com>
>>>> Sent: September 11, 2018 11:17 AM
>>>> To: Waheed, Numan <Numan.Waheed at windriver.com>; Rasa, Nimalini
>>>> <Nimalini.Rasa at windriver.com>;
>>>> starlingx-discuss at lists.starlingx.io;
>>>> Liu, ZhipengS <zhipengs.liu at intel.com>; Khalil, Ghada
>>>> <Ghada.Khalil at windriver.com>; Lin, Shuicheng
>>>> <shuicheng.lin at intel.com>; McKenna, Jason
>>>> <Jason.McKenna at windriver.com>; Xie, Cindy <cindy.xie at intel.com>
>>>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status update
>>>>
>>>> Hi
>>>>
>>>> You can add "debug loglevel=8" to the kernel boot option to get a
>>>> more detailed output of the crash (if any)
>>>>
>>>> -Erich
>>>>
>>>> On Tue, 2018-09-11 at 13:56 +0000, Xie, Cindy wrote:
>>>>> Shuicheng saw the symbol table error issue as well but it
>>>>> recovers in 8s, he is still working on debugging it. But 601204
>>>>> is just kernel upgrade but not intended to solve this issues.
>>>>> Are you running bare metal? I am wondering if this is HW config and driver issue.
>>>>>
>>>>> Is it possible that you try to deploy it in virtual environment?
>>>>>
>>>>> Thx. - cindy
>>>>>
>>>>> From: McKenna, Jason [mailto:Jason.McKenna at windriver.com]
>>>>> Sent: Tuesday, September 11, 2018 9:50 PM
>>>>> To: Xie, Cindy <cindy.xie at intel.com>; Rasa, Nimalini
>>>>> <Nimalini.Rasa at w indriver.com>; Waheed, Numan
>>>>> <Numan.Waheed at windriver.com>; Lin, Shuicheng
>>>>> <shuicheng.lin at intel.com>; Liu, ZhipengS <zhipengs.liu at inte
>>>>> l.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>;
>>>>> 'starlingx-discus s at lists.starlingx.io'
>>>>> <starlingx-discuss at lists.starlingx.io>
>>>>> Subject: RE: CentOS 7.5 upgrade status update
>>>>>
>>>>> Hi Cindy,
>>>>>
>>>>> We got past the installation device error, but are coming up
>>>>> against the symbol table error. Our system hangs after this
>>>>> message. I suspect a kernel issue. I see there is an open code
>>>>> review to refactor some kernel patches to a rebased kernel
>>>>> (https://review.open
>>>>> stack.org/#/c/601204/) but I do not know if that is expected to
>>>>> resolve the issue
>>>>>
>>>>> -J
>>>>>
>>>>> From: Xie, Cindy <cindy.xie at intel.com>
>>>>> Sent: September 11, 2018 9:32 AM
>>>>> To: Rasa, Nimalini <Nimalini.Rasa at windriver.com>; Waheed, Numan
>>>> <Numa
>>>>> n.Waheed at windriver.com>; Lin, Shuicheng
>> <shuicheng.lin at intel.com>;
>>>>> McKenna, Jason <Jason.McKenna at windriver.com>; Liu, ZhipengS
>>>> <zhipengs
>>>>> .liu at intel.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>;
>>>>> 'starlin gx-discuss at lists.starlingx.io'
>>>>> <starlingx-discuss at lists.starlingx.io>
>>>>> Subject: RE: CentOS 7.5 upgrade status update
>>>>>
>>>>> But as Jason reported yesterday, the build cannot be installed
>>>>> with error messages:
>>>>>
>>>>> Installation failed.
>>>>> ERROR: Specified installation (sda) or boot (sda) device is invalid.
>>>>>
>>>>> We cannot debug without more information, the ISO installs in
>>>>> Intel’s HW. Thus Shuicheng created a new build and attached a
>>>>> manifest in this email thread. Did WindRiver has a chance to
>>>>> create a new ISO using today’s manifest file and re-run the test?
>>>>>
>>>>> Thx. - cindy
>>>>>
>>>>> From: Rasa, Nimalini [mailto:Nimalini.Rasa at windriver.com]
>>>>> Sent: Tuesday, September 11, 2018 9:25 PM
>>>>> To: Xie, Cindy <cindy.xie at intel.com>; Waheed, Numan
>>>> <Numan.Waheed at win
>>>>> driver.com>; Lin, Shuicheng <shuicheng.lin at intel.com>; McKenna,
>>>>> Jason <Jason.McKenna at windriver.com>; Liu, ZhipengS
>>>> <zhipengs.liu at intel.com>
>>>>> ; Khalil, Ghada <Ghada.Khalil at windriver.com>;
>>>>> 'starlingx-discuss at list s.starlingx.io'
>>>>> <starlingx-discuss at lists.starlingx.io>
>>>>> Subject: RE: CentOS 7.5 upgrade status update
>>>>>
>>>>> This is on the same build (yesterday’s build).
>>>>>
>>>>> From: Xie, Cindy [mailto:cindy.xie at intel.com]
>>>>> Sent: Tuesday, September 11, 2018 9:22 AM
>>>>> To: Rasa, Nimalini; Waheed, Numan; Lin, Shuicheng; McKenna,
>>>>> Jason; Liu, ZhipengS; Khalil, Ghada; 'starlingx-discuss at lists.starlingx.io'
>>>>> Subject: RE: CentOS 7.5 upgrade status update
>>>>>
>>>>> Is this the same issue as yesterday’s build? or this is
>>>>> different issue now?
>>>>>
>>>>> From: Rasa, Nimalini [mailto:Nimalini.Rasa at windriver.com]
>>>>> Sent: Tuesday, September 11, 2018 9:06 PM
>>>>> To: Waheed, Numan <Numan.Waheed at windriver.com>; Lin,
>> Shuicheng
>>>> <shuic
>>>>> heng.lin at intel.com>; McKenna, Jason
>>> <Jason.McKenna at windriver.com>;
>>>>> Liu, ZhipengS <zhipengs.liu at intel.com>; Khalil, Ghada
>>>>> <Ghada.Khalil at w indriver.com>;
>>>>> 'starlingx-discuss at lists.starlingx.io' <starlingx-disc
>>>>> uss at lists.starlingx.io>; Xie, Cindy <cindy.xie at intel.com>
>>>>> Subject: RE: CentOS 7.5 upgrade status update
>>>>>
>>>>> It is not booting in my case. After the following , it just
>>>>> hangs
>>>>> there:
>>>>> “Error: no symbol table.
>>>>> Press any key to continue…”
>>>>>
>>>>> -Nimalini
>>>>>
>>>>> From: Waheed, Numan
>>>>> Sent: Tuesday, September 11, 2018 8:50 AM
>>>>> To: Lin, Shuicheng; McKenna, Jason; Liu, ZhipengS; Khalil,
>>>>> Ghada; 'st arlingx-discuss at lists.starlingx.io'; Xie, Cindy;
>>>>> Rasa, Nimalini
>>>>> Subject: RE: CentOS 7.5 upgrade status update
>>>>>
>>>>> Adding Nimalini
>>>>>
>>>>> From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
>>>>> Sent: September-11-18 2:17 AM
>>>>> To: McKenna, Jason; Liu, ZhipengS; Khalil, Ghada;
>>>>> 'starlingx-discuss@ lists.starlingx.io'; Xie, Cindy; Waheed,
>>>>> Numan
>>>>> Subject: RE: CentOS 7.5 upgrade status update
>>>>>
>>>>> Hi Jason,
>>>>> Just finished deploy image with latest code, I could config
>>>>> controller successfully in bare metal environment.
>>>>> Manifest is created for you reference. There are two patches I
>>>>> add
>>>>> manually:
>>>>> https://review.openstack.org/596855 for
>>>>> python-horizon from master
>>>>> https://review.openstack.org/599892 for build-iso
>>>>>
>>>>>
>>>>> There is an issue found when reboot controller node.
>>>>> After the OS selection menu, it will show below error. And it
>>>>> will auto continue boot after around 8 seconds. I will check the
>>>>> issue later.
>>>>> “
>>>>> Error: no symbol table.
>>>>> Press any key to continue…
>>>>> “
>>>>>
>>>>> Best Regards
>>>>> Shuicheng
>>>>>
>>>>> From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
>>>>> Sent: Tuesday, September 11, 2018 9:35 AM
>>>>> To: McKenna, Jason <Jason.McKenna at windriver.com>; Liu, ZhipengS
>>>> <zhip
>>>>> engs.liu at intel.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>;
>>>>> 'sta rlingx-discuss at lists.starlingx.io'
>>>>> <starlingx-discuss at lists.starlingx
>>>>> .io>; Xie, Cindy <cindy.xie at intel.com>; Waheed, Numan
>>>> <Numan.Waheed at w
>>>>> indriver.com>
>>>>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status
>>>>> update
>>>>>
>>>>> Hi Jason,
>>>>> I just tried my last Friday’s image at bare metal machine. There
>>>>> is no issue to install controller node.
>>>>> Will you help check what cause the install error in your environment?
>>>>>
>>>>> I am syncing my code to latest to double confirm it.
>>>>>
>>>>> Best Regards
>>>>> Shuicheng
>>>>>
>>>>> From: McKenna, Jason [mailto:Jason.McKenna at windriver.com]
>>>>> Sent: Tuesday, September 11, 2018 12:13 AM
>>>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>; Lin, Shuicheng
>>>>> <shuicheng .lin at intel.com>; Khalil, Ghada
>>>>> <Ghada.Khalil at windriver.com>; 'starlin
>>>>> gx-discuss at lists.starlingx.io'
>>>>> <starlingx-discuss at lists.starlingx.io>
>>>>> ; Xie, Cindy <cindy.xie at intel.com>; Waheed, Numan
>>>> <Numan.Waheed at windr
>>>>> iver.com>
>>>>> Subject: RE: CentOS 7.5 upgrade status update
>>>>>
>>>>> Status update. The build from the centos75 branch does not
>>>>> install on our hardware. The error message we are seeing is:
>>>>>
>>>>> Installation failed.
>>>>>
>>>>> ERROR: Specified installation (sda) or boot (sda) device is invalid.
>>>>>
>>>>> -Jason
>>>>>
>>>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>>>> Sent: September 8, 2018 9:56 AM
>>>>> To: Lin, Shuicheng <shuicheng.lin at intel.com>; McKenna, Jason
>>>>> <Jason.M cKenna at windriver.com>; Khalil, Ghada
>>>>> <Ghada.Khalil at windriver.com>; 's
>>>>> tarlingx-discuss at lists.starlingx.io'
>>>>> <starlingx-discuss at lists.starlin gx.io>; Xie, Cindy
>>>>> <cindy.xie at intel.com>; Waheed, Numan
>>>> <Numan.Waheed
>>>>> @windriver.com>
>>>>> Subject: RE: CentOS 7.5 upgrade status update
>>>>>
>>>>> Hi Jason,
>>>>>
>>>>> I have fixed the issue you mentioned below.
>>>>> - Furthermore, the tpm2-tools package was requesting version
>>>>> tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7).
>>>>> I hacked the spec file for tpm2-tools and rebuilt it to require
>>>>> the version mentioned, but there is some rework to be done there.
>>>>> [slin14]: I have the same issue also. Let’s further check it to
>>>>> find the root cause. I use rebuild tpm2-tools/tss2 to workaround
>>>>> it without any code.
>>>>>
>>>>> https://review.openstack.org/#/c/600920/
>>>>>
>>>>> Thanks!
>>>>> Zhipeng
>>>>> From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
>>>>> Sent: 2018年9月8日 21:14
>>>>> To: McKenna, Jason <Jason.McKenna at windriver.com>; Khalil, Ghada
>>>> <Ghad
>>>>> a.Khalil at windriver.com>; 'starlingx-discuss at lists.starlingx.io'
>>>>> <star lingx-discuss at lists.starlingx.io>; Xie, Cindy
>>>>> <cindy.xie at intel.com>; Waheed, Numan
>>> <Numan.Waheed at windriver.com>
>>>>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status
>>>>> update
>>>>>
>>>>> Hi Jason,
>>>>> I have fixed the error in the openstack-aodh. Patch is updated:
>>>>> https://review.openstack.org/#/c/598388/
>>>>>
>>>>> It is due to my local version is not the latest one.
>>>>> Sorry for the trouble caused, and thanks for your test.
>>>>>
>>>>> Best Regards
>>>>> Shuicheng
>>>>>
>>>>> From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
>>>>> Sent: Saturday, September 8, 2018 9:23 AM
>>>>> To: McKenna, Jason <Jason.McKenna at windriver.com>; Khalil, Ghada
>>>> <Ghad
>>>>> a.Khalil at windriver.com>; 'starlingx-discuss at lists.starlingx.io'
>>>>> <star lingx-discuss at lists.starlingx.io>; Xie, Cindy
>>>>> <cindy.xie at intel.com>; Waheed, Numan
>>> <Numan.Waheed at windriver.com>
>>>>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status
>>>>> update
>>>>>
>>>>> Hi Jason,
>>>>> Thanks very much for such detail feedback.
>>>>> I added some comments inline, and will check the issue in my
>>>>> environment also.
>>>>> Feel free to submit patch if you have to fix them. J
>>>>>
>>>>>
>>>>> Best Regards
>>>>> Shuicheng
>>>>>
>>>>> From: McKenna, Jason [mailto:Jason.McKenna at windriver.com]
>>>>> Sent: Saturday, September 8, 2018 3:47 AM
>>>>> To: McKenna, Jason <Jason.McKenna at windriver.com>; Khalil, Ghada
>>>> <Ghad
>>>>> a.Khalil at windriver.com>; Lin, Shuicheng
>>>>> <shuicheng.lin at intel.com>; 's tarlingx-discuss at lists.starlingx.io'
>>>>> <starlingx-discuss at lists.starlin gx.io>; Xie, Cindy
>>>>> <cindy.xie at intel.com>; Waheed, Numan
>>>> <Numan.Waheed
>>>>> @windriver.com>
>>>>> Subject: RE: CentOS 7.5 upgrade status update
>>>>>
>>>>> Hi Shuicheng,
>>>>>
>>>>> I’ve completed a build of the code. We have not yet run
>>>>> run-time tests, but here are the build-time issues I encountered:
>>>>>
>>>>> - The .lst file contains lighttpd-1.4.49, but this is no
>>>>> longer available for download, and the download_mirrors.sh step fails.
>>>>> It appears than CentOS has updated to lighttpd-1.4.50. I
>>>>> manually built a lighttpd-1.4.49-1.src.rpm to satisfy the .lst
>>>>> - Even though the .lst file specifies lighttpd-1.4.49, the
>>>>> code still references 1.4.45
>>>>> - At build-pkgs time (the build-srpms step), the meta-patches
>>>>> to openstack-aodh do not apply, as two of them are malformed. https:
>>>>> //review.openstack.org/#/c/598388/6/openstack/openstack-
>>>>> aodh/centos/meta_patches/0001-Modify-service-files-and-create-
>>>>> expirer-cron-script.patch -- In the new version, on line 190,
>>>>> the patch states that the old code covered 6 lines, and the new
>>>>> code covers 9 lines (-283,6 +286,9). But there are only two
>>>>> lines inserted by the patch. There is also a new blank line in
>>>>> the patch at line 194. Removing this blank line and editing the
>>>>> hunk description to only insert 2 lines resolves the issue for
>>>>> this hunk (there are other similar issues in this patch, and in
>>>>> 0001-meta-
>>>>> modify-aodh-api.patch)
>>>>> [slin14]: Let me double check it.
>>>>> - At binary build time, I had build failures on python-django-
>>>>> horizon and some magnum packages. I had to pull in
>>>>> https://review.op
>>>>> enstack.org/#/c/596855/2 (from the master branch, but not in the
>>>>> centos75 branch of stx-upstream) to get past this.
>>>>> [slin14]: It is due to I have not synced latest master code, so
>>>>> I don’t have this issue. I used last week’s master code.
>>>>> - After building all packages, I attempted to build the ISO.
>>>>> The tpm2-tss-1.3.0-2.el7.x86_64.rpm package were missing, and
>>>>> should be added to the .lst files.
>>>>> [slin14]: This package is already in the lst.
>>>>> - Furthermore, the tpm2-tools package was requesting version
>>>>> tpm2-tss-1.3.0-2.el7.centos (rather than tpm2-tss-1.3.0-2.el7).
>>>>> I hacked the spec file for tpm2-tools and rebuilt it to require
>>>>> the version mentioned, but there is some rework to be done there.
>>>>> [slin14]: I have the same issue also. Let’s further check it to
>>>>> find the root cause. I use rebuild tpm2-tools/tss2 to workaround
>>>>> it without any code.
>>>>>
>>>>> After these changes, I was able to produce a .iso
>>>>>
>>>>> -Jason
>>>>>
>>>>> From: McKenna, Jason <Jason.McKenna at windriver.com>
>>>>> Sent: September 7, 2018 10:08 AM
>>>>> To: Khalil, Ghada <Ghada.Khalil at windriver.com>; Lin, Shuicheng
>>>>> <shuic heng.lin at intel.com>; 'starlingx-discuss at lists.starlingx.io'
>>>>> <starling x-discuss at lists.starlingx.io>; Xie, Cindy
>>>>> <cindy.xie at intel.com>; Waheed, Numan
>>> <Numan.Waheed at windriver.com>
>>>>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status
>>>>> update
>>>>>
>>>>> Hi Shuicheng,
>>>>>
>>>>> As a download_mirrors.sh time issue, the version of lighttpd has
>>>>> been changed from 1.4.45 to 1.4.49. The
>>>>> lighttpd-1.4.49-1.el7.src.rpm file no longer exists on EPEL or
>>>>> CentOS servers, as they’ve moved to 1.4.50. I am attempting to
>>>>> locate an archived version of the 1.4.49 .src rpm.
>>>>>
>>>>> -Jason
>>>>>
>>>>> From: Khalil, Ghada <Ghada.Khalil at windriver.com>
>>>>> Sent: September 7, 2018 9:36 AM
>>>>> To: Lin, Shuicheng <shuicheng.lin at intel.com>;
>>>>> 'starlingx-discuss at list s.starlingx.io'
>>>>> <starlingx-discuss at lists.starlingx.io>; Xie, Cindy <c
>>>>> indy.xie at intel.com>; Waheed, Numan
>>> <Numan.Waheed at windriver.com>
>>>>> Subject: Re: [Starlingx-discuss] CentOS 7.5 upgrade status
>>>>> update
>>>>>
>>>>> Thanks Shuicheng. We’ll run some tests in Wind River. Numan will
>>>>> be priming this activity. He’ll share the results with the team
>>>>> sometime next week.
>>>>>
>>>>> Regards,
>>>>> Ghada
>>>>>
>>>>> From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
>>>>> Sent: Friday, September 07, 2018 1:32 AM
>>>>> To: 'starlingx-discuss at lists.starlingx.io'; Khalil, Ghada; Xie,
>>>>> Cindy
>>>>> Subject: CentOS 7.5 upgrade status update
>>>>>
>>>>> Hi all,
>>>>> Most task of CentOS7.5 upgrade in story 2003389 is done. Please
>>>>> go ahead to have a try and verify it.
>>>>> I expect there should be some issue due to we just did limited
>>>>> deploy test yet.
>>>>>
>>>>> Here is the detail status update of this week.
>>>>> For the total 44 srpm we plan to upgrade:
>>>>> 38 already merged in feature branch.
>>>>> 3 kept as original version. Anaconda upgrade is not needed.
>>>>> python- os-vif has higher version than 7.5. python-eventlet kept
>>>>> due to new src rpm itself have known issue.
>>>>> 2 under review with code ready. Puppet-haproxy and openstack-aodh.
>>>>> 1 under debug. Lighttpd.
>>>>>
>>>>> Besides srpm, we also upgraded 260+ rpm packages to 7.5 version.
>>>>>
>>>>> Here is the build instruction:
>>>>> 1. Switch to f/centos75 branch for stx-tools/stx-integ/stx-
>>>>> upstream/stx-root.
>>>>> Cherry-pick below un-merged patch:
>>>>> For puppet-haproxy:
>>>>> https://review.openstack.org/600369
>>>>> https://review.openstack.org/600359
>>>>> For openstack-aodh:
>>>>> https://review.openstack.org/598388
>>>>> https://review.openstack.org/600591
>>>>> For build-iso:
>>>>> https://review.openstack.org/599892
>>>>> For build-pkgs:
>>>>> https://review.openstack.org/599901
>>>>> https://review.openstack.org/600639
>>>>>
>>>>> 2. Run mirror script to get the upgraded src rpm and related
>>>>> rpm.
>>>>> 3. Try to build packages and ISO.
>>>>>
>>>>> For the detail status of CentOS7.5 upgrade, please check below story:
>>>>> https://storyboard.openstack.org/#!/story/2003389
>>>>>
>>>>> Feel free to contact me if you have trouble on it.
>>>>>
>>>>> Best Regards
>>>>> Shuicheng
>>>>> _______________________________________________
>>>>> Starlingx-discuss mailing list
>>>>> Starlingx-discuss at lists.starlingx.io
>>>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis
>>>>> cu
>>>>> ss
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
More information about the Starlingx-discuss
mailing list