[Starlingx-discuss] [ Test ] meeting notes - 03/31/2020
Agenda/meeting notes are posted here: https://etherpad.openstack.org/p/stx-test Agenda for 03/31/2020 Attendees: Yang, Mihail, Nicolae, George, Cosmin, Sharath, Claudiu, Bruce, ChrisW, Wendy 1. Sanity Status: · Sanity issues reported by WR: * 1855474 - OpenStack pods were not recovered after force reboot active controller * 1859645 - platform-integ-apps fail to apply initially due to could not find the requested resource (get configmaps) * https://bugs.launchpad.net/starlingx/+bug/1863795 - "helm list" failed by network is unreachable after host-unlock * https://bugs.launchpad.net/starlingx/+bug/1868567 - CMD application-delete failed by Timeout while waiting on RPC response * https://bugs.launchpad.net/starlingx/+bug/1869739 - 250.001 controller Configuration is out-of-date alarm not cleared after system installation · Sanity issues reported by Intel: * Green * Private registry used by Intel: some container images changed and not picked up in private registry - now resolved * May look into adding ipv6 configs for the long term 2. stx4.0 testing · Release plan: * https://docs.google.com/spreadsheets/d/1a93wt0XO0_JvajnPzQwnqFkXfdDysKVnHpbr... * Kata: intend to start early. target test complete by 05/08 * Rook: intend to start early. target test complete by 05/22 * Fault containerization: target test complete by 05/29 * Centos8: no server with TPM, mellanox cards; integrity may not be testable * TSN support in kata container: target test complete by 05/29 * OpenStack clients and platform services: eng build will be available in a week. testing: 04-13 - 05-04 * OpenStack upversion: § automated and manual test update needed, may be time consuming § target 06-05 * Layered build is tested daily with issues being found and resolved § image did not build - fixed § 03-31 build still failed, but went further · Targeted feature completion date: 06-08 * Risk: OpenStack upversion to Ussuris coming in late, and requires large amount of testing · Targeted regression completion date: 06-15 (pushing out 1 week) * Propose no openstack regression between 06-08 to 06-15 3. Unified santiy * Intel will bring in robot sanity as is into pytest framework * For feature test cases porting from robot framework in future, should look into organizing/porting by feature, and porting only the missing ones * Need to look into external traffic to/from openstack vm via neutron router in future to enable neutron testing in Intel environment 4. Opens · Servers/test environment setup within Nic's team: * In progress - cabling, configuring ports, etc * Will know more by end of week. * Risk reduced compared to last week.
Yang, I'm puzzled by the different Sanity results. Do you and Nic know why the Intel team is seeing Green Sanity runs, and the WR team is seeing multiple issues? Are we running different tests? Different images? Meanwhile, congratulations on pulling the Feature and Regression plan together. It's looking good, thanks to great work by you and the teams! bruej From: Liu, Yang (YOW) [mailto:yang.liu@windriver.com] Sent: Tuesday, March 31, 2020 9:09 AM To: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [ Test ] meeting notes - 03/31/2020 Agenda/meeting notes are posted here: https://etherpad.openstack.org/p/stx-test Agenda for 03/31/2020 Attendees: Yang, Mihail, Nicolae, George, Cosmin, Sharath, Claudiu, Bruce, ChrisW, Wendy 1. Sanity Status: * Sanity issues reported by WR: * 1855474 - OpenStack pods were not recovered after force reboot active controller * 1859645 - platform-integ-apps fail to apply initially due to could not find the requested resource (get configmaps) * https://bugs.launchpad.net/starlingx/+bug/1863795 - "helm list" failed by network is unreachable after host-unlock * https://bugs.launchpad.net/starlingx/+bug/1868567 - CMD application-delete failed by Timeout while waiting on RPC response * https://bugs.launchpad.net/starlingx/+bug/1869739 - 250.001 controller Configuration is out-of-date alarm not cleared after system installation * Sanity issues reported by Intel: * Green * Private registry used by Intel: some container images changed and not picked up in private registry - now resolved * May look into adding ipv6 configs for the long term 2. stx4.0 testing * Release plan: * https://docs.google.com/spreadsheets/d/1a93wt0XO0_JvajnPzQwnqFkXfdDysKVnHpbr... * Kata: intend to start early. target test complete by 05/08 * Rook: intend to start early. target test complete by 05/22 * Fault containerization: target test complete by 05/29 * Centos8: no server with TPM, mellanox cards; integrity may not be testable * TSN support in kata container: target test complete by 05/29 * OpenStack clients and platform services: eng build will be available in a week. testing: 04-13 - 05-04 * OpenStack upversion: ? automated and manual test update needed, may be time consuming ? target 06-05 * Layered build is tested daily with issues being found and resolved ? image did not build - fixed ? 03-31 build still failed, but went further * Targeted feature completion date: 06-08 * Risk: OpenStack upversion to Ussuris coming in late, and requires large amount of testing * Targeted regression completion date: 06-15 (pushing out 1 week) * Propose no openstack regression between 06-08 to 06-15 3. Unified santiy * Intel will bring in robot sanity as is into pytest framework * For feature test cases porting from robot framework in future, should look into organizing/porting by feature, and porting only the missing ones * Need to look into external traffic to/from openstack vm via neutron router in future to enable neutron testing in Intel environment 4. Opens * Servers/test environment setup within Nic's team: * In progress - cabling, configuring ports, etc * Will know more by end of week. * Risk reduced compared to last week.
Hi Bruce, Hi Yang, Looks like the bugs 1863795 and 1859645 are duplicates of 1856078. This was raised by Cristopher and each time we encountered, we've stated this in the report. Seems that we need an upgrade to Helm V3. For the rest of bugs, we need to replicate them after the daily finishes. I guess that Yang is executing also other sanity tests. Regards, Nicolae Jascanu, Ph.D. TSD Engineering Manager [intel-logo] Internet Of Things Group Galati, Romania From: Jones, Bruce E <bruce.e.jones@intel.com> Sent: Thursday, April 2, 2020 22:53 To: Liu, Yang (YOW) <yang.liu@windriver.com>; Jascanu, Nicolae <nicolae.jascanu@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: [ Test ] meeting notes - 03/31/2020 Yang, I'm puzzled by the different Sanity results. Do you and Nic know why the Intel team is seeing Green Sanity runs, and the WR team is seeing multiple issues? Are we running different tests? Different images? Meanwhile, congratulations on pulling the Feature and Regression plan together. It's looking good, thanks to great work by you and the teams! bruej From: Liu, Yang (YOW) [mailto:yang.liu@windriver.com] Sent: Tuesday, March 31, 2020 9:09 AM To: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: [Starlingx-discuss] [ Test ] meeting notes - 03/31/2020 Agenda/meeting notes are posted here: https://etherpad.openstack.org/p/stx-test Agenda for 03/31/2020 Attendees: Yang, Mihail, Nicolae, George, Cosmin, Sharath, Claudiu, Bruce, ChrisW, Wendy 1. Sanity Status: * Sanity issues reported by WR: * 1855474 - OpenStack pods were not recovered after force reboot active controller * 1859645 - platform-integ-apps fail to apply initially due to could not find the requested resource (get configmaps) * https://bugs.launchpad.net/starlingx/+bug/1863795 - "helm list" failed by network is unreachable after host-unlock * https://bugs.launchpad.net/starlingx/+bug/1868567 - CMD application-delete failed by Timeout while waiting on RPC response * https://bugs.launchpad.net/starlingx/+bug/1869739 - 250.001 controller Configuration is out-of-date alarm not cleared after system installation * Sanity issues reported by Intel: * Green * Private registry used by Intel: some container images changed and not picked up in private registry - now resolved * May look into adding ipv6 configs for the long term 2. stx4.0 testing * Release plan: * https://docs.google.com/spreadsheets/d/1a93wt0XO0_JvajnPzQwnqFkXfdDysKVnHpbr... * Kata: intend to start early. target test complete by 05/08 * Rook: intend to start early. target test complete by 05/22 * Fault containerization: target test complete by 05/29 * Centos8: no server with TPM, mellanox cards; integrity may not be testable * TSN support in kata container: target test complete by 05/29 * OpenStack clients and platform services: eng build will be available in a week. testing: 04-13 - 05-04 * OpenStack upversion: ? automated and manual test update needed, may be time consuming ? target 06-05 * Layered build is tested daily with issues being found and resolved ? image did not build - fixed ? 03-31 build still failed, but went further * Targeted feature completion date: 06-08 * Risk: OpenStack upversion to Ussuris coming in late, and requires large amount of testing * Targeted regression completion date: 06-15 (pushing out 1 week) * Propose no openstack regression between 06-08 to 06-15 3. Unified santiy * Intel will bring in robot sanity as is into pytest framework * For feature test cases porting from robot framework in future, should look into organizing/porting by feature, and porting only the missing ones * Need to look into external traffic to/from openstack vm via neutron router in future to enable neutron testing in Intel environment 4. Opens * Servers/test environment setup within Nic's team: * In progress - cabling, configuring ports, etc * Will know more by end of week. * Risk reduced compared to last week.
There was a comparison done by previous Intel test team, and on a high level, sanity test cases were mostly the same between us with 1 extra test in WR sanity suite. The main difference may be our configurations and verifications. There are couple of things I know of that might cause the differences in results: · OpenStack neutron configurations: o In our config, the external traffic goes through neutron routers to reach the vms, and connectivity verification is included in most openstack sanity test we run. Perhaps that is something Nic's team can look into. · Couple of the platform sanity issues we reported were more frequently seen on IPv6 systems (Christopher originally reported in a different LP). This is another item Nic can consider. · 1868567 is a stx-monitor test case we recently added to the sanity suite to test system applications, this test is available in stx-test. Nic could pick up this test if needed. · 1869739 is a configuration alarm issue without obvious system impact, this can only be found if the overall system health including alarms were verified explicitly in every test. Nic could also pick up this one easily because it's already included in pytest framework. Hope that helps. Regards, Yang From: Jascanu, Nicolae [mailto:nicolae.jascanu@intel.com] Sent: April-03-20 7:49 AM To: Jones, Bruce E; Liu, Yang (YOW) Cc: 'starlingx-discuss@lists.starlingx.io' Subject: RE: [ Test ] meeting notes - 03/31/2020 Hi Bruce, Hi Yang, Looks like the bugs 1863795 and 1859645 are duplicates of 1856078. This was raised by Cristopher and each time we encountered, we've stated this in the report. Seems that we need an upgrade to Helm V3. For the rest of bugs, we need to replicate them after the daily finishes. I guess that Yang is executing also other sanity tests. Regards, Nicolae Jascanu, Ph.D. TSD Engineering Manager [intel-logo] Internet Of Things Group Galati, Romania From: Jones, Bruce E <bruce.e.jones@intel.com> Sent: Thursday, April 2, 2020 22:53 To: Liu, Yang (YOW) <yang.liu@windriver.com>; Jascanu, Nicolae <nicolae.jascanu@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: [ Test ] meeting notes - 03/31/2020 Yang, I'm puzzled by the different Sanity results. Do you and Nic know why the Intel team is seeing Green Sanity runs, and the WR team is seeing multiple issues? Are we running different tests? Different images? Meanwhile, congratulations on pulling the Feature and Regression plan together. It's looking good, thanks to great work by you and the teams! bruej From: Liu, Yang (YOW) [mailto:yang.liu@windriver.com] Sent: Tuesday, March 31, 2020 9:09 AM To: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: [Starlingx-discuss] [ Test ] meeting notes - 03/31/2020 Agenda/meeting notes are posted here: https://etherpad.openstack.org/p/stx-test Agenda for 03/31/2020 Attendees: Yang, Mihail, Nicolae, George, Cosmin, Sharath, Claudiu, Bruce, ChrisW, Wendy 1. Sanity Status: * Sanity issues reported by WR: * 1855474 - OpenStack pods were not recovered after force reboot active controller * 1859645 - platform-integ-apps fail to apply initially due to could not find the requested resource (get configmaps) * https://bugs.launchpad.net/starlingx/+bug/1863795 - "helm list" failed by network is unreachable after host-unlock * https://bugs.launchpad.net/starlingx/+bug/1868567 - CMD application-delete failed by Timeout while waiting on RPC response * https://bugs.launchpad.net/starlingx/+bug/1869739 - 250.001 controller Configuration is out-of-date alarm not cleared after system installation * Sanity issues reported by Intel: * Green * Private registry used by Intel: some container images changed and not picked up in private registry - now resolved * May look into adding ipv6 configs for the long term 2. stx4.0 testing * Release plan: * https://docs.google.com/spreadsheets/d/1a93wt0XO0_JvajnPzQwnqFkXfdDysKVnHpbr... * Kata: intend to start early. target test complete by 05/08 * Rook: intend to start early. target test complete by 05/22 * Fault containerization: target test complete by 05/29 * Centos8: no server with TPM, mellanox cards; integrity may not be testable * TSN support in kata container: target test complete by 05/29 * OpenStack clients and platform services: eng build will be available in a week. testing: 04-13 - 05-04 * OpenStack upversion: * automated and manual test update needed, may be time consuming * target 06-05 * Layered build is tested daily with issues being found and resolved * image did not build - fixed * 03-31 build still failed, but went further * Targeted feature completion date: 06-08 * Risk: OpenStack upversion to Ussuris coming in late, and requires large amount of testing * Targeted regression completion date: 06-15 (pushing out 1 week) * Propose no openstack regression between 06-08 to 06-15 3. Unified santiy * Intel will bring in robot sanity as is into pytest framework * For feature test cases porting from robot framework in future, should look into organizing/porting by feature, and porting only the missing ones * Need to look into external traffic to/from openstack vm via neutron router in future to enable neutron testing in Intel environment 4. Opens * Servers/test environment setup within Nic's team: * In progress - cabling, configuring ports, etc * Will know more by end of week. * Risk reduced compared to last week.
Yang, Nic - thank you for the update on this. Can we please work toward one common Sanity test suite and setup? I know Nic's team is working on updating our automation to Pytest which I assume is the long pole - but I'd like to see us take on the goal of fully common test cases and setups for Sanity. Can we do that for 4.0? bruej From: Liu, Yang (YOW) [mailto:yang.liu@windriver.com] Sent: Friday, April 3, 2020 6:53 AM To: Jascanu, Nicolae <nicolae.jascanu@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: [ Test ] meeting notes - 03/31/2020 There was a comparison done by previous Intel test team, and on a high level, sanity test cases were mostly the same between us with 1 extra test in WR sanity suite. The main difference may be our configurations and verifications. There are couple of things I know of that might cause the differences in results: * OpenStack neutron configurations: o In our config, the external traffic goes through neutron routers to reach the vms, and connectivity verification is included in most openstack sanity test we run. Perhaps that is something Nic's team can look into. * Couple of the platform sanity issues we reported were more frequently seen on IPv6 systems (Christopher originally reported in a different LP). This is another item Nic can consider. * 1868567 is a stx-monitor test case we recently added to the sanity suite to test system applications, this test is available in stx-test. Nic could pick up this test if needed. * 1869739 is a configuration alarm issue without obvious system impact, this can only be found if the overall system health including alarms were verified explicitly in every test. Nic could also pick up this one easily because it's already included in pytest framework. Hope that helps. Regards, Yang From: Jascanu, Nicolae [mailto:nicolae.jascanu@intel.com] Sent: April-03-20 7:49 AM To: Jones, Bruce E; Liu, Yang (YOW) Cc: 'starlingx-discuss@lists.starlingx.io' Subject: RE: [ Test ] meeting notes - 03/31/2020 Hi Bruce, Hi Yang, Looks like the bugs 1863795 and 1859645 are duplicates of 1856078. This was raised by Cristopher and each time we encountered, we've stated this in the report. Seems that we need an upgrade to Helm V3. For the rest of bugs, we need to replicate them after the daily finishes. I guess that Yang is executing also other sanity tests. Regards, Nicolae Jascanu, Ph.D. TSD Engineering Manager [intel-logo] Internet Of Things Group Galati, Romania From: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>> Sent: Thursday, April 2, 2020 22:53 To: Liu, Yang (YOW) <yang.liu@windriver.com<mailto:yang.liu@windriver.com>>; Jascanu, Nicolae <nicolae.jascanu@intel.com<mailto:nicolae.jascanu@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: [ Test ] meeting notes - 03/31/2020 Yang, I'm puzzled by the different Sanity results. Do you and Nic know why the Intel team is seeing Green Sanity runs, and the WR team is seeing multiple issues? Are we running different tests? Different images? Meanwhile, congratulations on pulling the Feature and Regression plan together. It's looking good, thanks to great work by you and the teams! bruej From: Liu, Yang (YOW) [mailto:yang.liu@windriver.com] Sent: Tuesday, March 31, 2020 9:09 AM To: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: [Starlingx-discuss] [ Test ] meeting notes - 03/31/2020 Agenda/meeting notes are posted here: https://etherpad.openstack.org/p/stx-test Agenda for 03/31/2020 Attendees: Yang, Mihail, Nicolae, George, Cosmin, Sharath, Claudiu, Bruce, ChrisW, Wendy 1. Sanity Status: * Sanity issues reported by WR: * 1855474 - OpenStack pods were not recovered after force reboot active controller * 1859645 - platform-integ-apps fail to apply initially due to could not find the requested resource (get configmaps) * https://bugs.launchpad.net/starlingx/+bug/1863795 - "helm list" failed by network is unreachable after host-unlock * https://bugs.launchpad.net/starlingx/+bug/1868567 - CMD application-delete failed by Timeout while waiting on RPC response * https://bugs.launchpad.net/starlingx/+bug/1869739 - 250.001 controller Configuration is out-of-date alarm not cleared after system installation * Sanity issues reported by Intel: * Green * Private registry used by Intel: some container images changed and not picked up in private registry - now resolved * May look into adding ipv6 configs for the long term 2. stx4.0 testing * Release plan: * https://docs.google.com/spreadsheets/d/1a93wt0XO0_JvajnPzQwnqFkXfdDysKVnHpbr... * Kata: intend to start early. target test complete by 05/08 * Rook: intend to start early. target test complete by 05/22 * Fault containerization: target test complete by 05/29 * Centos8: no server with TPM, mellanox cards; integrity may not be testable * TSN support in kata container: target test complete by 05/29 * OpenStack clients and platform services: eng build will be available in a week. testing: 04-13 - 05-04 * OpenStack upversion: * automated and manual test update needed, may be time consuming * target 06-05 * Layered build is tested daily with issues being found and resolved * image did not build - fixed * 03-31 build still failed, but went further * Targeted feature completion date: 06-08 * Risk: OpenStack upversion to Ussuris coming in late, and requires large amount of testing * Targeted regression completion date: 06-15 (pushing out 1 week) * Propose no openstack regression between 06-08 to 06-15 3. Unified santiy * Intel will bring in robot sanity as is into pytest framework * For feature test cases porting from robot framework in future, should look into organizing/porting by feature, and porting only the missing ones * Need to look into external traffic to/from openstack vm via neutron router in future to enable neutron testing in Intel environment 4. Opens * Servers/test environment setup within Nic's team: * In progress - cabling, configuring ports, etc * Will know more by end of week. * Risk reduced compared to last week.
participants (3)
-
Jascanu, Nicolae
-
Jones, Bruce E
-
Liu, Yang (YOW)