Thanks guys. Btw, I ignored those "Digest" emails when I saw them, thought I had inadvertently enabled a digest & meant to go check later. Better, I think, to respond to original mail itself, rather than to the digest. So, you guys agree that 1918318 shouldn't block the release since it's a test script issue, and you're checking if the same is true for 1918420 - keep us posted. Thanks, Bill... -----Original Message----- From: Dimofte, Alexandru <alexandru.dimofte@intel.com> Sent: Tuesday, May 18, 2021 2:44 AM To: Sun, Austin <austin.sun@intel.com>; Zvonar, Bill <Bill.Zvonar@windriver.com>; Chen, Haochuan Z <haochuan.z.chen@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: RE: stx.5.0 Launchpads [Please note: This e-mail is from an EXTERNAL e-mail address] Hi guys, https://bugs.launchpad.net/starlingx/+bug/1918420 for this issue, latest comments from martin sent to mail list [1]. @Alex, would you like check if any test script can improve this? Alex: I will check this Austin. https://bugs.launchpad.net/starlingx/+bug/1918318 for this issue, Alex sent maillist about this issue, test scripts should be updated. I don’t think this issue will block release. Alex: Regarding this issue(which is a sporadic issue) I created 2 patches for our tests which unblock the rest of the test suite and I will monitor the sanity to see if this bug will be observed again. However I think this issue will not block the release. [1] http://lists.starlingx.io/pipermail/starlingx-discuss/2021-May/011417.html [2] http://lists.starlingx.io/pipermail/starlingx-discuss/2021-May/011438.html BR, Alex -----Original Message----- From: Sun, Austin <austin.sun@intel.com> Sent: Tuesday, May 18, 2021 4:01 AM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; Dimofte, Alexandru <alexandru.dimofte@intel.com>; Chen, Haochuan Z <haochuan.z.chen@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: RE: stx.5.0 Launchpads https://bugs.launchpad.net/starlingx/+bug/1918420 for this issue, latest comments from martin sent to mail list [1]. @Alex, would you like check if any test script can improve this? https://bugs.launchpad.net/starlingx/+bug/1918318 for this issue, Alex sent maillist about this issue, test scripts should be updated. I don’t think this issue will block release. [1] http://lists.starlingx.io/pipermail/starlingx-discuss/2021-May/011417.html [2] http://lists.starlingx.io/pipermail/starlingx-discuss/2021-May/011438.html Thanks. BR Austin Sun. -----Original Message----- From: Zvonar, Bill <Bill.Zvonar@windriver.com> Sent: Tuesday, May 18, 2021 2:21 AM To: Sun, Austin <austin.sun@intel.com>; Dimofte, Alexandru <alexandru.dimofte@intel.com>; Chen, Haochuan Z <haochuan.z.chen@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: RE: stx.5.0 Launchpads Hi guys - coming back to this now that the build issues have been resolved. Martin/Austin - can you provide an update on https://bugs.launchpad.net/starlingx/+bug/1918420 (it's still seen in the latest sanity). Austin - you've got the last comment on https://bugs.launchpad.net/starlingx/+bug/1918318 - do you think this still needs to hold up the release? Alex - can you update https://bugs.launchpad.net/starlingx/+bug/1928341 to confirm that it wasn't seen in the 5.0 sanity, then we'll remove the stx.5.0 tag. Bill... -----Original Message----- From: Zvonar, Bill <Bill.Zvonar@windriver.com> Sent: Thursday, May 13, 2021 7:19 AM To: Sun, Austin <austin.sun@intel.com>; Dimofte, Alexandru <alexandru.dimofte@intel.com>; Chen, Haochuan Z <haochuan.z.chen@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] stx.5.0 Launchpads Thanks Austin. Based on the discussion on the community call yesterday [0], we're holding the release for 2 issues: https://bugs.launchpad.net/starlingx/+bug/1918318 - Alex, please keep us posted on your test results, as mentioned in [1] https://bugs.launchpad.net/starlingx/+bug/1918420 - Martin/Austin, please let us know if you were able to determine whether this is a testcase issue or a real issue Thanks, Bill... [0] http://lists.starlingx.io/pipermail/starlingx-discuss/2021-May/011430.html [1] http://lists.starlingx.io/pipermail/starlingx-discuss/2021-May/011438.html -----Original Message----- From: Sun, Austin <austin.sun@intel.com> Sent: Tuesday, May 11, 2021 9:21 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; starlingx-discuss@lists.starlingx.io Subject: RE: stx.5.0 Launchpads [Please note: This e-mail is from an EXTERNAL e-mail address] Hi Bill: Some comments inline. We can discuss detail in community meeting. Thanks. BR Austin Sun -----Original Message----- From: Zvonar, Bill <Bill.Zvonar@windriver.com> Sent: Wednesday, May 12, 2021 12:29 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] stx.5.0 Launchpads Hi all - we have 3 open High Importance launchpads for stx.5.0 - 2 have reviews up & one needs input from Alex. I'm not sure if we should be considering 1918420 to be high importance as well, but I included it below - it has a review up as well. Bill... Two High importance bugs have reviews up, one needs more info: - 1922299 (Martin): Openstack-Helm mariadb crash recovery failed - https://bugs.launchpad.net/starlingx/+bug/1922299 - Martin seems to have fixed it, needs his review to merge: https://review.opendev.org/c/starlingx/openstack-armada-app/+/790321 [Austin] After analysis , This issue only some errors in logs , the openstack function is not impacted, Martin's fix will eliminate this logs. Suggest downgrade the to Medium. - 1925668 (Zhipeng): Bootstrap replay fails when changing mgmt subnet - https://bugs.launchpad.net/starlingx/+bug/1925668 - fix up for review https://review.opendev.org/c/starlingx/ansible-playbooks/+/787942 - Zhipeng addressed latest comments, needs to be reviewed [Austin] Zhipeng has uploaded new patchset , waiting new review. - 1918318 (Martin): controller-0 is degraded due to the failure of its 'pci-irq-affinity-agent' process. Auto recovery of this major process is in progress." - https://bugs.launchpad.net/starlingx/+bug/1918318 - Alex to provide info on the frequency of this intermittent issue [Austin] 0507/0510/0511 not see this issue. 0505 see this issue. And this one is Medium, not High, but gets a mention in the sanity report - we - 1918420 (Martin): Alarm 750.003 - Application remove failure for stx-openstack - not high, but mentioned in stx.5.0 sanity report - https://bugs.launchpad.net/starlingx/+bug/1918420 - review is up: https://review.opendev.org/c/starlingx/ha/+/788897 [1] https://etherpad.opendev.org/p/stx-gating-bugs -----Original Message----- From: Zvonar, Bill Sent: Monday, April 12, 2021 8:23 AM To: starlingx-discuss@lists.starlingx.io Subject: RE: stx.5.0 Launchpads Hi all, as of now we have 5 high importance bugs tagged against stx.5.0 - 2 new ones came in recently. From https://etherpad.opendev.org/p/stx-gating-bugs: Build 1891416 (Davlet): stx-platformclients docker image build doesn't include the correct content https://bugs.launchpad.net/starlingx/+bug/1891416 fix in progress Security 1921511 (Zhipeng): kubernetes-ca cert/key are replaced by etcd-ca cert/key https://bugs.launchpad.net/starlingx/+bug/1921511 fix is under review https://review.opendev.org/c/starlingx/ansible-playbooks/+/784006 Update 1913455 (Adriano): Fail to swact to controller-1 after upgrade https://bugs.launchpad.net/starlingx/+bug/1913455 testing is underway, going well so far Dist Cloud 19230793 (Tee): Distributed Cloud: Remote install failure during orchestrated subcloud upgrade due to non-encoded bmc password https://bugs.launchpad.net/starlingx/+bug/1923073 new as of Friday Helm 1923185 (Mihnea): Restore fails when launching Armada with Helm v3 https://bugs.launchpad.net/starlingx/+bug/1923185 new as of Friday -----Original Message----- From: Zvonar, Bill Sent: Wednesday, March 31, 2021 12:28 PM To: starlingx-discuss@lists.starlingx.io Subject: stx.5.0 Launchpads Hi all, As discussed in the Community & Release Team calls, as we near the end of stx.5.0, we need to focus on fixing stx.5.0 launchpads Per the stx.5.0 release plan [0], we have 2 weeks until RC1 (4/14) and just over a month until the forecast release date (5/5). Generally, we want to have fixes for all known issues in place by RC1, so let's say we have 2 weeks. And this means all Critical & High importance Launchpads, and as many Medium importance Launchpads as possible. We currently have 1 Critical, 5 High & ~33 Medium, as of a snapshot this morning. The full list of stx.5.0 Launchpads is at [2]. So, let's put focus on fixing the bugs in our respective domains. I created an etherpad [1] with a snapshot of the bugs per domain & copied that list below. Bill... [0] https://docs.google.com/spreadsheets/d/1JbOQELqXG_GDoP1jo6YoRDytpTkJEs89TVim... [1] https://etherpad.opendev.org/p/stx-gating-bugs [2] https://bugs.launchpad.net/starlingx/+bugs?field.tag=stx.5.0 *Importance = Critical OpenStack 1917308: Stx-openstack apply-fail after swact standby controller, lock, unlock standby controller https://bugs.launchpad.net/starlingx/+bug/1917308 *Importance = High Apps 1917352: stx-sdo apply fails to pull images from registry.local https://bugs.launchpad.net/starlingx/+bug/1917352 Build 1891416: stx-platformclients docker image build doesn't include the correct content https://bugs.launchpad.net/starlingx/+bug/1891416 Config 1918943: Regression - Bootstrap replay following a cluster_host_subnet reconfiguration fails https://bugs.launchpad.net/starlingx/+bug/1918943 Security 1921511: kubernetes-ca cert/key are replaced by etcd-ca cert/key https://bugs.launchpad.net/starlingx/+bug/1921511 Update 1913455: Fail to swact to controller-1 after upgrade https://bugs.launchpad.net/starlingx/+bug/1913455 *Importance = Medium Apps 1888900: System remove,apply fails to start the vault pods with TLS handshake error https://bugs.launchpad.net/starlingx/+bug/1888900 Apps, Containers 1882485: cert-manager application in apply-failed status since initial controller-1 unlock https://bugs.launchpad.net/starlingx/+bug/1882485 1883555: applying custom system application on controller-1 failed https://bugs.launchpad.net/starlingx/+bug/1883555 Build 1904893: clarity of Layered Build Reference in StarlingX https://bugs.launchpad.net/starlingx/+bug/1904893 1921149: centos7: Python < 2.7.9 not supported by PyPI https://bugs.launchpad.net/starlingx/+bug/1921149 Config 1884335: DC: Worker node went for second reboot after all nodes power down/up https://bugs.launchpad.net/starlingx/+bug/1884335 1889101: 100.114 alarm "NTP cannot reach external time source; syncing with peer controller only " raised when other external NTP servers are available https://bugs.launchpad.net/starlingx/+bug/1889101 1890970: controller unable to unlock after reinstall (inv_state stuck at reinstalling) https://bugs.launchpad.net/starlingx/+bug/1890970 1895555: OAM IP change needs double lock/unlock controllers for IPV6 system https://bugs.launchpad.net/starlingx/+bug/1895555 1918139: On AIO hosts, kuberenetes is starting before key resources are initialized https://bugs.launchpad.net/starlingx/+bug/1918139 Containers 1883791: DC subcloud bootsrap failed - nginx-ingress-controller apply failed https://bugs.launchpad.net/starlingx/+bug/1883791 1883953: Cert-manager app failed to apply after controller-1 of the system controller is upgraded https://bugs.launchpad.net/starlingx/+bug/1883953 1887105: Using python helm plugin to dynamically override helmchart will cause an issue https://bugs.launchpad.net/starlingx/+bug/1887105 1887438: Controller-0 Not Ready after force rebooting active controller (Controller-1) https://bugs.launchpad.net/starlingx/+bug/1887438 1901911: platform-integration-manifest apply failed due to error Install release: stx-rbd-provisioner https://bugs.launchpad.net/starlingx/+bug/1901911 1903090: DC: helm operation failure during merging helm overrides https://bugs.launchpad.net/starlingx/+bug/1903090 Containers, DC 1902266: Failed scheduling of armada pod resulting in subcloud bootstrap failure https://bugs.launchpad.net/starlingx/+bug/1902266 DC, GUI 1886536: Horizon: error occurred when attempt to download subcloud openrc file https://bugs.launchpad.net/starlingx/+bug/1886536 GUI 1890503: starlingx gui dc_admin deploy status field needs updating https://bugs.launchpad.net/starlingx/+bug/1890503 HA 1893669: swact is not triggered after killing dnsmasq process within 90 seconds https://bugs.launchpad.net/starlingx/+bug/1893669 Networking 1911213: Global network policy not found in default namespace https://bugs.launchpad.net/starlingx/+bug/1911213 OpenStack 1888178: Neturon vm in error status after resize revert due to "Virtual Interface creation failed" https://bugs.launchpad.net/starlingx/+bug/1888178 1904729: Deprecate vcpu_pin_set for stx-openstack in favor of new configs https://bugs.launchpad.net/starlingx/+bug/1904729 Storage 1883975: Disk partitions not aligned https://bugs.launchpad.net/starlingx/+bug/1883975 1884136: file system modify give error and modifies fs https://bugs.launchpad.net/starlingx/+bug/1884136 1884184: Storage node configuration failed after performing DOR https://bugs.launchpad.net/starlingx/+bug/1884184 1890293: ceph-osds are down after swact operation generating a coredump https://bugs.launchpad.net/starlingx/+bug/1890293 1892907: Ceph storage condition alarm was not cleared after lock and unlock controller and swact https://bugs.launchpad.net/starlingx/+bug/1892907 1896628: rook-ceph-apps upload failed to find an application tarball https://bugs.launchpad.net/starlingx/+bug/1896628 Tools 1886868: Need detailed resource cpu and memory info in collect https://bugs.launchpad.net/starlingx/+bug/1886868 1911935: collect is missing etcdctl output https://bugs.launchpad.net/starlingx/+bug/1911935 Update 1894050: AIO-DX: Backup and Restore: After Reinstall controller-1, auto apply cert-manager and platform-integration failed https://bugs.launchpad.net/starlingx/+bug/1894050 1913465: AIODX upgrade：cgts-vg partition space is decreased on controller-0 https://bugs.launchpad.net/starlingx/+bug/1913465 _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss