Hi all - we have 3 open High Importance launchpads for stx.5.0 - 2 have reviews up & one needs input from Alex. I'm not sure if we should be considering 1918420 to be high importance as well, but I included it below - it has a review up as well. Bill... Two High importance bugs have reviews up, one needs more info: - 1922299 (Martin): Openstack-Helm mariadb crash recovery failed - https://bugs.launchpad.net/starlingx/+bug/1922299 - Martin seems to have fixed it, needs his review to merge: https://review.opendev.org/c/starlingx/openstack-armada-app/+/790321 - 1925668 (Zhipeng): Bootstrap replay fails when changing mgmt subnet - https://bugs.launchpad.net/starlingx/+bug/1925668 - fix up for review https://review.opendev.org/c/starlingx/ansible-playbooks/+/787942 - Zhipeng addressed latest comments, needs to be reviewed - 1918318 (Martin): controller-0 is degraded due to the failure of its 'pci-irq-affinity-agent' process. Auto recovery of this major process is in progress." - https://bugs.launchpad.net/starlingx/+bug/1918318 - Alex to provide info on the frequency of this intermittent issue And this one is Medium, not High, but gets a mention in the sanity report - we - 1918420 (Martin): Alarm 750.003 - Application remove failure for stx-openstack - not high, but mentioned in stx.5.0 sanity report - https://bugs.launchpad.net/starlingx/+bug/1918420 - review is up: https://review.opendev.org/c/starlingx/ha/+/788897 [1] https://etherpad.opendev.org/p/stx-gating-bugs -----Original Message----- From: Zvonar, Bill Sent: Monday, April 12, 2021 8:23 AM To: starlingx-discuss@lists.starlingx.io Subject: RE: stx.5.0 Launchpads Hi all, as of now we have 5 high importance bugs tagged against stx.5.0 - 2 new ones came in recently. From https://etherpad.opendev.org/p/stx-gating-bugs: Build 1891416 (Davlet): stx-platformclients docker image build doesn't include the correct content https://bugs.launchpad.net/starlingx/+bug/1891416 fix in progress Security 1921511 (Zhipeng): kubernetes-ca cert/key are replaced by etcd-ca cert/key https://bugs.launchpad.net/starlingx/+bug/1921511 fix is under review https://review.opendev.org/c/starlingx/ansible-playbooks/+/784006 Update 1913455 (Adriano): Fail to swact to controller-1 after upgrade https://bugs.launchpad.net/starlingx/+bug/1913455 testing is underway, going well so far Dist Cloud 19230793 (Tee): Distributed Cloud: Remote install failure during orchestrated subcloud upgrade due to non-encoded bmc password https://bugs.launchpad.net/starlingx/+bug/1923073 new as of Friday Helm 1923185 (Mihnea): Restore fails when launching Armada with Helm v3 https://bugs.launchpad.net/starlingx/+bug/1923185 new as of Friday -----Original Message----- From: Zvonar, Bill Sent: Wednesday, March 31, 2021 12:28 PM To: starlingx-discuss@lists.starlingx.io Subject: stx.5.0 Launchpads Hi all, As discussed in the Community & Release Team calls, as we near the end of stx.5.0, we need to focus on fixing stx.5.0 launchpads Per the stx.5.0 release plan [0], we have 2 weeks until RC1 (4/14) and just over a month until the forecast release date (5/5). Generally, we want to have fixes for all known issues in place by RC1, so let's say we have 2 weeks. And this means all Critical & High importance Launchpads, and as many Medium importance Launchpads as possible. We currently have 1 Critical, 5 High & ~33 Medium, as of a snapshot this morning. The full list of stx.5.0 Launchpads is at [2]. So, let's put focus on fixing the bugs in our respective domains. I created an etherpad [1] with a snapshot of the bugs per domain & copied that list below. Bill... [0] https://docs.google.com/spreadsheets/d/1JbOQELqXG_GDoP1jo6YoRDytpTkJEs89TVim... [1] https://etherpad.opendev.org/p/stx-gating-bugs [2] https://bugs.launchpad.net/starlingx/+bugs?field.tag=stx.5.0 *Importance = Critical OpenStack 1917308: Stx-openstack apply-fail after swact standby controller, lock, unlock standby controller https://bugs.launchpad.net/starlingx/+bug/1917308 *Importance = High Apps 1917352: stx-sdo apply fails to pull images from registry.local https://bugs.launchpad.net/starlingx/+bug/1917352 Build 1891416: stx-platformclients docker image build doesn't include the correct content https://bugs.launchpad.net/starlingx/+bug/1891416 Config 1918943: Regression - Bootstrap replay following a cluster_host_subnet reconfiguration fails https://bugs.launchpad.net/starlingx/+bug/1918943 Security 1921511: kubernetes-ca cert/key are replaced by etcd-ca cert/key https://bugs.launchpad.net/starlingx/+bug/1921511 Update 1913455: Fail to swact to controller-1 after upgrade https://bugs.launchpad.net/starlingx/+bug/1913455 *Importance = Medium Apps 1888900: System remove,apply fails to start the vault pods with TLS handshake error https://bugs.launchpad.net/starlingx/+bug/1888900 Apps, Containers 1882485: cert-manager application in apply-failed status since initial controller-1 unlock https://bugs.launchpad.net/starlingx/+bug/1882485 1883555: applying custom system application on controller-1 failed https://bugs.launchpad.net/starlingx/+bug/1883555 Build 1904893: clarity of Layered Build Reference in StarlingX https://bugs.launchpad.net/starlingx/+bug/1904893 1921149: centos7: Python < 2.7.9 not supported by PyPI https://bugs.launchpad.net/starlingx/+bug/1921149 Config 1884335: DC: Worker node went for second reboot after all nodes power down/up https://bugs.launchpad.net/starlingx/+bug/1884335 1889101: 100.114 alarm "NTP cannot reach external time source; syncing with peer controller only " raised when other external NTP servers are available https://bugs.launchpad.net/starlingx/+bug/1889101 1890970: controller unable to unlock after reinstall (inv_state stuck at reinstalling) https://bugs.launchpad.net/starlingx/+bug/1890970 1895555: OAM IP change needs double lock/unlock controllers for IPV6 system https://bugs.launchpad.net/starlingx/+bug/1895555 1918139: On AIO hosts, kuberenetes is starting before key resources are initialized https://bugs.launchpad.net/starlingx/+bug/1918139 Containers 1883791: DC subcloud bootsrap failed - nginx-ingress-controller apply failed https://bugs.launchpad.net/starlingx/+bug/1883791 1883953: Cert-manager app failed to apply after controller-1 of the system controller is upgraded https://bugs.launchpad.net/starlingx/+bug/1883953 1887105: Using python helm plugin to dynamically override helmchart will cause an issue https://bugs.launchpad.net/starlingx/+bug/1887105 1887438: Controller-0 Not Ready after force rebooting active controller (Controller-1) https://bugs.launchpad.net/starlingx/+bug/1887438 1901911: platform-integration-manifest apply failed due to error Install release: stx-rbd-provisioner https://bugs.launchpad.net/starlingx/+bug/1901911 1903090: DC: helm operation failure during merging helm overrides https://bugs.launchpad.net/starlingx/+bug/1903090 Containers, DC 1902266: Failed scheduling of armada pod resulting in subcloud bootstrap failure https://bugs.launchpad.net/starlingx/+bug/1902266 DC, GUI 1886536: Horizon: error occurred when attempt to download subcloud openrc file https://bugs.launchpad.net/starlingx/+bug/1886536 GUI 1890503: starlingx gui dc_admin deploy status field needs updating https://bugs.launchpad.net/starlingx/+bug/1890503 HA 1893669: swact is not triggered after killing dnsmasq process within 90 seconds https://bugs.launchpad.net/starlingx/+bug/1893669 Networking 1911213: Global network policy not found in default namespace https://bugs.launchpad.net/starlingx/+bug/1911213 OpenStack 1888178: Neturon vm in error status after resize revert due to "Virtual Interface creation failed" https://bugs.launchpad.net/starlingx/+bug/1888178 1904729: Deprecate vcpu_pin_set for stx-openstack in favor of new configs https://bugs.launchpad.net/starlingx/+bug/1904729 Storage 1883975: Disk partitions not aligned https://bugs.launchpad.net/starlingx/+bug/1883975 1884136: file system modify give error and modifies fs https://bugs.launchpad.net/starlingx/+bug/1884136 1884184: Storage node configuration failed after performing DOR https://bugs.launchpad.net/starlingx/+bug/1884184 1890293: ceph-osds are down after swact operation generating a coredump https://bugs.launchpad.net/starlingx/+bug/1890293 1892907: Ceph storage condition alarm was not cleared after lock and unlock controller and swact https://bugs.launchpad.net/starlingx/+bug/1892907 1896628: rook-ceph-apps upload failed to find an application tarball https://bugs.launchpad.net/starlingx/+bug/1896628 Tools 1886868: Need detailed resource cpu and memory info in collect https://bugs.launchpad.net/starlingx/+bug/1886868 1911935: collect is missing etcdctl output https://bugs.launchpad.net/starlingx/+bug/1911935 Update 1894050: AIO-DX: Backup and Restore: After Reinstall controller-1, auto apply cert-manager and platform-integration failed https://bugs.launchpad.net/starlingx/+bug/1894050 1913465: AIODX upgrade：cgts-vg partition space is decreased on controller-0 https://bugs.launchpad.net/starlingx/+bug/1913465