* Test strategy review (Ian) * [brucej] We have not had a passing Sanity for some time. There are numerous issues. Many thanks to the folks debugging them. Can we / should we take additional steps to get back to passing Sanity? * The Release team recommends that we limit code checkins to bug fixes until we have passing Sanity in both test labs. * What work would this impact? We have quite a bit of work left on the Container feature(s) - the target for code merge is June 6th three weeks from now. Delaying those checkins. OVS update in a container is also in flight. Ansible work is in except for SDK work which can be delayed. OVS Firewall has merged and testing in progress. * Let's tag the issues impacting Sanity with stx.sanity. Ghada to tag the issues below. Test teams to tag them as new issues are found. * Do we want to allow check-in of non-Sanity bugs or limit check-ins to only Sanity impacting features. * Hoping to see a Yellow Sanity today and a Green(ish) Sanity on Friday. * For this week please limit check-ins to Sanity tagged issues or Critical/High defects. ? If a team needs a exception to this short term policy, feel free to raise the issue on the mailing list. Release team can grant approval. * (Ghada) Summary of sanity/critical issues: * Ceph api synchronization issue: Application upload / application apply hanging ? https://bugs.launchpad.net/starlingx/+bug/1827521 ? https://bugs.launchpad.net/starlingx/+bug/1828899 ? https://bugs.launchpad.net/starlingx/+bug/1828059 ? A ceph upstream fix was identified and merged in starlingx on May 14. PR: https://github.com/starlingx-staging/stx-ceph/pull/30 ? The following two LPs are also suspected to be the same root-cause: (needs review/retest): * https://bugs.launchpad.net/starlingx/+bug/1828729 * https://bugs.launchpad.net/starlingx/+bug/1828863 * rbd provisioner preventing application apply ? https://bugs.launchpad.net/starlingx/+bug/1828896 ? Review is in progress. Workaround documented in the LP. * Issues with pci-nova-irq-agent - two issues reported affecting sanity ? controller in degraded state: https://bugs.launchpad.net/starlingx/+bug/1828877 * Zhipeng is working on fix ? nodes failing after unlock: https://bugs.launchpad.net/starlingx/+bug/1828903 * May be an issue with when the stx build was done by the reporter. Reporter needs to retest. * Issues w/ nova ? A recent commit back-ported to nova stein is causing issues with VM launches * nova bug opened by Gerry Kopec: https://bugs.launchpad.net/nova/+bug/1829062 * The issue was also reported by the test team via: https://bugs.launchpad.net/starlingx/+bug/1829060 o This stx bug will be used to point the stx docker image builds to the stx nova fork (since it has an older snapshot of the stein branch). o The plan is for the stx nova fork to be rebased at a regular cadence (frequency?). The stx.disto.openstack team will need to monitor the nova bug above to ensure a fix is available before the next rebase of the fork. * Any other issues? * [brucej] The distro.openstack weekly call has not had many participants. Is there a conflict or some other issue? Likely due to vacations.