Hi Ghada, Daniel, I added additional details to the bug. Please, let me know if you need additional details. Thanks & Regards From: "Khalil, Ghada" <Ghada.Khalil@windriver.com> Date: Thursday, May 16, 2019 at 3:11 PM To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Cc: "Qi, Mingyuan" <mingyuan.qi@intel.com>, "Badea, Daniel" <Daniel.Badea@windriver.com>, "Church, Robert" <Robert.Church@windriver.com> Subject: RE: [Starlingx-discuss] UPDATE: Limited code merges until sanity results improve Thanks Christopher. Can you please add a note to the bug as well with more details: load used, steps if there is any variation from the original report, etc? Daniel, Please start triaging the new logs. Thanks, Ghada From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras@intel.com] Sent: Thursday, May 16, 2019 3:43 PM To: Khalil, Ghada; starlingx-discuss@lists.starlingx.io Cc: Qi, Mingyuan; Badea, Daniel; Church, Robert Subject: Re: [Starlingx-discuss] UPDATE: Limited code merges until sanity results improve Hi Ghada, I uploaded a collect from a Standard External storage (2+2+2) to https://bugs.launchpad.net/starlingx/+bug/1828059 . We got the error on all 4 configs. Thanks & Regards, Cristopher Lemus From: "Khalil, Ghada" <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Thursday, May 16, 2019 at 7:39 AM To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: "Qi, Mingyuan" <mingyuan.qi@intel.com<mailto:mingyuan.qi@intel.com>>, "Badea, Daniel" <Daniel.Badea@windriver.com<mailto:Daniel.Badea@windriver.com>>, "Church, Robert" <Robert.Church@windriver.com<mailto:Robert.Church@windriver.com>> Subject: RE: [Starlingx-discuss] UPDATE: Limited code merges until sanity results improve Thanks Christopher for the update. I appreciate you trying the new build so promptly. I’ve assigned https://bugs.launchpad.net/starlingx/+bug/1829299 to Mingyuan for investigation as this looks specific to the Intel setup. Regarding the cinder volume issue, are you referring to https://bugs.launchpad.net/starlingx/+bug/1828059 ? If so, can you please do a collect and add it to the launchpad so that Daniel and the storage team can investigate. Thanks, Ghada From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras@intel.com] Sent: Thursday, May 16, 2019 1:11 AM To: Khalil, Ghada; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] UPDATE: Limited code merges until sanity results improve Used CENGN image that was created with the listed patches on previous email: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190515T220... Automation for our lab is still broken (local mirror registry dependency): https://bugs.launchpad.net/starlingx/+bug/1829299 However, by manually applying the workaround that is documented on the same launchpad, I was able to do a full provisioning on Baremetal Simplex, Duplex and Standard (2+2). About sanity tests, at least against duplex, the issue with the Volumes is still present: Launch Instances :: Launch Cirros instances. | FAIL | Invalid volume: Invalid input received: Invalid volume: Volume attachments can not be created if the volume is in an error state. The Volume f80e9811-a45e-4646-9634-56122eee813e currently has a status of: error (HTTP 400) (Request-ID: req-31db1545-9a79-4ed9-b3df-dd8d5d9d5c60) (HTTP 400) (Request-ID: req-d4bbb73f-207f-4365-9f5a-4f0625d2c5d0) Other tests that are not depending on “Volumes” are passing. Sanity is currently running against simplex and standard, will update tomorrow morning, and, as soon as possible, a formal Sanity Report will be sent. Great improvement!, much appreciated! Cristopher Lemus From: "Khalil, Ghada" <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Wednesday, May 15, 2019 at 5:10 PM To: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: [Starlingx-discuss] UPDATE: Limited code merges until sanity results improve ** limited merges are still in effect ** Update: The following sanity issues have fixes merged in stx master: * https://bugs.launchpad.net/starlingx/+bug/1829260 * https://bugs.launchpad.net/starlingx/+bug/1829060 * https://bugs.launchpad.net/starlingx/+bug/1828896 * https://bugs.launchpad.net/starlingx/+bug/1828877 * https://bugs.launchpad.net/starlingx/+bug/1827521 * https://bugs.launchpad.net/starlingx/+bug/1828059 (dup) * https://bugs.launchpad.net/starlingx/+bug/1828899 (dup) A build has been triggered on cengn. ** Test team, please start sanity on the new build ** Regards, Ghada From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, May 15, 2019 11:09 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Limited code merges until sanity results improve Importance: High Hello all, As discussed in the community meeting today, effective immediately, we ask the core reviewers to only merge code for sanity gating issues and critical/high priority bugs only. This is in an effort to get some stability in the load and address the sanity issues recently reported. A new label, stx.sanity, will be introduced to easily help the community identify issues affecting sanity moving forward. https://bugs.launchpad.net/starlingx/+bugs?field.tag=stx.sanity (tagging of the current issues is in progress) For more details on the current issues, see today’s meeting minutes on the community etherpad: https://etherpad.openstack.org/p/stx-status Another email will be sent once the limited merge is lifted. We appreciate your patience as we work together to get stable loads. Regards, Ghada PS: Developers can continue to post reviews as per the normal process. Core reviewers can continue to review code as well, but not to merge them unless they meet the criteria stated above.