[Starlingx-discuss] Limited code merges no longer in effect

[Khalil, Ghada]

We agreed in the community call today to allow code to merge in stx master given we had two yellow sanity reports.

There are still a few sanity issues that are under investigation, but we agreed as a community to move forward with merges. See below for a list of issues still under investigation.

We ask all developers to be diligent with testing and help us maintain stability.

Please note that code (that is ready) will be merged gradually over the next few days.

List of sanity issues still under investigation:

  *   New issue reported by sanity
?  https://bugs.launchpad.net/starlingx/+bug/1829941
?  A better description of the issue is that: After a swact, cannot login to controller-0
?  Sanity hasn't reached this point in a while, so don't know when this was an issue previously
?  Action: Christopher to update the bug with additional info and interface configuration
?  We will continue to investigate, but agreed that we wouldn't hold up merges for this bug.

  *   One sanity issue may have been reproduced in WR lab
?  https://bugs.launchpad.net/starlingx/+bug/1828896 - Application-apply failed due to error copying secret ceph-pool-kube-rbd
*        not reported for Intel sanity team since fix merged. Christopher confirmed that he hasn't seen this.
*        reproduction in WR was seen on AIO-SX only, not on other configs.
*        Intel sanity has been waiting about 5mins after the controller comes up before running application apply to allow the system to settle down

  *   Any other critical / high severity bugs impacting sanity or load stability?
?  From Bruce -- Ada indicated that she has passing sanity for simplex and duplex. A report to be sent out shortly.
?  From Numan
*        https://bugs.launchpad.net/starlingx/+bug/1829931 - only seen on Duplex
*        https://bugs.launchpad.net/starlingx/+bug/1829933 - only seen on Force Reboot on Duplex
*        Action: Do initial triage to see what the issue is. These issues are likely present in previous loads as well.
*        Agreed to not block merges, but continue to investigate as high priority.


From: Khalil, Ghada
Sent: Tuesday, May 21, 2019 1:03 PM
To: starlingx-discuss at lists.starlingx.io
Subject: UPDATE: Limited code merges until sanity results improve

The latest sanity report sent by Maria Perez on Monday May 20 reported two issues:


-        cinder volume create failed https://bugs.launchpad.net/starlingx/+bug/1828059

o   This is believed to be the result of sanity not picking the correct docker images which include the fix. See email from Don Penney. Need the sanity team to re-test.

-        SRIOV and Multus images not pulled from local mirror registry https://bugs.launchpad.net/starlingx/+bug/1829299

o   Fix merged and is available in cengn build: 20190521T132734Z

Based on the above, I ask the core reviewers to start allowing any critical or high priority bugs to merge, not only sanity issues.

We will evaluate in the community meeting tomorrow morning whether we start allowing other commits in starting tomorrow.

Regards,
Ghada


From: Khalil, Ghada [mailto:Ghada.Khalil at windriver.com]
Sent: Thursday, May 16, 2019 6:41 PM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] UPDATE: Limited code merges until sanity results improve

** limited merges are still in effect **

The following sanity issues are still open:


-        https://bugs.launchpad.net/starlingx/+bug/1828562 - issue was reproduced and re-opened

-        https://bugs.launchpad.net/starlingx/+bug/1828059 - issue was reproduced and re-opened

-        https://bugs.launchpad.net/starlingx/+bug/1829299 - new issue in the 20190515T220331Z load

-        https://bugs.launchpad.net/starlingx/+bug/1829433 - new issue in the 20190515T220331Z load

Note: As of now, there are no additional fixes merged since the 20190515T220331Z cengn load.

Core reviewers, please only merge fixes for critical/high sanity issues.


From: Khalil, Ghada
Sent: Wednesday, May 15, 2019 6:09 PM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: UPDATE: Limited code merges until sanity results improve

** limited merges are still in effect **

Update:
The following sanity issues have fixes merged in stx master:

-        https://bugs.launchpad.net/starlingx/+bug/1829260

-        https://bugs.launchpad.net/starlingx/+bug/1829060

-        https://bugs.launchpad.net/starlingx/+bug/1828896

-        https://bugs.launchpad.net/starlingx/+bug/1828877

-        https://bugs.launchpad.net/starlingx/+bug/1827521

-        https://bugs.launchpad.net/starlingx/+bug/1828059 (dup)

-        https://bugs.launchpad.net/starlingx/+bug/1828899 (dup)

A build has been triggered on cengn.

** Test team, please start sanity on the new build **

Regards,
Ghada

From: Khalil, Ghada [mailto:Ghada.Khalil at windriver.com]
Sent: Wednesday, May 15, 2019 11:09 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] Limited code merges until sanity results improve
Importance: High


Hello all,
As discussed in the community meeting today, effective immediately, we ask the core reviewers to only merge code for sanity gating issues and critical/high priority bugs only.

This is in an effort to get some stability in the load and address the sanity issues recently reported.

A new label, stx.sanity, will be introduced to easily help the community identify issues affecting sanity moving forward.
https://bugs.launchpad.net/starlingx/+bugs?field.tag=stx.sanity
(tagging of the current issues is in progress)

For more details on the current issues, see today's meeting minutes on the community etherpad:
https://etherpad.openstack.org/p/stx-status

Another email will be sent once the limited merge is lifted.

We appreciate your patience as we work together to get stable loads.

Regards,
Ghada

PS: Developers can continue to post reviews as per the normal process. Core reviewers can continue to review code as well, but not to merge them unless they meet the criteria stated above.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190522/b4f37b4b/attachment-0001.html>