Thanks Bill and Ghada drive the discussion thread to a closure. I think this is a great balance btw "time-based release" strategy we agreed earlier and also put quality as top priority. Thanks! - cindy -----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 25, 2019 4:18 AM To: Xie, Cindy <cindy.xie@intel.com>; Perez Carranza, Jose <jose.perez.carranza@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority As discussed in the Community call today, Ghada & I are proposing the following handling of bugs of the different priorities ("Importance" in Launchpad) as follows. This will be discussed again tomorrow during the Release Team meeting as well... - Critical: - must fix by release date (Release Build: Aug 23) - High: - must be fixed for the stx.2.0 release, but could be fixed *after* the release date (in a maintenance release) - fixes will be backported to stx.2.0 - Medium: - continue working until release date - fix as many as possible - defer to 3.0 after release date - fixes will be backported between RC1 and the release date - fixes will not be backported to stx.2.0 after the release date - Low: - optional, will be deferred to stx.3.0 at RC1, fixes will not be backported to stx.2.0 Bill... -----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Friday, July 19, 2019 7:56 PM To: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority Another idea is to using mailing list: each day, triage lead sends out a list of "new" bugs need triage and sub-project leads response in mailing list so that we keep the information public, we can assign bugs to appropriate owners (or people volunteer). Thx. - cindy -----Original Message----- From: Perez Carranza, Jose Sent: Friday, July 19, 2019 11:02 PM To: Xie, Cindy <cindy.xie@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Xie, Cindy Sent: Friday, July 19, 2019 9:35 AM To: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
Jose, Just to clarify: for the weekly bug triage meeting, you only ask to triage the new bugs, right?
Yes, only the new ones should be triaged.
My concern is about the triage frequency: right now, the new bugs are triaged almost on daily basis, mostly by Ghada by consulting technical expert. If we switch to a triage meeting, now sure how the new LP can be handled timely.
But agree that having a triage meeting is a good idea. Thx. - cindy
To mitigate this concern as Saul pointed out we should ensure to have a "triage section"section on subproject meeting but ensuring all the stakeholders for the specific bugs are online to provide feedback.
-----Original Message----- From: Perez Carranza, Jose [mailto:jose.perez.carranza@intel.com] Sent: Friday, July 19, 2019 8:16 PM To: Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map to our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs.
Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed Medium+for a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone. Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs
without prior approval of the Triage team.
Medium+ -- Bug fixing is planned before the milestone and must be Medium+ fixed or have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience. Medium -- These are important issues we keep track and try to plan fixing for the release. They have limited impact for the system functions and releases. Low -- Bug fixing is only done opportunistically. Generally not planned for the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category. Undecided -- These issues are newly reported and are undecided before Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it. Note: High impact but Low Priority bugs can be documented in the release notes.
The Severity indicates how much the issue impacted the person reporting the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components, memory leak etc. Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain circumstance. This is the default Severity. Minor -- Minor loss of functionality, or issues with easy workaround available. Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tr ac king #Importance
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance (which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels for defects in different domains?
Once we have those agreed and written down somewhere, they can be used as guidance for people that are opening Launchpads, and for
On 7/17/19 3:41 AM, Zvonar, Bill wrote: those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain,
we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that
more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; starlingx-
Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and
discuss@lists.starlingx.io; Khalil, Ghada <Ghada.Khalil@windriver.com> priority together, then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand
that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io;
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4
Khalil, Ghada <Ghada.Khalil@windriver.com> priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discu ss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss