[Starlingx-discuss] bug severity and priority

Wed Jul 24 20:17:34 UTC 2019

As discussed in the Community call today, Ghada & I are proposing the following handling of bugs of the different priorities ("Importance" in Launchpad) as follows.

This will be discussed again tomorrow during the Release Team meeting as well... 

      - Critical:
           - must fix by release date (Release Build: Aug 23)

      - High: 
           - must be fixed for the stx.2.0 release, but could be fixed *after* the release date (in a maintenance release)
           - fixes will be backported to stx.2.0

      - Medium: 
           - continue working until release date
           - fix as many as possible 
           - defer to 3.0 after release date
           - fixes will be backported between RC1 and the release date
           - fixes will not be backported to stx.2.0 after the release date

      - Low: 
           - optional, will be deferred to stx.3.0 at RC1, fixes will not be backported to stx.2.0

Bill... 

-----Original Message-----
From: Xie, Cindy <cindy.xie at intel.com> 
Sent: Friday, July 19, 2019 7:56 PM
To: Perez Carranza, Jose <jose.perez.carranza at intel.com>; Saul Wold <sgw at linux.intel.com>; starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] bug severity and priority

Another idea is to using mailing list: each day, triage lead sends out a list of "new" bugs need triage and sub-project leads response in mailing list so that we keep the information public, we can assign bugs to appropriate owners (or people volunteer). 

Thx. - cindy

-----Original Message-----
From: Perez Carranza, Jose 
Sent: Friday, July 19, 2019 11:02 PM
To: Xie, Cindy <cindy.xie at intel.com>; Saul Wold <sgw at linux.intel.com>; starlingx-discuss at lists.starlingx.io
Subject: RE: [Starlingx-discuss] bug severity and priority

> -----Original Message-----
> From: Xie, Cindy
> Sent: Friday, July 19, 2019 9:35 AM
> To: Perez Carranza, Jose <jose.perez.carranza at intel.com>; Saul Wold 
> <sgw at linux.intel.com>; starlingx-discuss at lists.starlingx.io
> Subject: RE: [Starlingx-discuss] bug severity and priority
> 
> Jose,
> Just to clarify: for the weekly bug triage meeting, you only ask to 
> triage the new bugs, right?

Yes, only the new ones should be triaged. 

> 
> My concern is about the triage frequency: right now, the new bugs are 
> triaged almost on daily basis, mostly by Ghada by consulting technical expert.
> If we switch to a triage meeting, now sure how the new LP can be 
> handled timely.
> 
> But agree that having a triage meeting is a good idea.
> Thx. - cindy
> 

To mitigate this concern as Saul pointed out we should ensure to have a "triage section"section on subproject meeting but ensuring all the stakeholders for the specific bugs are online to provide feedback. 

> -----Original Message-----
> From: Perez Carranza, Jose [mailto:jose.perez.carranza at intel.com]
> Sent: Friday, July 19, 2019 8:16 PM
> To: Saul Wold <sgw at linux.intel.com>; 
> starlingx-discuss at lists.starlingx.io
> Subject: Re: [Starlingx-discuss] bug severity and priority
> 
> > -----Original Message-----
> > From: Saul Wold [mailto:sgw at linux.intel.com]
> > Sent: Thursday, July 18, 2019 10:57 PM
> > To: starlingx-discuss at lists.starlingx.io
> > Subject: Re: [Starlingx-discuss] bug severity and priority
> >
> >
> > Folks,
> >
> > As I mentioned in a prior email about a previous project (Yocto 
> > Project), we were also time-based (every 6 months).  We defined 
> > Importance [0] of the bug based on Severity (chosen by submitter) 
> > and Priority (assigned during a triage process). We had 5  Priory 
> > levels in
> > Bugzilla: High, Medium+, Medium, Low and Undecided, these would map
> to
> > our Critical, High, Medium, Low and Undecided.
> 
> Those triage meetings were very helpful because they were live 
> discussions about the bugs with all the stockholders. I think we 
> should consider to have a weekly meeting just to triage bugs.
> 
> Regards,
> José
> >
> > This clearly frames it based on Milestones and releases due to the 
> > time based nature of the Yocto Project.  Notice that the 
> > High/Critical is the only one that is truly "gating" or 
> > milestone/release blocker, the
> > Medium+, our High, won't block a milestone but be should be fixed 
> > Medium+for a
> > release, but could be a dot.dot soon after the release.
> >
> > > Importance
> > > The Importance of the bug is defined by its Priority and Severity.
> > > The
> > Priority classifies the bug's fixing order. In other words, how soon 
> > will it get fixed relative to other bugs? Priorities are set during 
> > the bug Triage meeting and cannot be changed by the user. The 
> > priority appears to the left of the Severity field. Here are the 
> > values that Priority can be set to during the Triage
> > meeting:
> > >
> > > High -- Bug fixing is planned immediately for the target milestone.
> > Milestone cannot be released if there is a high bug opened against 
> > the milestone. High priority issues cause major functional loss of a 
> > specific feature that is POR for the up-comping milestone. These 
> > issues are easily hit by the user and greatly impact the user 
> > experience or customer requirements. Finally, these issues could be 
> > urgent security fixes that need to be corrected in a prior release.
> > The bug assignee is not to change the target milestones for High 
> > bugs
> without prior approval of the Triage team.
> > > Medium+ -- Bug fixing is planned before the milestone and must be 
> > > Medium+ fixed or
> > have a solution planned before the release is finalized. These 
> > issues are not show-stoppers but have somewhat significant impact to 
> > system functions and user experience.
> > > Medium -- These are important issues we keep track and try to plan 
> > > fixing
> > for the release. They have limited impact for the system functions 
> > and releases.
> > > Low -- Bug fixing is only done opportunistically. Generally not 
> > > planned for
> > the up-coming project release. Issues that are not a POR feature 
> > request, or are hard to reproduce fall into this category.
> > > Undecided -- These issues are newly reported and are undecided 
> > > before
> > Triage. Issues that are a feature request, which isn't approved for 
> > future release yet. This issue will be changed to have an actual 
> > Priority after the Triage team approves it.
> > > Note: High impact but Low Priority bugs can be documented in the 
> > > release
> > notes.
> > >
> > > The Severity indicates how much the issue impacted the person 
> > > reporting
> > the bug. Severity can be categorized into five areas.
> > >
> > > Critical -- Crashes, hang, loss of data, negative impact to other 
> > > components,
> > memory leak etc.
> > > Major -- Major loss of functionality of POR.
> > > Normal -- Regular issue, some loss of functionality under certain
> > circumstance. This is the default Severity.
> > > Minor -- Minor loss of functionality, or issues with easy 
> > > workaround
> > available.
> > > Enhancement -- Request for enhancement or new feature to be worked.
> >
> > I hope the helps by provide a different viewpoint from another project.
> >
> > Sau!
> >
> > [0]
> > https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tr
> > ac
> > king
> > #Importance
> >
> > On 7/17/19 3:41 AM, Zvonar, Bill wrote:
> > > Hi Cindy,
> > >
> > > Thought about this some more, sorry it took me so long to respond
> further.
> > >
> > > I agree with splitting out the definitions of release 
> > > priority/importance
> > (which is subjective) from the technical severity (which is I'd say 
> > much less subjective).
> > >
> > > Do we agree that one of the key next steps is to define the 
> > > severity levels
> > for defects in different domains?
> > >
> > > Once we have those agreed and written down somewhere, they can be
> > used as guidance for people that are opening Launchpads, and for 
> > those that screen them.  Someone will note that some bugs cross 
> > domains, so it's not as simple as looking at one set of severity 
> > definitions, but let's cross that bridge next.
> > >
> > > Then, if we've got general alignment on the severity definitions 
> > > per domain,
> > we can sort out what to use as a QRC formula for a release, I think.
> > >
> > > Btw, it'd be nice if Launchpad had a field for Severity, so we 
> > > could track that
> > more easily - does anybody know if we can just request this & get it 
> > added as a custom field?
> > >
> > > Bill...
> > >
> > > -----Original Message-----
> > > From: Xie, Cindy <cindy.xie at intel.com>
> > > Sent: Wednesday, July 10, 2019 7:13 PM
> > > To: Zvonar, Bill <Bill.Zvonar at windriver.com>; starlingx-
> > discuss at lists.starlingx.io; Khalil, Ghada 
> > <Ghada.Khalil at windriver.com>
> > > Subject: RE: bug severity and priority
> > >
> > > Bill,
> > > I definitely agree that not all Medium shall be pushed to stx.3.0, 
> > > this needs
> > to be assessed carefully. But if we combine the severity and 
> > priority together, then this decision needs to put resource factor 
> > in consideration
> as well.
> > >
> > > Actually, I think it's confusing of calling individual LP "gating" 
> > > - I understand
> > that we want to get the product quality to a good shape and want to 
> > get bugs fixed as many as possible before we ship it. I will suggest 
> > to use defects# as part of release criteria (QRC). Example could be:
> > >
> > > 	Number of Critical P1 defects	Zero
> > > 	Number of High P2 defects	< x
> > > 	Number of Medium P3 defects	< y
> > >
> > > And the only thing we need to agree on is the "x" and "y". It 
> > > makes TSC or
> > release team to make decision easier. The QRC needs to be agreed 
> > earlier instead of right before the release decision shall be made.
> > This way, we can really direct our engineering resource working on 
> > the most important items and we all have an agreed common goal.
> > >
> > > Thanks. - cindy
> > >
> > > -----Original Message-----
> > > From: Zvonar, Bill [mailto:Bill.Zvonar at windriver.com]
> > > Sent: Thursday, July 11, 2019 1:39 AM
> > > To: Xie, Cindy <cindy.xie at intel.com>; 
> > > starlingx-discuss at lists.starlingx.io;
> > Khalil, Ghada <Ghada.Khalil at windriver.com>
> > > Subject: RE: bug severity and priority
> > >
> > > Hi Cindy,
> > >
> > > Thanks for sending this, I think this gives us something to start 
> > > the
> > discussion.
> > >
> > > However we decide to align on severity/priority (I'll comment on 
> > > that more
> > later, need to think about it more), I think we need to be careful 
> > before we move all mediums to 3.0, it may be too much of a Gordian 
> > knot
> solution.
> > >
> > > I think we need to assess the mediums (as Yong suggested earlier) 
> > > to say
> > why they should or should not be in 2.0.  I also think this may help 
> > us sort out what our gating criteria are.
> > >
> > > Bill...
> > >
> > > -----Original Message-----
> > > From: Xie, Cindy <cindy.xie at intel.com>
> > > Sent: Wednesday, July 10, 2019 10:42 AM
> > > To: starlingx-discuss at lists.starlingx.io; Zvonar, Bill
> > <Bill.Zvonar at windriver.com>; Khalil, Ghada 
> > <Ghada.Khalil at windriver.com>
> > > Subject: bug severity and priority
> > >
> > > Bill/Ghada,
> > > I am sending out my definition of bug severity and priority:
> > >
> > > Bug Exposure or Severity	Definition
> > > 1- Critical	Product or key feature is not usable for intended purpose.
> > > 2- High		Product or key feature is not reliably usable for
> > intended purpose or use is significantly impaired
> > > 3 - Medium	Product or key feature is usable provided by a workaround
> > > 4 - Low		Tolerable impact to user experience with minimal
> > service and support costs
> > >
> > > Bug Priority	Definition
> > > P1 - Stopper	Resolution of this defect takes precedence over other
> defects
> > and most other development activities. This level is used to focus 
> > maximum development team resources to resolve a defect in the 
> > shortest possible timeframe.
> > > P2 - High	Resolution of the defect has precedence over resolving other
> > defects with lesser classifications of priority.  The urgency to fix 
> > a
> > P2 priority defect is imminent. - P2 priority defects are intended 
> > to be resolved by the next planned external release of the software.
> > > P3 - Medium	Resolution of the defect has precedence over
> resolving other
> > defects with lesser classifications of priority.  - P3 priority 
> > defects must have a planned timeframe for a verified resolution.
> > > P4 - Low	Resolution of the defect has least urgency to resolve, P4
> > priority defects may or may not have plans to resolve.
> > >
> > > Let's discuss this and agree how we'd like to use them. My 
> > > suggestion for
> > current "Medium" is to we can mark them as "stx.3.0" and then in the 
> > beginning of stx.3, they can move Priority to "high" due to the fact 
> > they want to get them fixed in 3.0.
> > >
> > > But the bug severity should never change because they are standard.
> > >
> > > Thx. - cindy
> > >
> > >
> > > _______________________________________________
> > > Starlingx-discuss mailing list
> > > Starlingx-discuss at lists.starlingx.io
> > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discu
> > > ss
> > >
> >
> > _______________________________________________
> > Starlingx-discuss mailing list
> > Starlingx-discuss at lists.starlingx.io
> > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss