[Starlingx-discuss] bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4 priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill Bill.Zvonar@windriver.com; Khalil, Ghada Ghada.Khalil@windriver.com Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4 priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
On 7/10/19 10:38 AM, Zvonar, Bill wrote:
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
+1, we went through a very similar community process with the Yocto Project early on, everyone has their ideas of Priority/Severity and who sets what and when. We will work through this.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
I agree, we need to take a measured approach to the existing mediums and determine if they are truly 2.0 gating, in which case elevate them, the rest could/should be marked for 3.0.
As I mentioned on the phone, having a burn-down chart for the Critical (1) and Highs (16) currently might help, as we should be approaching 0 on these existing ones entering RC1.
Sau!
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill Bill.Zvonar@windriver.com; Khalil, Ghada Ghada.Khalil@windriver.com Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4 priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy cindy.xie@intel.com; starlingx-discuss@lists.starlingx.io; Khalil, Ghada Ghada.Khalil@windriver.com Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill Bill.Zvonar@windriver.com; Khalil, Ghada Ghada.Khalil@windriver.com Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4 priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance (which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels for defects in different domains?
Once we have those agreed and written down somewhere, they can be used as guidance for people that are opening Launchpads, and for those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain, we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill Bill.Zvonar@windriver.com; starlingx-discuss@lists.starlingx.io; Khalil, Ghada Ghada.Khalil@windriver.com Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy cindy.xie@intel.com; starlingx-discuss@lists.starlingx.io; Khalil, Ghada Ghada.Khalil@windriver.com Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill Bill.Zvonar@windriver.com; Khalil, Ghada Ghada.Khalil@windriver.com Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4 priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
On 2019-07-17 10:41:09 +0000 (+0000), Zvonar, Bill wrote: [...]
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that more easily - does anybody know if we can just request this & get it added as a custom field?
[...]
The data model for Launchpad hasn't changed in many years and I gather its codebase has very few remaining maintainers at Canonical these days. In the past we've had some limited luck requesting new methods added to the LP API, so I suppose it can't hurt to ask. You could file such a feature request here:
https://bugs.launchpad.net/launchpad/+filebug
An alternative (which they are just as likely to recommend, if they respond to your request) is to use bugtags for this purpose. Those can have whatever names you want and you can still query based on them.
It's worth noting that the subjective and non-project-agnostic nature of severity/importance is what led StoryBoard to not include fields for them, opting instead for a combination of extensible worklists (where relative priority can be indicated by manually ranking different stories, and you can have multiple worklists which rank them differently according to a variety of arbitrarily-chosen metrics or individual opinions) and storytags (which can have whatever names you want and are also usable to populate automatic worklists and board lanes).
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map to our Critical, High, Medium, Low and Undecided.
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed for a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone. Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs without prior approval of the Triage team. Medium+ -- Bug fixing is planned before the milestone and must be fixed or have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience. Medium -- These are important issues we keep track and try to plan fixing for the release. They have limited impact for the system functions and releases. Low -- Bug fixing is only done opportunistically. Generally not planned for the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category. Undecided -- These issues are newly reported and are undecided before Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it. Note: High impact but Low Priority bugs can be documented in the release notes.
The Severity indicates how much the issue impacted the person reporting the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components, memory leak etc. Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain circumstance. This is the default Severity. Minor -- Minor loss of functionality, or issues with easy workaround available. Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking#I...
On 7/17/19 3:41 AM, Zvonar, Bill wrote:
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance (which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels for defects in different domains?
Once we have those agreed and written down somewhere, they can be used as guidance for people that are opening Launchpads, and for those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain, we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill Bill.Zvonar@windriver.com; starlingx-discuss@lists.starlingx.io; Khalil, Ghada Ghada.Khalil@windriver.com Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy cindy.xie@intel.com; starlingx-discuss@lists.starlingx.io; Khalil, Ghada Ghada.Khalil@windriver.com Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill Bill.Zvonar@windriver.com; Khalil, Ghada Ghada.Khalil@windriver.com Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4 priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map to our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs.
Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed for a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The
Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone.
Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs without prior approval of the Triage team.
Medium+ -- Bug fixing is planned before the milestone and must be fixed or
have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience.
Medium -- These are important issues we keep track and try to plan fixing
for the release. They have limited impact for the system functions and releases.
Low -- Bug fixing is only done opportunistically. Generally not planned for
the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category.
Undecided -- These issues are newly reported and are undecided before
Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it.
Note: High impact but Low Priority bugs can be documented in the release
notes.
The Severity indicates how much the issue impacted the person reporting
the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components,
memory leak etc.
Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain
circumstance. This is the default Severity.
Minor -- Minor loss of functionality, or issues with easy workaround
available.
Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking #Importance
On 7/17/19 3:41 AM, Zvonar, Bill wrote:
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance
(which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels
for defects in different domains?
Once we have those agreed and written down somewhere, they can be
used as guidance for people that are opening Launchpads, and for those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain,
we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that
more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill Bill.Zvonar@windriver.com; starlingx-
discuss@lists.starlingx.io; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs
to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand
that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy cindy.xie@intel.com; starlingx-discuss@lists.starlingx.io;
Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the
discussion.
However we decide to align on severity/priority (I'll comment on that more
later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say
why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill
Bill.Zvonar@windriver.com; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for
intended purpose or use is significantly impaired
3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal
service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects
and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe.
P2 - High Resolution of the defect has precedence over resolving other
defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software.
P3 - Medium Resolution of the defect has precedence over resolving other
defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution.
P4 - Low Resolution of the defect has least urgency to resolve, P4
priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Jose, Just to clarify: for the weekly bug triage meeting, you only ask to triage the new bugs, right?
My concern is about the triage frequency: right now, the new bugs are triaged almost on daily basis, mostly by Ghada by consulting technical expert. If we switch to a triage meeting, now sure how the new LP can be handled timely.
But agree that having a triage meeting is a good idea. Thx. - cindy
-----Original Message----- From: Perez Carranza, Jose [mailto:jose.perez.carranza@intel.com] Sent: Friday, July 19, 2019 8:16 PM To: Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map to our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs.
Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed for Medium+a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The
Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone.
Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs without prior approval of the Triage team.
Medium+ -- Bug fixing is planned before the milestone and must be Medium+ fixed or
have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience.
Medium -- These are important issues we keep track and try to plan fixing
for the release. They have limited impact for the system functions and releases.
Low -- Bug fixing is only done opportunistically. Generally not planned for
the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category.
Undecided -- These issues are newly reported and are undecided before
Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it.
Note: High impact but Low Priority bugs can be documented in the release
notes.
The Severity indicates how much the issue impacted the person reporting
the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components,
memory leak etc.
Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain
circumstance. This is the default Severity.
Minor -- Minor loss of functionality, or issues with easy workaround
available.
Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Trac king #Importance
On 7/17/19 3:41 AM, Zvonar, Bill wrote:
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance
(which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels
for defects in different domains?
Once we have those agreed and written down somewhere, they can be
used as guidance for people that are opening Launchpads, and for those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain,
we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that
more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill Bill.Zvonar@windriver.com; starlingx-
discuss@lists.starlingx.io; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs
to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand
that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy cindy.xie@intel.com; starlingx-discuss@lists.starlingx.io;
Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the
discussion.
However we decide to align on severity/priority (I'll comment on that more
later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say
why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill
Bill.Zvonar@windriver.com; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for
intended purpose or use is significantly impaired
3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal
service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects
and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe.
P2 - High Resolution of the defect has precedence over resolving other
defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software.
P3 - Medium Resolution of the defect has precedence over resolving other
defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution.
P4 - Low Resolution of the defect has least urgency to resolve, P4
priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
-----Original Message----- From: Xie, Cindy Sent: Friday, July 19, 2019 9:35 AM To: Perez Carranza, Jose jose.perez.carranza@intel.com; Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
Jose, Just to clarify: for the weekly bug triage meeting, you only ask to triage the new bugs, right?
Yes, only the new ones should be triaged.
My concern is about the triage frequency: right now, the new bugs are triaged almost on daily basis, mostly by Ghada by consulting technical expert. If we switch to a triage meeting, now sure how the new LP can be handled timely.
But agree that having a triage meeting is a good idea. Thx. - cindy
To mitigate this concern as Saul pointed out we should ensure to have a "triage section"section on subproject meeting but ensuring all the stakeholders for the specific bugs are online to provide feedback.
-----Original Message----- From: Perez Carranza, Jose [mailto:jose.perez.carranza@intel.com] Sent: Friday, July 19, 2019 8:16 PM To: Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map
to
our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs.
Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed for Medium+a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The
Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone.
Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs
without prior approval of the Triage team.
Medium+ -- Bug fixing is planned before the milestone and must be Medium+ fixed or
have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience.
Medium -- These are important issues we keep track and try to plan fixing
for the release. They have limited impact for the system functions and releases.
Low -- Bug fixing is only done opportunistically. Generally not planned for
the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category.
Undecided -- These issues are newly reported and are undecided before
Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it.
Note: High impact but Low Priority bugs can be documented in the release
notes.
The Severity indicates how much the issue impacted the person reporting
the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components,
memory leak etc.
Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain
circumstance. This is the default Severity.
Minor -- Minor loss of functionality, or issues with easy workaround
available.
Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Trac king #Importance
On 7/17/19 3:41 AM, Zvonar, Bill wrote:
Hi Cindy,
Thought about this some more, sorry it took me so long to respond
further.
I agree with splitting out the definitions of release priority/importance
(which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels
for defects in different domains?
Once we have those agreed and written down somewhere, they can be
used as guidance for people that are opening Launchpads, and for those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain,
we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that
more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill Bill.Zvonar@windriver.com; starlingx-
discuss@lists.starlingx.io; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs
to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration
as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand
that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy cindy.xie@intel.com; starlingx-discuss@lists.starlingx.io;
Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the
discussion.
However we decide to align on severity/priority (I'll comment on that more
later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot
solution.
I think we need to assess the mediums (as Yong suggested earlier) to say
why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill
Bill.Zvonar@windriver.com; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for
intended purpose or use is significantly impaired
3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal
service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other
defects
and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe.
P2 - High Resolution of the defect has precedence over resolving other
defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software.
P3 - Medium Resolution of the defect has precedence over
resolving other
defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution.
P4 - Low Resolution of the defect has least urgency to resolve, P4
priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Another idea is to using mailing list: each day, triage lead sends out a list of "new" bugs need triage and sub-project leads response in mailing list so that we keep the information public, we can assign bugs to appropriate owners (or people volunteer).
Thx. - cindy
-----Original Message----- From: Perez Carranza, Jose Sent: Friday, July 19, 2019 11:02 PM To: Xie, Cindy cindy.xie@intel.com; Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Xie, Cindy Sent: Friday, July 19, 2019 9:35 AM To: Perez Carranza, Jose jose.perez.carranza@intel.com; Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
Jose, Just to clarify: for the weekly bug triage meeting, you only ask to triage the new bugs, right?
Yes, only the new ones should be triaged.
My concern is about the triage frequency: right now, the new bugs are triaged almost on daily basis, mostly by Ghada by consulting technical expert. If we switch to a triage meeting, now sure how the new LP can be handled timely.
But agree that having a triage meeting is a good idea. Thx. - cindy
To mitigate this concern as Saul pointed out we should ensure to have a "triage section"section on subproject meeting but ensuring all the stakeholders for the specific bugs are online to provide feedback.
-----Original Message----- From: Perez Carranza, Jose [mailto:jose.perez.carranza@intel.com] Sent: Friday, July 19, 2019 8:16 PM To: Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map
to
our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs.
Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed Medium+for a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The
Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone.
Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs
without prior approval of the Triage team.
Medium+ -- Bug fixing is planned before the milestone and must be Medium+ fixed or
have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience.
Medium -- These are important issues we keep track and try to plan fixing
for the release. They have limited impact for the system functions and releases.
Low -- Bug fixing is only done opportunistically. Generally not planned for
the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category.
Undecided -- These issues are newly reported and are undecided before
Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it.
Note: High impact but Low Priority bugs can be documented in the release
notes.
The Severity indicates how much the issue impacted the person reporting
the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components,
memory leak etc.
Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain
circumstance. This is the default Severity.
Minor -- Minor loss of functionality, or issues with easy workaround
available.
Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tr ac king #Importance
On 7/17/19 3:41 AM, Zvonar, Bill wrote:
Hi Cindy,
Thought about this some more, sorry it took me so long to respond
further.
I agree with splitting out the definitions of release priority/importance
(which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels
for defects in different domains?
Once we have those agreed and written down somewhere, they can be
used as guidance for people that are opening Launchpads, and for those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain,
we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that
more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill Bill.Zvonar@windriver.com; starlingx-
discuss@lists.starlingx.io; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs
to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration
as well.
Actually, I think it's confusing of calling individual LP "gating"
- I understand
that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy cindy.xie@intel.com; starlingx-discuss@lists.starlingx.io;
Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the
discussion.
However we decide to align on severity/priority (I'll comment on that more
later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot
solution.
I think we need to assess the mediums (as Yong suggested earlier) to say
why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill
Bill.Zvonar@windriver.com; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for
intended purpose or use is significantly impaired
3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal
service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other
defects
and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe.
P2 - High Resolution of the defect has precedence over resolving other
defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software.
P3 - Medium Resolution of the defect has precedence over
resolving other
defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution.
P4 - Low Resolution of the defect has least urgency to resolve, P4
priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discu ss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
As discussed in the Community call today, Ghada & I are proposing the following handling of bugs of the different priorities ("Importance" in Launchpad) as follows.
This will be discussed again tomorrow during the Release Team meeting as well...
- Critical: - must fix by release date (Release Build: Aug 23)
- High: - must be fixed for the stx.2.0 release, but could be fixed *after* the release date (in a maintenance release) - fixes will be backported to stx.2.0
- Medium: - continue working until release date - fix as many as possible - defer to 3.0 after release date - fixes will be backported between RC1 and the release date - fixes will not be backported to stx.2.0 after the release date
- Low: - optional, will be deferred to stx.3.0 at RC1, fixes will not be backported to stx.2.0
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Friday, July 19, 2019 7:56 PM To: Perez Carranza, Jose jose.perez.carranza@intel.com; Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Another idea is to using mailing list: each day, triage lead sends out a list of "new" bugs need triage and sub-project leads response in mailing list so that we keep the information public, we can assign bugs to appropriate owners (or people volunteer).
Thx. - cindy
-----Original Message----- From: Perez Carranza, Jose Sent: Friday, July 19, 2019 11:02 PM To: Xie, Cindy cindy.xie@intel.com; Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Xie, Cindy Sent: Friday, July 19, 2019 9:35 AM To: Perez Carranza, Jose jose.perez.carranza@intel.com; Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
Jose, Just to clarify: for the weekly bug triage meeting, you only ask to triage the new bugs, right?
Yes, only the new ones should be triaged.
My concern is about the triage frequency: right now, the new bugs are triaged almost on daily basis, mostly by Ghada by consulting technical expert. If we switch to a triage meeting, now sure how the new LP can be handled timely.
But agree that having a triage meeting is a good idea. Thx. - cindy
To mitigate this concern as Saul pointed out we should ensure to have a "triage section"section on subproject meeting but ensuring all the stakeholders for the specific bugs are online to provide feedback.
-----Original Message----- From: Perez Carranza, Jose [mailto:jose.perez.carranza@intel.com] Sent: Friday, July 19, 2019 8:16 PM To: Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map
to
our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs.
Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed Medium+for a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The
Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone.
Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs
without prior approval of the Triage team.
Medium+ -- Bug fixing is planned before the milestone and must be Medium+ fixed or
have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience.
Medium -- These are important issues we keep track and try to plan fixing
for the release. They have limited impact for the system functions and releases.
Low -- Bug fixing is only done opportunistically. Generally not planned for
the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category.
Undecided -- These issues are newly reported and are undecided before
Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it.
Note: High impact but Low Priority bugs can be documented in the release
notes.
The Severity indicates how much the issue impacted the person reporting
the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components,
memory leak etc.
Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain
circumstance. This is the default Severity.
Minor -- Minor loss of functionality, or issues with easy workaround
available.
Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tr ac king #Importance
On 7/17/19 3:41 AM, Zvonar, Bill wrote:
Hi Cindy,
Thought about this some more, sorry it took me so long to respond
further.
I agree with splitting out the definitions of release priority/importance
(which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels
for defects in different domains?
Once we have those agreed and written down somewhere, they can be
used as guidance for people that are opening Launchpads, and for those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain,
we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that
more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill Bill.Zvonar@windriver.com; starlingx-
discuss@lists.starlingx.io; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs
to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration
as well.
Actually, I think it's confusing of calling individual LP "gating"
- I understand
that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy cindy.xie@intel.com; starlingx-discuss@lists.starlingx.io;
Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the
discussion.
However we decide to align on severity/priority (I'll comment on that more
later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot
solution.
I think we need to assess the mediums (as Yong suggested earlier) to say
why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill
Bill.Zvonar@windriver.com; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for
intended purpose or use is significantly impaired
3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal
service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other
defects
and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe.
P2 - High Resolution of the defect has precedence over resolving other
defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software.
P3 - Medium Resolution of the defect has precedence over
resolving other
defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution.
P4 - Low Resolution of the defect has least urgency to resolve, P4
priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discu ss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Thanks Bill and Ghada drive the discussion thread to a closure. I think this is a great balance btw "time-based release" strategy we agreed earlier and also put quality as top priority.
Thanks! - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 25, 2019 4:18 AM To: Xie, Cindy cindy.xie@intel.com; Perez Carranza, Jose jose.perez.carranza@intel.com; Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
As discussed in the Community call today, Ghada & I are proposing the following handling of bugs of the different priorities ("Importance" in Launchpad) as follows.
This will be discussed again tomorrow during the Release Team meeting as well...
- Critical: - must fix by release date (Release Build: Aug 23)
- High: - must be fixed for the stx.2.0 release, but could be fixed *after* the release date (in a maintenance release) - fixes will be backported to stx.2.0
- Medium: - continue working until release date - fix as many as possible - defer to 3.0 after release date - fixes will be backported between RC1 and the release date - fixes will not be backported to stx.2.0 after the release date
- Low: - optional, will be deferred to stx.3.0 at RC1, fixes will not be backported to stx.2.0
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Friday, July 19, 2019 7:56 PM To: Perez Carranza, Jose jose.perez.carranza@intel.com; Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Another idea is to using mailing list: each day, triage lead sends out a list of "new" bugs need triage and sub-project leads response in mailing list so that we keep the information public, we can assign bugs to appropriate owners (or people volunteer).
Thx. - cindy
-----Original Message----- From: Perez Carranza, Jose Sent: Friday, July 19, 2019 11:02 PM To: Xie, Cindy cindy.xie@intel.com; Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Xie, Cindy Sent: Friday, July 19, 2019 9:35 AM To: Perez Carranza, Jose jose.perez.carranza@intel.com; Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
Jose, Just to clarify: for the weekly bug triage meeting, you only ask to triage the new bugs, right?
Yes, only the new ones should be triaged.
My concern is about the triage frequency: right now, the new bugs are triaged almost on daily basis, mostly by Ghada by consulting technical expert. If we switch to a triage meeting, now sure how the new LP can be handled timely.
But agree that having a triage meeting is a good idea. Thx. - cindy
To mitigate this concern as Saul pointed out we should ensure to have a "triage section"section on subproject meeting but ensuring all the stakeholders for the specific bugs are online to provide feedback.
-----Original Message----- From: Perez Carranza, Jose [mailto:jose.perez.carranza@intel.com] Sent: Friday, July 19, 2019 8:16 PM To: Saul Wold sgw@linux.intel.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map
to
our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs.
Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed Medium+for a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The
Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone.
Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs
without prior approval of the Triage team.
Medium+ -- Bug fixing is planned before the milestone and must be Medium+ fixed or
have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience.
Medium -- These are important issues we keep track and try to plan fixing
for the release. They have limited impact for the system functions and releases.
Low -- Bug fixing is only done opportunistically. Generally not planned for
the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category.
Undecided -- These issues are newly reported and are undecided before
Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it.
Note: High impact but Low Priority bugs can be documented in the release
notes.
The Severity indicates how much the issue impacted the person reporting
the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components,
memory leak etc.
Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain
circumstance. This is the default Severity.
Minor -- Minor loss of functionality, or issues with easy workaround
available.
Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tr ac king #Importance
On 7/17/19 3:41 AM, Zvonar, Bill wrote:
Hi Cindy,
Thought about this some more, sorry it took me so long to respond
further.
I agree with splitting out the definitions of release priority/importance
(which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels
for defects in different domains?
Once we have those agreed and written down somewhere, they can be
used as guidance for people that are opening Launchpads, and for those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain,
we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that
more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill Bill.Zvonar@windriver.com; starlingx-
discuss@lists.starlingx.io; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs
to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration
as well.
Actually, I think it's confusing of calling individual LP "gating"
- I understand
that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy cindy.xie@intel.com; starlingx-discuss@lists.starlingx.io;
Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the
discussion.
However we decide to align on severity/priority (I'll comment on that more
later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot
solution.
I think we need to assess the mediums (as Yong suggested earlier) to say
why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill
Bill.Zvonar@windriver.com; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for
intended purpose or use is significantly impaired
3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal
service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other
defects
and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe.
P2 - High Resolution of the defect has precedence over resolving other
defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software.
P3 - Medium Resolution of the defect has precedence over
resolving other
defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution.
P4 - Low Resolution of the defect has least urgency to resolve, P4
priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discu ss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 7/19/19 5:15 AM, Perez Carranza, Jose wrote:
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map to our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs.
Not sure we need a bug triage meeting, that should be happening in the sub-team meetings.
Sau!
Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed for a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The
Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone.
Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs without prior approval of the Triage team.
Medium+ -- Bug fixing is planned before the milestone and must be fixed or
have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience.
Medium -- These are important issues we keep track and try to plan fixing
for the release. They have limited impact for the system functions and releases.
Low -- Bug fixing is only done opportunistically. Generally not planned for
the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category.
Undecided -- These issues are newly reported and are undecided before
Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it.
Note: High impact but Low Priority bugs can be documented in the release
notes.
The Severity indicates how much the issue impacted the person reporting
the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components,
memory leak etc.
Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain
circumstance. This is the default Severity.
Minor -- Minor loss of functionality, or issues with easy workaround
available.
Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking #Importance
On 7/17/19 3:41 AM, Zvonar, Bill wrote:
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance
(which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels
for defects in different domains?
Once we have those agreed and written down somewhere, they can be
used as guidance for people that are opening Launchpads, and for those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain,
we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that
more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill Bill.Zvonar@windriver.com; starlingx-
discuss@lists.starlingx.io; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs
to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand
that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy cindy.xie@intel.com; starlingx-discuss@lists.starlingx.io;
Khalil, Ghada Ghada.Khalil@windriver.com
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the
discussion.
However we decide to align on severity/priority (I'll comment on that more
later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say
why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy cindy.xie@intel.com Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill
Bill.Zvonar@windriver.com; Khalil, Ghada Ghada.Khalil@windriver.com
Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for
intended purpose or use is significantly impaired
3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal
service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects
and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe.
P2 - High Resolution of the defect has precedence over resolving other
defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software.
P3 - Medium Resolution of the defect has precedence over resolving other
defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution.
P4 - Low Resolution of the defect has least urgency to resolve, P4
priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (5)
-
Jeremy Stanley
-
Perez Carranza, Jose
-
Saul Wold
-
Xie, Cindy
-
Zvonar, Bill