[Starlingx-discuss] bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority: Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4 priority defects may or may not have plans to resolve. Let's discuss this and agree how we'd like to use them. My suggestion for current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0. But the bug severity should never change because they are standard. Thx. - cindy
Hi Cindy, Thanks for sending this, I think this gives us something to start the discussion. However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution. I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are. Bill... -----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: bug severity and priority Bill/Ghada, I am sending out my definition of bug severity and priority: Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4 priority defects may or may not have plans to resolve. Let's discuss this and agree how we'd like to use them. My suggestion for current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0. But the bug severity should never change because they are standard. Thx. - cindy
On 7/10/19 10:38 AM, Zvonar, Bill wrote:
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
+1, we went through a very similar community process with the Yocto Project early on, everyone has their ideas of Priority/Severity and who sets what and when. We will work through this.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
I agree, we need to take a measured approach to the existing mediums and determine if they are truly 2.0 gating, in which case elevate them, the rest could/should be marked for 3.0. As I mentioned on the phone, having a burn-down chart for the Critical (1) and Highs (16) currently might help, as we should be approaching 0 on these existing ones entering RC1. Sau!
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4 priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration as well. Actually, I think it's confusing of calling individual LP "gating" - I understand that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be: Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y And the only thing we need to agree on is the "x" and "y". It makes TSC or release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal. Thanks. - cindy -----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: bug severity and priority Hi Cindy, Thanks for sending this, I think this gives us something to start the discussion. However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution. I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are. Bill... -----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: bug severity and priority Bill/Ghada, I am sending out my definition of bug severity and priority: Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4 priority defects may or may not have plans to resolve. Let's discuss this and agree how we'd like to use them. My suggestion for current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0. But the bug severity should never change because they are standard. Thx. - cindy
Hi Cindy, Thought about this some more, sorry it took me so long to respond further. I agree with splitting out the definitions of release priority/importance (which is subjective) from the technical severity (which is I'd say much less subjective). Do we agree that one of the key next steps is to define the severity levels for defects in different domains? Once we have those agreed and written down somewhere, they can be used as guidance for people that are opening Launchpads, and for those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next. Then, if we've got general alignment on the severity definitions per domain, we can sort out what to use as a QRC formula for a release, I think. Btw, it'd be nice if Launchpad had a field for Severity, so we could track that more easily - does anybody know if we can just request this & get it added as a custom field? Bill... -----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; starlingx-discuss@lists.starlingx.io; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: bug severity and priority Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration as well. Actually, I think it's confusing of calling individual LP "gating" - I understand that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be: Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y And the only thing we need to agree on is the "x" and "y". It makes TSC or release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal. Thanks. - cindy -----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: bug severity and priority Hi Cindy, Thanks for sending this, I think this gives us something to start the discussion. However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution. I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are. Bill... -----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: bug severity and priority Bill/Ghada, I am sending out my definition of bug severity and priority: Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4 priority defects may or may not have plans to resolve. Let's discuss this and agree how we'd like to use them. My suggestion for current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0. But the bug severity should never change because they are standard. Thx. - cindy
On 2019-07-17 10:41:09 +0000 (+0000), Zvonar, Bill wrote: [...]
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that more easily - does anybody know if we can just request this & get it added as a custom field? [...]
The data model for Launchpad hasn't changed in many years and I gather its codebase has very few remaining maintainers at Canonical these days. In the past we've had some limited luck requesting new methods added to the LP API, so I suppose it can't hurt to ask. You could file such a feature request here: https://bugs.launchpad.net/launchpad/+filebug An alternative (which they are just as likely to recommend, if they respond to your request) is to use bugtags for this purpose. Those can have whatever names you want and you can still query based on them. It's worth noting that the subjective and non-project-agnostic nature of severity/importance is what led StoryBoard to not include fields for them, opting instead for a combination of extensible worklists (where relative priority can be indicated by manually ranking different stories, and you can have multiple worklists which rank them differently according to a variety of arbitrarily-chosen metrics or individual opinions) and storytags (which can have whatever names you want and are also usable to populate automatic worklists and board lanes). -- Jeremy Stanley
Folks, As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map to our Critical, High, Medium, Low and Undecided. This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed for a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone. Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs without prior approval of the Triage team. Medium+ -- Bug fixing is planned before the milestone and must be fixed or have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience. Medium -- These are important issues we keep track and try to plan fixing for the release. They have limited impact for the system functions and releases. Low -- Bug fixing is only done opportunistically. Generally not planned for the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category. Undecided -- These issues are newly reported and are undecided before Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it. Note: High impact but Low Priority bugs can be documented in the release notes.
The Severity indicates how much the issue impacted the person reporting the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components, memory leak etc. Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain circumstance. This is the default Severity. Minor -- Minor loss of functionality, or issues with easy workaround available. Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project. Sau! [0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking#I... On 7/17/19 3:41 AM, Zvonar, Bill wrote:
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance (which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels for defects in different domains?
Once we have those agreed and written down somewhere, they can be used as guidance for people that are opening Launchpads, and for those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain, we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; starlingx-discuss@lists.starlingx.io; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4 priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map to our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs. Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed for a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone. Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs without prior approval of the Triage team. Medium+ -- Bug fixing is planned before the milestone and must be fixed or have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience. Medium -- These are important issues we keep track and try to plan fixing for the release. They have limited impact for the system functions and releases. Low -- Bug fixing is only done opportunistically. Generally not planned for the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category. Undecided -- These issues are newly reported and are undecided before Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it. Note: High impact but Low Priority bugs can be documented in the release notes.
The Severity indicates how much the issue impacted the person reporting the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components, memory leak etc. Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain circumstance. This is the default Severity. Minor -- Minor loss of functionality, or issues with easy workaround available. Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking #Importance
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance (which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels for defects in different domains?
Once we have those agreed and written down somewhere, they can be used as guidance for people that are opening Launchpads, and for those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain, we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; starlingx- discuss@lists.starlingx.io; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and priority together,
On 7/17/19 3:41 AM, Zvonar, Bill wrote: then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand
that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io;
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a
P4 - Low Resolution of the defect has least urgency to resolve, P4
Khalil, Ghada <Ghada.Khalil@windriver.com> planned timeframe for a verified resolution. priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Jose, Just to clarify: for the weekly bug triage meeting, you only ask to triage the new bugs, right? My concern is about the triage frequency: right now, the new bugs are triaged almost on daily basis, mostly by Ghada by consulting technical expert. If we switch to a triage meeting, now sure how the new LP can be handled timely. But agree that having a triage meeting is a good idea. Thx. - cindy -----Original Message----- From: Perez Carranza, Jose [mailto:jose.perez.carranza@intel.com] Sent: Friday, July 19, 2019 8:16 PM To: Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map to our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs. Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed for Medium+a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone. Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs without prior approval of the Triage team. Medium+ -- Bug fixing is planned before the milestone and must be Medium+ fixed or have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience. Medium -- These are important issues we keep track and try to plan fixing for the release. They have limited impact for the system functions and releases. Low -- Bug fixing is only done opportunistically. Generally not planned for the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category. Undecided -- These issues are newly reported and are undecided before Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it. Note: High impact but Low Priority bugs can be documented in the release notes.
The Severity indicates how much the issue impacted the person reporting the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components, memory leak etc. Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain circumstance. This is the default Severity. Minor -- Minor loss of functionality, or issues with easy workaround available. Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Trac king #Importance
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance (which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels for defects in different domains?
Once we have those agreed and written down somewhere, they can be used as guidance for people that are opening Launchpads, and for those
On 7/17/19 3:41 AM, Zvonar, Bill wrote: that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain,
we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that
more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; starlingx-
Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand
discuss@lists.starlingx.io; Khalil, Ghada <Ghada.Khalil@windriver.com> that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io;
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest
P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4
Khalil, Ghada <Ghada.Khalil@windriver.com> possible timeframe. priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
-----Original Message----- From: Xie, Cindy Sent: Friday, July 19, 2019 9:35 AM To: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
Jose, Just to clarify: for the weekly bug triage meeting, you only ask to triage the new bugs, right?
Yes, only the new ones should be triaged.
My concern is about the triage frequency: right now, the new bugs are triaged almost on daily basis, mostly by Ghada by consulting technical expert. If we switch to a triage meeting, now sure how the new LP can be handled timely.
But agree that having a triage meeting is a good idea. Thx. - cindy
To mitigate this concern as Saul pointed out we should ensure to have a "triage section"section on subproject meeting but ensuring all the stakeholders for the specific bugs are online to provide feedback.
-----Original Message----- From: Perez Carranza, Jose [mailto:jose.perez.carranza@intel.com] Sent: Friday, July 19, 2019 8:16 PM To: Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map to our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs.
Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed for Medium+a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone. Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs
without prior approval of the Triage team.
Medium+ -- Bug fixing is planned before the milestone and must be Medium+ fixed or have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience. Medium -- These are important issues we keep track and try to plan fixing for the release. They have limited impact for the system functions and releases. Low -- Bug fixing is only done opportunistically. Generally not planned for the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category. Undecided -- These issues are newly reported and are undecided before Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it. Note: High impact but Low Priority bugs can be documented in the release notes.
The Severity indicates how much the issue impacted the person reporting the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components, memory leak etc. Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain circumstance. This is the default Severity. Minor -- Minor loss of functionality, or issues with easy workaround available. Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Trac king #Importance
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance (which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels for defects in different domains?
Once we have those agreed and written down somewhere, they can be used as guidance for people that are opening Launchpads, and for those
On 7/17/19 3:41 AM, Zvonar, Bill wrote: that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain,
we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that
more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; starlingx-
Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and priority together, then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand
discuss@lists.starlingx.io; Khalil, Ghada <Ghada.Khalil@windriver.com> that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io;
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest
P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4
Khalil, Ghada <Ghada.Khalil@windriver.com> possible timeframe. priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Another idea is to using mailing list: each day, triage lead sends out a list of "new" bugs need triage and sub-project leads response in mailing list so that we keep the information public, we can assign bugs to appropriate owners (or people volunteer). Thx. - cindy -----Original Message----- From: Perez Carranza, Jose Sent: Friday, July 19, 2019 11:02 PM To: Xie, Cindy <cindy.xie@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Xie, Cindy Sent: Friday, July 19, 2019 9:35 AM To: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
Jose, Just to clarify: for the weekly bug triage meeting, you only ask to triage the new bugs, right?
Yes, only the new ones should be triaged.
My concern is about the triage frequency: right now, the new bugs are triaged almost on daily basis, mostly by Ghada by consulting technical expert. If we switch to a triage meeting, now sure how the new LP can be handled timely.
But agree that having a triage meeting is a good idea. Thx. - cindy
To mitigate this concern as Saul pointed out we should ensure to have a "triage section"section on subproject meeting but ensuring all the stakeholders for the specific bugs are online to provide feedback.
-----Original Message----- From: Perez Carranza, Jose [mailto:jose.perez.carranza@intel.com] Sent: Friday, July 19, 2019 8:16 PM To: Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map to our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs.
Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed Medium+for a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone. Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs
without prior approval of the Triage team.
Medium+ -- Bug fixing is planned before the milestone and must be Medium+ fixed or have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience. Medium -- These are important issues we keep track and try to plan fixing for the release. They have limited impact for the system functions and releases. Low -- Bug fixing is only done opportunistically. Generally not planned for the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category. Undecided -- These issues are newly reported and are undecided before Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it. Note: High impact but Low Priority bugs can be documented in the release notes.
The Severity indicates how much the issue impacted the person reporting the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components, memory leak etc. Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain circumstance. This is the default Severity. Minor -- Minor loss of functionality, or issues with easy workaround available. Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tr ac king #Importance
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance (which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels for defects in different domains?
Once we have those agreed and written down somewhere, they can be used as guidance for people that are opening Launchpads, and for
On 7/17/19 3:41 AM, Zvonar, Bill wrote: those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain,
we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that
more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; starlingx-
Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and
discuss@lists.starlingx.io; Khalil, Ghada <Ghada.Khalil@windriver.com> priority together, then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand
that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io;
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4
Khalil, Ghada <Ghada.Khalil@windriver.com> priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discu ss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
As discussed in the Community call today, Ghada & I are proposing the following handling of bugs of the different priorities ("Importance" in Launchpad) as follows. This will be discussed again tomorrow during the Release Team meeting as well... - Critical: - must fix by release date (Release Build: Aug 23) - High: - must be fixed for the stx.2.0 release, but could be fixed *after* the release date (in a maintenance release) - fixes will be backported to stx.2.0 - Medium: - continue working until release date - fix as many as possible - defer to 3.0 after release date - fixes will be backported between RC1 and the release date - fixes will not be backported to stx.2.0 after the release date - Low: - optional, will be deferred to stx.3.0 at RC1, fixes will not be backported to stx.2.0 Bill... -----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Friday, July 19, 2019 7:56 PM To: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority Another idea is to using mailing list: each day, triage lead sends out a list of "new" bugs need triage and sub-project leads response in mailing list so that we keep the information public, we can assign bugs to appropriate owners (or people volunteer). Thx. - cindy -----Original Message----- From: Perez Carranza, Jose Sent: Friday, July 19, 2019 11:02 PM To: Xie, Cindy <cindy.xie@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Xie, Cindy Sent: Friday, July 19, 2019 9:35 AM To: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
Jose, Just to clarify: for the weekly bug triage meeting, you only ask to triage the new bugs, right?
Yes, only the new ones should be triaged.
My concern is about the triage frequency: right now, the new bugs are triaged almost on daily basis, mostly by Ghada by consulting technical expert. If we switch to a triage meeting, now sure how the new LP can be handled timely.
But agree that having a triage meeting is a good idea. Thx. - cindy
To mitigate this concern as Saul pointed out we should ensure to have a "triage section"section on subproject meeting but ensuring all the stakeholders for the specific bugs are online to provide feedback.
-----Original Message----- From: Perez Carranza, Jose [mailto:jose.perez.carranza@intel.com] Sent: Friday, July 19, 2019 8:16 PM To: Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map to our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs.
Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed Medium+for a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone. Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs
without prior approval of the Triage team.
Medium+ -- Bug fixing is planned before the milestone and must be Medium+ fixed or have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience. Medium -- These are important issues we keep track and try to plan fixing for the release. They have limited impact for the system functions and releases. Low -- Bug fixing is only done opportunistically. Generally not planned for the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category. Undecided -- These issues are newly reported and are undecided before Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it. Note: High impact but Low Priority bugs can be documented in the release notes.
The Severity indicates how much the issue impacted the person reporting the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components, memory leak etc. Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain circumstance. This is the default Severity. Minor -- Minor loss of functionality, or issues with easy workaround available. Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tr ac king #Importance
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance (which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels for defects in different domains?
Once we have those agreed and written down somewhere, they can be used as guidance for people that are opening Launchpads, and for
On 7/17/19 3:41 AM, Zvonar, Bill wrote: those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain,
we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that
more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; starlingx-
Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and
discuss@lists.starlingx.io; Khalil, Ghada <Ghada.Khalil@windriver.com> priority together, then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand
that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io;
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4
Khalil, Ghada <Ghada.Khalil@windriver.com> priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discu ss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Thanks Bill and Ghada drive the discussion thread to a closure. I think this is a great balance btw "time-based release" strategy we agreed earlier and also put quality as top priority. Thanks! - cindy -----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 25, 2019 4:18 AM To: Xie, Cindy <cindy.xie@intel.com>; Perez Carranza, Jose <jose.perez.carranza@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority As discussed in the Community call today, Ghada & I are proposing the following handling of bugs of the different priorities ("Importance" in Launchpad) as follows. This will be discussed again tomorrow during the Release Team meeting as well... - Critical: - must fix by release date (Release Build: Aug 23) - High: - must be fixed for the stx.2.0 release, but could be fixed *after* the release date (in a maintenance release) - fixes will be backported to stx.2.0 - Medium: - continue working until release date - fix as many as possible - defer to 3.0 after release date - fixes will be backported between RC1 and the release date - fixes will not be backported to stx.2.0 after the release date - Low: - optional, will be deferred to stx.3.0 at RC1, fixes will not be backported to stx.2.0 Bill... -----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Friday, July 19, 2019 7:56 PM To: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority Another idea is to using mailing list: each day, triage lead sends out a list of "new" bugs need triage and sub-project leads response in mailing list so that we keep the information public, we can assign bugs to appropriate owners (or people volunteer). Thx. - cindy -----Original Message----- From: Perez Carranza, Jose Sent: Friday, July 19, 2019 11:02 PM To: Xie, Cindy <cindy.xie@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Xie, Cindy Sent: Friday, July 19, 2019 9:35 AM To: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] bug severity and priority
Jose, Just to clarify: for the weekly bug triage meeting, you only ask to triage the new bugs, right?
Yes, only the new ones should be triaged.
My concern is about the triage frequency: right now, the new bugs are triaged almost on daily basis, mostly by Ghada by consulting technical expert. If we switch to a triage meeting, now sure how the new LP can be handled timely.
But agree that having a triage meeting is a good idea. Thx. - cindy
To mitigate this concern as Saul pointed out we should ensure to have a "triage section"section on subproject meeting but ensuring all the stakeholders for the specific bugs are online to provide feedback.
-----Original Message----- From: Perez Carranza, Jose [mailto:jose.perez.carranza@intel.com] Sent: Friday, July 19, 2019 8:16 PM To: Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map to our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs.
Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed Medium+for a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone. Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs
without prior approval of the Triage team.
Medium+ -- Bug fixing is planned before the milestone and must be Medium+ fixed or have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience. Medium -- These are important issues we keep track and try to plan fixing for the release. They have limited impact for the system functions and releases. Low -- Bug fixing is only done opportunistically. Generally not planned for the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category. Undecided -- These issues are newly reported and are undecided before Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it. Note: High impact but Low Priority bugs can be documented in the release notes.
The Severity indicates how much the issue impacted the person reporting the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components, memory leak etc. Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain circumstance. This is the default Severity. Minor -- Minor loss of functionality, or issues with easy workaround available. Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tr ac king #Importance
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance (which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels for defects in different domains?
Once we have those agreed and written down somewhere, they can be used as guidance for people that are opening Launchpads, and for
On 7/17/19 3:41 AM, Zvonar, Bill wrote: those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain,
we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that
more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; starlingx-
Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and
discuss@lists.starlingx.io; Khalil, Ghada <Ghada.Khalil@windriver.com> priority together, then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand
that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io;
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a planned timeframe for a verified resolution. P4 - Low Resolution of the defect has least urgency to resolve, P4
Khalil, Ghada <Ghada.Khalil@windriver.com> priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discu ss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 7/19/19 5:15 AM, Perez Carranza, Jose wrote:
-----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Thursday, July 18, 2019 10:57 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bug severity and priority
Folks,
As I mentioned in a prior email about a previous project (Yocto Project), we were also time-based (every 6 months). We defined Importance [0] of the bug based on Severity (chosen by submitter) and Priority (assigned during a triage process). We had 5 Priory levels in Bugzilla: High, Medium+, Medium, Low and Undecided, these would map to our Critical, High, Medium, Low and Undecided.
Those triage meetings were very helpful because they were live discussions about the bugs with all the stockholders. I think we should consider to have a weekly meeting just to triage bugs.
Not sure we need a bug triage meeting, that should be happening in the sub-team meetings. Sau!
Regards, José
This clearly frames it based on Milestones and releases due to the time based nature of the Yocto Project. Notice that the High/Critical is the only one that is truly "gating" or milestone/release blocker, the Medium+, our High, won't block a milestone but be should be fixed for a release, but could be a dot.dot soon after the release.
Importance The Importance of the bug is defined by its Priority and Severity. The Priority classifies the bug's fixing order. In other words, how soon will it get fixed relative to other bugs? Priorities are set during the bug Triage meeting and cannot be changed by the user. The priority appears to the left of the Severity field. Here are the values that Priority can be set to during the Triage meeting:
High -- Bug fixing is planned immediately for the target milestone. Milestone cannot be released if there is a high bug opened against the milestone. High priority issues cause major functional loss of a specific feature that is POR for the up-comping milestone. These issues are easily hit by the user and greatly impact the user experience or customer requirements. Finally, these issues could be urgent security fixes that need to be corrected in a prior release. The bug assignee is not to change the target milestones for High bugs without prior approval of the Triage team. Medium+ -- Bug fixing is planned before the milestone and must be fixed or have a solution planned before the release is finalized. These issues are not show-stoppers but have somewhat significant impact to system functions and user experience. Medium -- These are important issues we keep track and try to plan fixing for the release. They have limited impact for the system functions and releases. Low -- Bug fixing is only done opportunistically. Generally not planned for the up-coming project release. Issues that are not a POR feature request, or are hard to reproduce fall into this category. Undecided -- These issues are newly reported and are undecided before Triage. Issues that are a feature request, which isn't approved for future release yet. This issue will be changed to have an actual Priority after the Triage team approves it. Note: High impact but Low Priority bugs can be documented in the release notes.
The Severity indicates how much the issue impacted the person reporting the bug. Severity can be categorized into five areas.
Critical -- Crashes, hang, loss of data, negative impact to other components, memory leak etc. Major -- Major loss of functionality of POR. Normal -- Regular issue, some loss of functionality under certain circumstance. This is the default Severity. Minor -- Minor loss of functionality, or issues with easy workaround available. Enhancement -- Request for enhancement or new feature to be worked.
I hope the helps by provide a different viewpoint from another project.
Sau!
[0] https://wiki.yoctoproject.org/wiki/Bugzilla_Configuration_and_Bug_Tracking #Importance
Hi Cindy,
Thought about this some more, sorry it took me so long to respond further.
I agree with splitting out the definitions of release priority/importance (which is subjective) from the technical severity (which is I'd say much less subjective).
Do we agree that one of the key next steps is to define the severity levels for defects in different domains?
Once we have those agreed and written down somewhere, they can be used as guidance for people that are opening Launchpads, and for those that screen them. Someone will note that some bugs cross domains, so it's not as simple as looking at one set of severity definitions, but let's cross that bridge next.
Then, if we've got general alignment on the severity definitions per domain, we can sort out what to use as a QRC formula for a release, I think.
Btw, it'd be nice if Launchpad had a field for Severity, so we could track that more easily - does anybody know if we can just request this & get it added as a custom field?
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 7:13 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; starlingx- discuss@lists.starlingx.io; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: RE: bug severity and priority
Bill, I definitely agree that not all Medium shall be pushed to stx.3.0, this needs to be assessed carefully. But if we combine the severity and priority together,
On 7/17/19 3:41 AM, Zvonar, Bill wrote: then this decision needs to put resource factor in consideration as well.
Actually, I think it's confusing of calling individual LP "gating" - I understand
that we want to get the product quality to a good shape and want to get bugs fixed as many as possible before we ship it. I will suggest to use defects# as part of release criteria (QRC). Example could be:
Number of Critical P1 defects Zero Number of High P2 defects < x Number of Medium P3 defects < y
And the only thing we need to agree on is the "x" and "y". It makes TSC or
release team to make decision easier. The QRC needs to be agreed earlier instead of right before the release decision shall be made. This way, we can really direct our engineering resource working on the most important items and we all have an agreed common goal.
Thanks. - cindy
-----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, July 11, 2019 1:39 AM To: Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io;
Subject: RE: bug severity and priority
Hi Cindy,
Thanks for sending this, I think this gives us something to start the discussion.
However we decide to align on severity/priority (I'll comment on that more later, need to think about it more), I think we need to be careful before we move all mediums to 3.0, it may be too much of a Gordian knot solution.
I think we need to assess the mediums (as Yong suggested earlier) to say why they should or should not be in 2.0. I also think this may help us sort out what our gating criteria are.
Bill...
-----Original Message----- From: Xie, Cindy <cindy.xie@intel.com> Sent: Wednesday, July 10, 2019 10:42 AM To: starlingx-discuss@lists.starlingx.io; Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Subject: bug severity and priority
Bill/Ghada, I am sending out my definition of bug severity and priority:
Bug Exposure or Severity Definition 1- Critical Product or key feature is not usable for intended purpose. 2- High Product or key feature is not reliably usable for intended purpose or use is significantly impaired 3 - Medium Product or key feature is usable provided by a workaround 4 - Low Tolerable impact to user experience with minimal service and support costs
Bug Priority Definition P1 - Stopper Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum development team resources to resolve a defect in the shortest possible timeframe. P2 - High Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. The urgency to fix a P2 priority defect is imminent. - P2 priority defects are intended to be resolved by the next planned external release of the software. P3 - Medium Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. - P3 priority defects must have a
P4 - Low Resolution of the defect has least urgency to resolve, P4
Khalil, Ghada <Ghada.Khalil@windriver.com> planned timeframe for a verified resolution. priority defects may or may not have plans to resolve.
Let's discuss this and agree how we'd like to use them. My suggestion for
current "Medium" is to we can mark them as "stx.3.0" and then in the beginning of stx.3, they can move Priority to "high" due to the fact they want to get them fixed in 3.0.
But the bug severity should never change because they are standard.
Thx. - cindy
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (5)
-
Jeremy Stanley
-
Perez Carranza, Jose
-
Saul Wold
-
Xie, Cindy
-
Zvonar, Bill