[Starlingx-discuss] DRAFT release policy
We've been working on a draft release policy in the Release team. Please review and share your comments, feedback or suggestions in the etherpad or on this thread. The draft is located at https://etherpad.openstack.org/p/stx-release-policy-draft The current text is below. Brucej Release Policy =========== The StarlingX project determines release schedules based on when the release content is ready to be released. We should explicitly move away from the idea of time based releases. We should continue our current twice per year release cycle * We should consider changing our release naming convention to something that doesn't include dates. The Release team, together with the Test team TLs/PLs, shall make a recommendation to the community and TSC that a release is ready to go. Upon TSC approval, the release branches are tagged and the release documented. That recommendation should be based on: 1. Whether or not all anchor features in the release are complete, as per the input of the team(s) implementing the features and the results of Test team testing of the features ? Assumption: The TSC will create a list of approved features during the release planning process, and that some (or all) of these features will be identified by the TSC as "release gating" or "anchor" features that would block a release if not complete. ? Assumption: We are planning our releases at the PTG meetings every 6 months. 2. The results of formal Testing performed by the Test team, measured by the percentage of planned tests attempted and the test pass rate ? Proposal: 100% test cases attemped and 95% test cases passing in all configurations 3. The severity and number of bugs open against the release ? Proposal: No open Critical or High severity bugs against the release candidate. Or maybe 1-3 Highs if we have a clear resolution plan (and a plan to release a patch against the release?) ? See Bug Severity definitions below. We should discuss the option of doing a mid-cycle "bug fix" release against the latest release - allowing key defects to get fixed more rapidly. In such a case, the release team would make a recommendation to the TSC to approve the release plan and content. If approved by the TSC, the fixes shall be cherry-picked or merged into the release branch and a new build tested (and managed as per this policy). Bug Severity ========== Critical: The software does not operate as intended (e.g. the software fails to install, does not run, crashes, etc...). Or the issue is a Securty/CVE issue with a rating of Critical or High. In Launchpad this is "Fix as soon as possible" High: An important feature of the software does not operate as intended (e.g. live migration, fault reporting, etc...). Or the issue is a Security/CVE issue with a rating of Medium or lower. In Launchpad this is "Fix soon". Medium: A minor feature of the software does not operate as intended or there is an intermitant failure. In Launchpad this is "Fix when convienent or schedule to fix later". Low: A bug that does not impact normal operation of the software. In Launchpad this is "Fix when convienent".
Bruce, Release team; Thank you for putting this in writing. I have to admit I find the basic strategy statements confusing: 1- We need to move away from time-based releases 2- We need to do twice a year releases. This stmt, by definition, implies a time-gated release. Maybe it isn't a specific date, but it is still time-gated. As a nascent project, I think we need to show gradual and consistent progress. I did listen to much of the release team meeting today, and realize the trade-offs between big-rocks and timing are very difficult. Given the difficult choice of functionality versus timing, I personally think we need to show progress in getting to Stein and a container based distribution as major milestones in 1H and perhaps defer the Distributed Cloud capability to a 2H release. I don't see anything intrinsically wrong with moving a specific date out; it happens all the time. But I also think a release should have some gate; i.e. we don't move out of 1H. And if some functionality isn't ready, then we move the functionality to another release in 2H. Anyway, that would be my vote, if I have one. From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Thursday, March 28, 2019 12:49 PM To: starlingx-discuss Subject: [Starlingx-discuss] DRAFT release policy We've been working on a draft release policy in the Release team. Please review and share your comments, feedback or suggestions in the etherpad or on this thread. The draft is located at https://etherpad.openstack.org/p/stx-release-policy-draft The current text is below. Brucej Release Policy =========== The StarlingX project determines release schedules based on when the release content is ready to be released. We should explicitly move away from the idea of time based releases. We should continue our current twice per year release cycle * We should consider changing our release naming convention to something that doesn't include dates. The Release team, together with the Test team TLs/PLs, shall make a recommendation to the community and TSC that a release is ready to go. Upon TSC approval, the release branches are tagged and the release documented. That recommendation should be based on: 1. Whether or not all anchor features in the release are complete, as per the input of the team(s) implementing the features and the results of Test team testing of the features * Assumption: The TSC will create a list of approved features during the release planning process, and that some (or all) of these features will be identified by the TSC as "release gating" or "anchor" features that would block a release if not complete. * Assumption: We are planning our releases at the PTG meetings every 6 months. 2. The results of formal Testing performed by the Test team, measured by the percentage of planned tests attempted and the test pass rate * Proposal: 100% test cases attemped and 95% test cases passing in all configurations 3. The severity and number of bugs open against the release * Proposal: No open Critical or High severity bugs against the release candidate. Or maybe 1-3 Highs if we have a clear resolution plan (and a plan to release a patch against the release?) * See Bug Severity definitions below. We should discuss the option of doing a mid-cycle "bug fix" release against the latest release - allowing key defects to get fixed more rapidly. In such a case, the release team would make a recommendation to the TSC to approve the release plan and content. If approved by the TSC, the fixes shall be cherry-picked or merged into the release branch and a new build tested (and managed as per this policy). Bug Severity ========== Critical: The software does not operate as intended (e.g. the software fails to install, does not run, crashes, etc...). Or the issue is a Securty/CVE issue with a rating of Critical or High. In Launchpad this is "Fix as soon as possible" High: An important feature of the software does not operate as intended (e.g. live migration, fault reporting, etc...). Or the issue is a Security/CVE issue with a rating of Medium or lower. In Launchpad this is "Fix soon". Medium: A minor feature of the software does not operate as intended or there is an intermitant failure. In Launchpad this is "Fix when convienent or schedule to fix later". Low: A bug that does not impact normal operation of the software. In Launchpad this is "Fix when convienent".
On Thu, Mar 28, 2019 at 6:39 PM Seiler, Glenn <glenn.seiler@windriver.com> wrote:
1- We need to move away from time-based releases 2- We need to do twice a year releases. This stmt, by definition, implies a time-gated release. Maybe it isn’t a specific date, but it is still time-gated.
The wording does need work, yes. After a short conversation with Bruce this afternoon (Bruce, correct me if I'm wrong here) I came away with the intention being more of increasing the lag from OpenStack releases rather than separating completely from the OpenStack release cycle which is likely to stay at approx 6 months for a while (that's a rabbit hole under the bike shed I'd like to avoid just now).
As a nascent project, I think we need to show gradual and consistent progress.
++
I did listen to much of the release team meeting today, and realize the trade-offs between big-rocks and timing are very difficult.
Given the difficult choice of functionality versus timing, I personally think we need to show progress in getting to Stein and a container based distribution as major milestones in 1H and perhaps defer the Distributed Cloud capability to a 2H release.
I don’t see anything intrinsically wrong with moving a specific date out; it happens all the time. But I also think a release should have some gate; i.e. we don’t move out of 1H. And if some functionality isn’t ready, then we move the functionality to another release in 2H.
We took a stab at estimating and missed, making adjustments now is normal and to be expected. I agree with considering pushing distcloud to the next release because it is a) new functionality, and b) devs overlap with the container work and I think making the k8s infrastructure rock solid is much more important. If we are too far off with system stability the ramifications will be harder to overcome than delaying a new feature.
Anyway, that would be my vote, if I have one.
You totally have a voice as part of the community, I would like to hear from more folks here... dt -- Dean Troyer dtroyer@gmail.com
Dean, Glenn - thank you for the feedback. I agree with it. There is also some feedback in the etherpad. I'm going to respond to both sets in the etherpad and try to improve the policy and the wording. https://etherpad.openstack.org/p/stx-release-policy-draft Thanks! brucej -----Original Message----- From: Dean Troyer [mailto:dtroyer@gmail.com] Sent: Thursday, March 28, 2019 7:29 PM To: Seiler, Glenn <glenn.seiler@windriver.com> Cc: Jones, Bruce E <bruce.e.jones@intel.com>; starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy On Thu, Mar 28, 2019 at 6:39 PM Seiler, Glenn <glenn.seiler@windriver.com> wrote:
1- We need to move away from time-based releases 2- We need to do twice a year releases. This stmt, by definition, implies a time-gated release. Maybe it isn’t a specific date, but it is still time-gated.
The wording does need work, yes. After a short conversation with Bruce this afternoon (Bruce, correct me if I'm wrong here) I came away with the intention being more of increasing the lag from OpenStack releases rather than separating completely from the OpenStack release cycle which is likely to stay at approx 6 months for a while (that's a rabbit hole under the bike shed I'd like to avoid just now).
As a nascent project, I think we need to show gradual and consistent progress.
++
I did listen to much of the release team meeting today, and realize the trade-offs between big-rocks and timing are very difficult.
Given the difficult choice of functionality versus timing, I personally think we need to show progress in getting to Stein and a container based distribution as major milestones in 1H and perhaps defer the Distributed Cloud capability to a 2H release.
I don’t see anything intrinsically wrong with moving a specific date out; it happens all the time. But I also think a release should have some gate; i.e. we don’t move out of 1H. And if some functionality isn’t ready, then we move the functionality to another release in 2H.
We took a stab at estimating and missed, making adjustments now is normal and to be expected. I agree with considering pushing distcloud to the next release because it is a) new functionality, and b) devs overlap with the container work and I think making the k8s infrastructure rock solid is much more important. If we are too far off with system stability the ramifications will be harder to overcome than delaying a new feature.
Anyway, that would be my vote, if I have one.
You totally have a voice as part of the community, I would like to hear from more folks here... dt -- Dean Troyer dtroyer@gmail.com
I've updated the etherpad with changes that reflect the current feedback. Please review and add any additional feedback there. Thank you! https://etherpad.openstack.org/p/stx-release-policy-draft brucej -----Original Message----- From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Friday, March 29, 2019 10:06 AM To: Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy Dean, Glenn - thank you for the feedback. I agree with it. There is also some feedback in the etherpad. I'm going to respond to both sets in the etherpad and try to improve the policy and the wording. https://etherpad.openstack.org/p/stx-release-policy-draft Thanks! brucej -----Original Message----- From: Dean Troyer [mailto:dtroyer@gmail.com] Sent: Thursday, March 28, 2019 7:29 PM To: Seiler, Glenn <glenn.seiler@windriver.com> Cc: Jones, Bruce E <bruce.e.jones@intel.com>; starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy On Thu, Mar 28, 2019 at 6:39 PM Seiler, Glenn <glenn.seiler@windriver.com> wrote:
1- We need to move away from time-based releases 2- We need to do twice a year releases. This stmt, by definition, implies a time-gated release. Maybe it isn’t a specific date, but it is still time-gated.
The wording does need work, yes. After a short conversation with Bruce this afternoon (Bruce, correct me if I'm wrong here) I came away with the intention being more of increasing the lag from OpenStack releases rather than separating completely from the OpenStack release cycle which is likely to stay at approx 6 months for a while (that's a rabbit hole under the bike shed I'd like to avoid just now).
As a nascent project, I think we need to show gradual and consistent progress.
++
I did listen to much of the release team meeting today, and realize the trade-offs between big-rocks and timing are very difficult.
Given the difficult choice of functionality versus timing, I personally think we need to show progress in getting to Stein and a container based distribution as major milestones in 1H and perhaps defer the Distributed Cloud capability to a 2H release.
I don’t see anything intrinsically wrong with moving a specific date out; it happens all the time. But I also think a release should have some gate; i.e. we don’t move out of 1H. And if some functionality isn’t ready, then we move the functionality to another release in 2H.
We took a stab at estimating and missed, making adjustments now is normal and to be expected. I agree with considering pushing distcloud to the next release because it is a) new functionality, and b) devs overlap with the container work and I think making the k8s infrastructure rock solid is much more important. If we are too far off with system stability the ramifications will be harder to overcome than delaying a new feature.
Anyway, that would be my vote, if I have one.
You totally have a voice as part of the community, I would like to hear from more folks here... dt -- Dean Troyer dtroyer@gmail.com _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Bruce, I have a comment regarding this point: The severity and number of bugs open against the release Proposal: No open Critical or High severity bugs against the release candidate. Or maybe 1-3 Highs if we have a clear resolution plan (and a plan to release a patch against the release?) (ghada) We use an explicit tag to identify which bugs gate a particular release (regardless of severity). The whole list will have to be reviewed and scrubbed prior to reaching the release milestone. I don't feel it is sufficient to only review Critical / Major issues. [Example: On April 1/2019, there are 85 release gating bugs: only 13 are Critical/High. Yet it wouldn't be sufficient to only fix those 13 to ensure a quality release]. In the Release Planning wiki[1] , we have previously stated this policy: All release gating issues are addressed or reviewed/accepted for deferral I feel we need to keep this. [1] https://wiki.openstack.org/wiki/StarlingX/Release_Plan -----Original Message----- From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Friday, March 29, 2019 2:10 PM To: Jones, Bruce E; Dean Troyer; Seiler, Glenn Cc: starlingx-discuss Subject: Re: [Starlingx-discuss] DRAFT release policy I've updated the etherpad with changes that reflect the current feedback. Please review and add any additional feedback there. Thank you! https://etherpad.openstack.org/p/stx-release-policy-draft brucej -----Original Message----- From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Friday, March 29, 2019 10:06 AM To: Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy Dean, Glenn - thank you for the feedback. I agree with it. There is also some feedback in the etherpad. I'm going to respond to both sets in the etherpad and try to improve the policy and the wording. https://etherpad.openstack.org/p/stx-release-policy-draft Thanks! brucej -----Original Message----- From: Dean Troyer [mailto:dtroyer@gmail.com] Sent: Thursday, March 28, 2019 7:29 PM To: Seiler, Glenn <glenn.seiler@windriver.com> Cc: Jones, Bruce E <bruce.e.jones@intel.com>; starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy On Thu, Mar 28, 2019 at 6:39 PM Seiler, Glenn <glenn.seiler@windriver.com> wrote:
1- We need to move away from time-based releases 2- We need to do twice a year releases. This stmt, by definition, implies a time-gated release. Maybe it isn’t a specific date, but it is still time-gated.
The wording does need work, yes. After a short conversation with Bruce this afternoon (Bruce, correct me if I'm wrong here) I came away with the intention being more of increasing the lag from OpenStack releases rather than separating completely from the OpenStack release cycle which is likely to stay at approx 6 months for a while (that's a rabbit hole under the bike shed I'd like to avoid just now).
As a nascent project, I think we need to show gradual and consistent progress.
++
I did listen to much of the release team meeting today, and realize the trade-offs between big-rocks and timing are very difficult.
Given the difficult choice of functionality versus timing, I personally think we need to show progress in getting to Stein and a container based distribution as major milestones in 1H and perhaps defer the Distributed Cloud capability to a 2H release.
I don’t see anything intrinsically wrong with moving a specific date out; it happens all the time. But I also think a release should have some gate; i.e. we don’t move out of 1H. And if some functionality isn’t ready, then we move the functionality to another release in 2H.
We took a stab at estimating and missed, making adjustments now is normal and to be expected. I agree with considering pushing distcloud to the next release because it is a) new functionality, and b) devs overlap with the container work and I think making the k8s infrastructure rock solid is much more important. If we are too far off with system stability the ramifications will be harder to overcome than delaying a new feature.
Anyway, that would be my vote, if I have one.
You totally have a voice as part of the community, I would like to hear from more folks here... dt -- Dean Troyer dtroyer@gmail.com _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Good feedback, Ghada, thank you. I have updated the document accordingly. I've also made a few other changes like adding test case readiness as a release criteria and proposed a method for handing anchor features that aren't completed by MS-3. I've removed the previously resolved (and much appreciated) feedback in the interests of readability. Link and updated text below. Brucej This file is: https://etherpad.openstack.org/p/stx-release-policy-draft This is a draft document for release planning. Comments and feedback welcomed! Openstack Release Policy ==================== The StarlingX project follows the release model defined in https://docs.openstack.org/project-team-guide/release-management.html using the "Trailing the common cycle" due to our dependency on upstream OpenStack projects. Release Planning ============== Initial release planning starts at the Open Infrastructure PTG meetings, where the TSC and community members discuss candidate features for the next release. The TSC then will review and approve a feature list for the release, will identify any release gating "anchor" features, and set a target date. The recommended target date is the date of the next OpenStack release plus 6 weeks. The PTG meeting is also an opportunity to review the community's goals and to define goals for the release. The overall Release Plan is created and managed by the Release sub-team by combining the TSC's input on content and target dates with input from the feature developers in the community and the Test team. The plan will include a standard set of milestones as per the usual OpenStack release management process. The Release team will actively manage the plan over the course of the release, recommending any adjustments in content and dates to the community and to the TSC for approval. We recognize that we are a new community working in a highly dynamic technology and that changes in our plans over time are normal and expected. We will work as a community to be open and transparent about our release process, and to minimize change from the original plan. Open issue: We should consider changing our release naming convention to something that isn't a date. Defect Tracking ============ The release team shall review active and incoming bug reports and make an initial call as to whether or not the bug needs to be fixed in the next release. If so, the bugs shall be tagged and tracked as the work on the release progresses. The list of release gating bugs will be actively managed, reviewed and scrubbed by the Release team to ensure that bugs are properly categorized as release gating. Release Policy =========== The Release team, together with the Test team TLs/PLs, shall make a recommendation to the community and TSC that a release is ready to go. Upon TSC approval, the release branches are tagged and the release documented. That recommendation should be based on: * Whether or not all anchor features in the release are complete, as per the input of the team(s) implementing the features and the results of Test team testing of the features * Proposal: All features identified as anchor features for a release need to be completed by the feature freeze milestone (MS-3). In the event that an anchor feature is not complete before the release feature freeze milestone, the Release team will make a recommendation to the TSC to extend the milestone date or to defer the feature to the next release. * Whether or not all test cases planned for the release are complete and ready to run * Proposal: All planned test cases shall be ready before the start of formal release candidate testing (RC1 milestone) * The completion and results of formal Testing performed by the Test team, measured by the percentage of planned tests attempted and the test pass rate * Proposal: 100% test cases attempted and 95% test cases passing in all configurations * The status of release gating bugs * Proposal: All release gating bugs must be fixed prior to a StarlingX release, ideally before the RC1 milestone but certainly before the final release. Bugfix Releases ============= The Release team, in conjunction with the community, can create a plan for a bug fix release. This would be an update to a previous release to address important defects that are impacting our users. The content would be fixes backported from master to the release branch, and would be based on both community and developer input regarding which fixes should be included. Testing of a bug fix release should include at least verification testing of the fixes and any additional testing needed as determined by the Test team. -----Original Message----- From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Monday, April 1, 2019 5:19 PM To: Jones, Bruce E <bruce.e.jones@intel.com>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] DRAFT release policy Hi Bruce, I have a comment regarding this point: The severity and number of bugs open against the release Proposal: No open Critical or High severity bugs against the release candidate. Or maybe 1-3 Highs if we have a clear resolution plan (and a plan to release a patch against the release?) (ghada) We use an explicit tag to identify which bugs gate a particular release (regardless of severity). The whole list will have to be reviewed and scrubbed prior to reaching the release milestone. I don't feel it is sufficient to only review Critical / Major issues. [Example: On April 1/2019, there are 85 release gating bugs: only 13 are Critical/High. Yet it wouldn't be sufficient to only fix those 13 to ensure a quality release]. In the Release Planning wiki[1] , we have previously stated this policy: All release gating issues are addressed or reviewed/accepted for deferral I feel we need to keep this. [1] https://wiki.openstack.org/wiki/StarlingX/Release_Plan -----Original Message----- From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Friday, March 29, 2019 2:10 PM To: Jones, Bruce E; Dean Troyer; Seiler, Glenn Cc: starlingx-discuss Subject: Re: [Starlingx-discuss] DRAFT release policy I've updated the etherpad with changes that reflect the current feedback. Please review and add any additional feedback there. Thank you! https://etherpad.openstack.org/p/stx-release-policy-draft brucej -----Original Message----- From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Friday, March 29, 2019 10:06 AM To: Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy Dean, Glenn - thank you for the feedback. I agree with it. There is also some feedback in the etherpad. I'm going to respond to both sets in the etherpad and try to improve the policy and the wording. https://etherpad.openstack.org/p/stx-release-policy-draft Thanks! brucej -----Original Message----- From: Dean Troyer [mailto:dtroyer@gmail.com] Sent: Thursday, March 28, 2019 7:29 PM To: Seiler, Glenn <glenn.seiler@windriver.com> Cc: Jones, Bruce E <bruce.e.jones@intel.com>; starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy On Thu, Mar 28, 2019 at 6:39 PM Seiler, Glenn <glenn.seiler@windriver.com> wrote:
1- We need to move away from time-based releases 2- We need to do twice a year releases. This stmt, by definition, implies a time-gated release. Maybe it isn’t a specific date, but it is still time-gated.
The wording does need work, yes. After a short conversation with Bruce this afternoon (Bruce, correct me if I'm wrong here) I came away with the intention being more of increasing the lag from OpenStack releases rather than separating completely from the OpenStack release cycle which is likely to stay at approx 6 months for a while (that's a rabbit hole under the bike shed I'd like to avoid just now).
As a nascent project, I think we need to show gradual and consistent progress.
++
I did listen to much of the release team meeting today, and realize the trade-offs between big-rocks and timing are very difficult.
Given the difficult choice of functionality versus timing, I personally think we need to show progress in getting to Stein and a container based distribution as major milestones in 1H and perhaps defer the Distributed Cloud capability to a 2H release.
I don’t see anything intrinsically wrong with moving a specific date out; it happens all the time. But I also think a release should have some gate; i.e. we don’t move out of 1H. And if some functionality isn’t ready, then we move the functionality to another release in 2H.
We took a stab at estimating and missed, making adjustments now is normal and to be expected. I agree with considering pushing distcloud to the next release because it is a) new functionality, and b) devs overlap with the container work and I think making the k8s infrastructure rock solid is much more important. If we are too far off with system stability the ramifications will be harder to overcome than delaying a new feature.
Anyway, that would be my vote, if I have one.
You totally have a voice as part of the community, I would like to hear from more folks here... dt -- Dean Troyer dtroyer@gmail.com _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Bruce, If none of the anchor features will be ready by MS-3, then we have no choice but to reforecast. Then we get into the question of by how much, and which, if any anchor features can be excluded (which doesn't totally make sense to me - why would we have called it an anchor in the first place). This Release Churn is another heading that we need to add to the release policy, and whatever we come up with, we'll need to get it approved by the TSC, I think. Bill... -----Original Message----- From: Jones, Bruce E <bruce.e.jones@intel.com> Sent: Tuesday, April 2, 2019 1:02 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Subject: Re: [Starlingx-discuss] DRAFT release policy Good feedback, Ghada, thank you. I have updated the document accordingly. I've also made a few other changes like adding test case readiness as a release criteria and proposed a method for handing anchor features that aren't completed by MS-3. I've removed the previously resolved (and much appreciated) feedback in the interests of readability. Link and updated text below. Brucej This file is: https://etherpad.openstack.org/p/stx-release-policy-draft This is a draft document for release planning. Comments and feedback welcomed! Openstack Release Policy ==================== The StarlingX project follows the release model defined in https://docs.openstack.org/project-team-guide/release-management.html using the "Trailing the common cycle" due to our dependency on upstream OpenStack projects. Release Planning ============== Initial release planning starts at the Open Infrastructure PTG meetings, where the TSC and community members discuss candidate features for the next release. The TSC then will review and approve a feature list for the release, will identify any release gating "anchor" features, and set a target date. The recommended target date is the date of the next OpenStack release plus 6 weeks. The PTG meeting is also an opportunity to review the community's goals and to define goals for the release. The overall Release Plan is created and managed by the Release sub-team by combining the TSC's input on content and target dates with input from the feature developers in the community and the Test team. The plan will include a standard set of milestones as per the usual OpenStack release management process. The Release team will actively manage the plan over the course of the release, recommending any adjustments in content and dates to the community and to the TSC for approval. We recognize that we are a new community working in a highly dynamic technology and that changes in our plans over time are normal and expected. We will work as a community to be open and transparent about our release process, and to minimize change from the original plan. Open issue: We should consider changing our release naming convention to something that isn't a date. Defect Tracking ============ The release team shall review active and incoming bug reports and make an initial call as to whether or not the bug needs to be fixed in the next release. If so, the bugs shall be tagged and tracked as the work on the release progresses. The list of release gating bugs will be actively managed, reviewed and scrubbed by the Release team to ensure that bugs are properly categorized as release gating. Release Policy =========== The Release team, together with the Test team TLs/PLs, shall make a recommendation to the community and TSC that a release is ready to go. Upon TSC approval, the release branches are tagged and the release documented. That recommendation should be based on: * Whether or not all anchor features in the release are complete, as per the input of the team(s) implementing the features and the results of Test team testing of the features * Proposal: All features identified as anchor features for a release need to be completed by the feature freeze milestone (MS-3). In the event that an anchor feature is not complete before the release feature freeze milestone, the Release team will make a recommendation to the TSC to extend the milestone date or to defer the feature to the next release. * Whether or not all test cases planned for the release are complete and ready to run * Proposal: All planned test cases shall be ready before the start of formal release candidate testing (RC1 milestone) * The completion and results of formal Testing performed by the Test team, measured by the percentage of planned tests attempted and the test pass rate * Proposal: 100% test cases attempted and 95% test cases passing in all configurations * The status of release gating bugs * Proposal: All release gating bugs must be fixed prior to a StarlingX release, ideally before the RC1 milestone but certainly before the final release. Bugfix Releases ============= The Release team, in conjunction with the community, can create a plan for a bug fix release. This would be an update to a previous release to address important defects that are impacting our users. The content would be fixes backported from master to the release branch, and would be based on both community and developer input regarding which fixes should be included. Testing of a bug fix release should include at least verification testing of the fixes and any additional testing needed as determined by the Test team. -----Original Message----- From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Monday, April 1, 2019 5:19 PM To: Jones, Bruce E <bruce.e.jones@intel.com>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] DRAFT release policy Hi Bruce, I have a comment regarding this point: The severity and number of bugs open against the release Proposal: No open Critical or High severity bugs against the release candidate. Or maybe 1-3 Highs if we have a clear resolution plan (and a plan to release a patch against the release?) (ghada) We use an explicit tag to identify which bugs gate a particular release (regardless of severity). The whole list will have to be reviewed and scrubbed prior to reaching the release milestone. I don't feel it is sufficient to only review Critical / Major issues. [Example: On April 1/2019, there are 85 release gating bugs: only 13 are Critical/High. Yet it wouldn't be sufficient to only fix those 13 to ensure a quality release]. In the Release Planning wiki[1] , we have previously stated this policy: All release gating issues are addressed or reviewed/accepted for deferral I feel we need to keep this. [1] https://wiki.openstack.org/wiki/StarlingX/Release_Plan -----Original Message----- From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Friday, March 29, 2019 2:10 PM To: Jones, Bruce E; Dean Troyer; Seiler, Glenn Cc: starlingx-discuss Subject: Re: [Starlingx-discuss] DRAFT release policy I've updated the etherpad with changes that reflect the current feedback. Please review and add any additional feedback there. Thank you! https://etherpad.openstack.org/p/stx-release-policy-draft brucej -----Original Message----- From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Friday, March 29, 2019 10:06 AM To: Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy Dean, Glenn - thank you for the feedback. I agree with it. There is also some feedback in the etherpad. I'm going to respond to both sets in the etherpad and try to improve the policy and the wording. https://etherpad.openstack.org/p/stx-release-policy-draft Thanks! brucej -----Original Message----- From: Dean Troyer [mailto:dtroyer@gmail.com] Sent: Thursday, March 28, 2019 7:29 PM To: Seiler, Glenn <glenn.seiler@windriver.com> Cc: Jones, Bruce E <bruce.e.jones@intel.com>; starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy On Thu, Mar 28, 2019 at 6:39 PM Seiler, Glenn <glenn.seiler@windriver.com> wrote:
1- We need to move away from time-based releases 2- We need to do twice a year releases. This stmt, by definition, implies a time-gated release. Maybe it isn’t a specific date, but it is still time-gated.
The wording does need work, yes. After a short conversation with Bruce this afternoon (Bruce, correct me if I'm wrong here) I came away with the intention being more of increasing the lag from OpenStack releases rather than separating completely from the OpenStack release cycle which is likely to stay at approx 6 months for a while (that's a rabbit hole under the bike shed I'd like to avoid just now).
As a nascent project, I think we need to show gradual and consistent progress.
++
I did listen to much of the release team meeting today, and realize the trade-offs between big-rocks and timing are very difficult.
Given the difficult choice of functionality versus timing, I personally think we need to show progress in getting to Stein and a container based distribution as major milestones in 1H and perhaps defer the Distributed Cloud capability to a 2H release.
I don’t see anything intrinsically wrong with moving a specific date out; it happens all the time. But I also think a release should have some gate; i.e. we don’t move out of 1H. And if some functionality isn’t ready, then we move the functionality to another release in 2H.
We took a stab at estimating and missed, making adjustments now is normal and to be expected. I agree with considering pushing distcloud to the next release because it is a) new functionality, and b) devs overlap with the container work and I think making the k8s infrastructure rock solid is much more important. If we are too far off with system stability the ramifications will be harder to overcome than delaying a new feature.
Anyway, that would be my vote, if I have one.
You totally have a voice as part of the community, I would like to hear from more folks here... dt -- Dean Troyer dtroyer@gmail.com _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Bill, yes if all anchor features are unready, there will be a delay for MS-3. Regarding the how much, and which features, and the churn you describe, I think (hope!) that the topic is covered already in the draft. The proposal is that it is managed by the Release team who makes recommendations to the TSC. brucej -----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Tuesday, April 2, 2019 11:24 AM To: Jones, Bruce E <bruce.e.jones@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Subject: RE: [Starlingx-discuss] DRAFT release policy Hi Bruce, If none of the anchor features will be ready by MS-3, then we have no choice but to reforecast. Then we get into the question of by how much, and which, if any anchor features can be excluded (which doesn't totally make sense to me - why would we have called it an anchor in the first place). This Release Churn is another heading that we need to add to the release policy, and whatever we come up with, we'll need to get it approved by the TSC, I think. Bill... -----Original Message----- From: Jones, Bruce E <bruce.e.jones@intel.com> Sent: Tuesday, April 2, 2019 1:02 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Subject: Re: [Starlingx-discuss] DRAFT release policy Good feedback, Ghada, thank you. I have updated the document accordingly. I've also made a few other changes like adding test case readiness as a release criteria and proposed a method for handing anchor features that aren't completed by MS-3. I've removed the previously resolved (and much appreciated) feedback in the interests of readability. Link and updated text below. Brucej This file is: https://etherpad.openstack.org/p/stx-release-policy-draft This is a draft document for release planning. Comments and feedback welcomed! Openstack Release Policy ==================== The StarlingX project follows the release model defined in https://docs.openstack.org/project-team-guide/release-management.html using the "Trailing the common cycle" due to our dependency on upstream OpenStack projects. Release Planning ============== Initial release planning starts at the Open Infrastructure PTG meetings, where the TSC and community members discuss candidate features for the next release. The TSC then will review and approve a feature list for the release, will identify any release gating "anchor" features, and set a target date. The recommended target date is the date of the next OpenStack release plus 6 weeks. The PTG meeting is also an opportunity to review the community's goals and to define goals for the release. The overall Release Plan is created and managed by the Release sub-team by combining the TSC's input on content and target dates with input from the feature developers in the community and the Test team. The plan will include a standard set of milestones as per the usual OpenStack release management process. The Release team will actively manage the plan over the course of the release, recommending any adjustments in content and dates to the community and to the TSC for approval. We recognize that we are a new community working in a highly dynamic technology and that changes in our plans over time are normal and expected. We will work as a community to be open and transparent about our release process, and to minimize change from the original plan. Open issue: We should consider changing our release naming convention to something that isn't a date. Defect Tracking ============ The release team shall review active and incoming bug reports and make an initial call as to whether or not the bug needs to be fixed in the next release. If so, the bugs shall be tagged and tracked as the work on the release progresses. The list of release gating bugs will be actively managed, reviewed and scrubbed by the Release team to ensure that bugs are properly categorized as release gating. Release Policy =========== The Release team, together with the Test team TLs/PLs, shall make a recommendation to the community and TSC that a release is ready to go. Upon TSC approval, the release branches are tagged and the release documented. That recommendation should be based on: * Whether or not all anchor features in the release are complete, as per the input of the team(s) implementing the features and the results of Test team testing of the features * Proposal: All features identified as anchor features for a release need to be completed by the feature freeze milestone (MS-3). In the event that an anchor feature is not complete before the release feature freeze milestone, the Release team will make a recommendation to the TSC to extend the milestone date or to defer the feature to the next release. * Whether or not all test cases planned for the release are complete and ready to run * Proposal: All planned test cases shall be ready before the start of formal release candidate testing (RC1 milestone) * The completion and results of formal Testing performed by the Test team, measured by the percentage of planned tests attempted and the test pass rate * Proposal: 100% test cases attempted and 95% test cases passing in all configurations * The status of release gating bugs * Proposal: All release gating bugs must be fixed prior to a StarlingX release, ideally before the RC1 milestone but certainly before the final release. Bugfix Releases ============= The Release team, in conjunction with the community, can create a plan for a bug fix release. This would be an update to a previous release to address important defects that are impacting our users. The content would be fixes backported from master to the release branch, and would be based on both community and developer input regarding which fixes should be included. Testing of a bug fix release should include at least verification testing of the fixes and any additional testing needed as determined by the Test team. -----Original Message----- From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Monday, April 1, 2019 5:19 PM To: Jones, Bruce E <bruce.e.jones@intel.com>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] DRAFT release policy Hi Bruce, I have a comment regarding this point: The severity and number of bugs open against the release Proposal: No open Critical or High severity bugs against the release candidate. Or maybe 1-3 Highs if we have a clear resolution plan (and a plan to release a patch against the release?) (ghada) We use an explicit tag to identify which bugs gate a particular release (regardless of severity). The whole list will have to be reviewed and scrubbed prior to reaching the release milestone. I don't feel it is sufficient to only review Critical / Major issues. [Example: On April 1/2019, there are 85 release gating bugs: only 13 are Critical/High. Yet it wouldn't be sufficient to only fix those 13 to ensure a quality release]. In the Release Planning wiki[1] , we have previously stated this policy: All release gating issues are addressed or reviewed/accepted for deferral I feel we need to keep this. [1] https://wiki.openstack.org/wiki/StarlingX/Release_Plan -----Original Message----- From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Friday, March 29, 2019 2:10 PM To: Jones, Bruce E; Dean Troyer; Seiler, Glenn Cc: starlingx-discuss Subject: Re: [Starlingx-discuss] DRAFT release policy I've updated the etherpad with changes that reflect the current feedback. Please review and add any additional feedback there. Thank you! https://etherpad.openstack.org/p/stx-release-policy-draft brucej -----Original Message----- From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Friday, March 29, 2019 10:06 AM To: Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy Dean, Glenn - thank you for the feedback. I agree with it. There is also some feedback in the etherpad. I'm going to respond to both sets in the etherpad and try to improve the policy and the wording. https://etherpad.openstack.org/p/stx-release-policy-draft Thanks! brucej -----Original Message----- From: Dean Troyer [mailto:dtroyer@gmail.com] Sent: Thursday, March 28, 2019 7:29 PM To: Seiler, Glenn <glenn.seiler@windriver.com> Cc: Jones, Bruce E <bruce.e.jones@intel.com>; starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy On Thu, Mar 28, 2019 at 6:39 PM Seiler, Glenn <glenn.seiler@windriver.com> wrote:
1- We need to move away from time-based releases 2- We need to do twice a year releases. This stmt, by definition, implies a time-gated release. Maybe it isn’t a specific date, but it is still time-gated.
The wording does need work, yes. After a short conversation with Bruce this afternoon (Bruce, correct me if I'm wrong here) I came away with the intention being more of increasing the lag from OpenStack releases rather than separating completely from the OpenStack release cycle which is likely to stay at approx 6 months for a while (that's a rabbit hole under the bike shed I'd like to avoid just now).
As a nascent project, I think we need to show gradual and consistent progress.
++
I did listen to much of the release team meeting today, and realize the trade-offs between big-rocks and timing are very difficult.
Given the difficult choice of functionality versus timing, I personally think we need to show progress in getting to Stein and a container based distribution as major milestones in 1H and perhaps defer the Distributed Cloud capability to a 2H release.
I don’t see anything intrinsically wrong with moving a specific date out; it happens all the time. But I also think a release should have some gate; i.e. we don’t move out of 1H. And if some functionality isn’t ready, then we move the functionality to another release in 2H.
We took a stab at estimating and missed, making adjustments now is normal and to be expected. I agree with considering pushing distcloud to the next release because it is a) new functionality, and b) devs overlap with the container work and I think making the k8s infrastructure rock solid is much more important. If we are too far off with system stability the ramifications will be harder to overcome than delaying a new feature.
Anyway, that would be my vote, if I have one.
You totally have a voice as part of the community, I would like to hear from more folks here... dt -- Dean Troyer dtroyer@gmail.com _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Bruce - I guess I'm saying that we need to consider a wider churn policy than just what to do if stuff isn't ready at MS-3. -----Original Message----- From: Jones, Bruce E <bruce.e.jones@intel.com> Sent: Tuesday, April 2, 2019 3:38 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Subject: RE: [Starlingx-discuss] DRAFT release policy Bill, yes if all anchor features are unready, there will be a delay for MS-3. Regarding the how much, and which features, and the churn you describe, I think (hope!) that the topic is covered already in the draft. The proposal is that it is managed by the Release team who makes recommendations to the TSC. brucej -----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Tuesday, April 2, 2019 11:24 AM To: Jones, Bruce E <bruce.e.jones@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Subject: RE: [Starlingx-discuss] DRAFT release policy Hi Bruce, If none of the anchor features will be ready by MS-3, then we have no choice but to reforecast. Then we get into the question of by how much, and which, if any anchor features can be excluded (which doesn't totally make sense to me - why would we have called it an anchor in the first place). This Release Churn is another heading that we need to add to the release policy, and whatever we come up with, we'll need to get it approved by the TSC, I think. Bill... -----Original Message----- From: Jones, Bruce E <bruce.e.jones@intel.com> Sent: Tuesday, April 2, 2019 1:02 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Subject: Re: [Starlingx-discuss] DRAFT release policy Good feedback, Ghada, thank you. I have updated the document accordingly. I've also made a few other changes like adding test case readiness as a release criteria and proposed a method for handing anchor features that aren't completed by MS-3. I've removed the previously resolved (and much appreciated) feedback in the interests of readability. Link and updated text below. Brucej This file is: https://etherpad.openstack.org/p/stx-release-policy-draft This is a draft document for release planning. Comments and feedback welcomed! Openstack Release Policy ==================== The StarlingX project follows the release model defined in https://docs.openstack.org/project-team-guide/release-management.html using the "Trailing the common cycle" due to our dependency on upstream OpenStack projects. Release Planning ============== Initial release planning starts at the Open Infrastructure PTG meetings, where the TSC and community members discuss candidate features for the next release. The TSC then will review and approve a feature list for the release, will identify any release gating "anchor" features, and set a target date. The recommended target date is the date of the next OpenStack release plus 6 weeks. The PTG meeting is also an opportunity to review the community's goals and to define goals for the release. The overall Release Plan is created and managed by the Release sub-team by combining the TSC's input on content and target dates with input from the feature developers in the community and the Test team. The plan will include a standard set of milestones as per the usual OpenStack release management process. The Release team will actively manage the plan over the course of the release, recommending any adjustments in content and dates to the community and to the TSC for approval. We recognize that we are a new community working in a highly dynamic technology and that changes in our plans over time are normal and expected. We will work as a community to be open and transparent about our release process, and to minimize change from the original plan. Open issue: We should consider changing our release naming convention to something that isn't a date. Defect Tracking ============ The release team shall review active and incoming bug reports and make an initial call as to whether or not the bug needs to be fixed in the next release. If so, the bugs shall be tagged and tracked as the work on the release progresses. The list of release gating bugs will be actively managed, reviewed and scrubbed by the Release team to ensure that bugs are properly categorized as release gating. Release Policy =========== The Release team, together with the Test team TLs/PLs, shall make a recommendation to the community and TSC that a release is ready to go. Upon TSC approval, the release branches are tagged and the release documented. That recommendation should be based on: * Whether or not all anchor features in the release are complete, as per the input of the team(s) implementing the features and the results of Test team testing of the features * Proposal: All features identified as anchor features for a release need to be completed by the feature freeze milestone (MS-3). In the event that an anchor feature is not complete before the release feature freeze milestone, the Release team will make a recommendation to the TSC to extend the milestone date or to defer the feature to the next release. * Whether or not all test cases planned for the release are complete and ready to run * Proposal: All planned test cases shall be ready before the start of formal release candidate testing (RC1 milestone) * The completion and results of formal Testing performed by the Test team, measured by the percentage of planned tests attempted and the test pass rate * Proposal: 100% test cases attempted and 95% test cases passing in all configurations * The status of release gating bugs * Proposal: All release gating bugs must be fixed prior to a StarlingX release, ideally before the RC1 milestone but certainly before the final release. Bugfix Releases ============= The Release team, in conjunction with the community, can create a plan for a bug fix release. This would be an update to a previous release to address important defects that are impacting our users. The content would be fixes backported from master to the release branch, and would be based on both community and developer input regarding which fixes should be included. Testing of a bug fix release should include at least verification testing of the fixes and any additional testing needed as determined by the Test team. -----Original Message----- From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Monday, April 1, 2019 5:19 PM To: Jones, Bruce E <bruce.e.jones@intel.com>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] DRAFT release policy Hi Bruce, I have a comment regarding this point: The severity and number of bugs open against the release Proposal: No open Critical or High severity bugs against the release candidate. Or maybe 1-3 Highs if we have a clear resolution plan (and a plan to release a patch against the release?) (ghada) We use an explicit tag to identify which bugs gate a particular release (regardless of severity). The whole list will have to be reviewed and scrubbed prior to reaching the release milestone. I don't feel it is sufficient to only review Critical / Major issues. [Example: On April 1/2019, there are 85 release gating bugs: only 13 are Critical/High. Yet it wouldn't be sufficient to only fix those 13 to ensure a quality release]. In the Release Planning wiki[1] , we have previously stated this policy: All release gating issues are addressed or reviewed/accepted for deferral I feel we need to keep this. [1] https://wiki.openstack.org/wiki/StarlingX/Release_Plan -----Original Message----- From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Friday, March 29, 2019 2:10 PM To: Jones, Bruce E; Dean Troyer; Seiler, Glenn Cc: starlingx-discuss Subject: Re: [Starlingx-discuss] DRAFT release policy I've updated the etherpad with changes that reflect the current feedback. Please review and add any additional feedback there. Thank you! https://etherpad.openstack.org/p/stx-release-policy-draft brucej -----Original Message----- From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Friday, March 29, 2019 10:06 AM To: Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy Dean, Glenn - thank you for the feedback. I agree with it. There is also some feedback in the etherpad. I'm going to respond to both sets in the etherpad and try to improve the policy and the wording. https://etherpad.openstack.org/p/stx-release-policy-draft Thanks! brucej -----Original Message----- From: Dean Troyer [mailto:dtroyer@gmail.com] Sent: Thursday, March 28, 2019 7:29 PM To: Seiler, Glenn <glenn.seiler@windriver.com> Cc: Jones, Bruce E <bruce.e.jones@intel.com>; starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy On Thu, Mar 28, 2019 at 6:39 PM Seiler, Glenn <glenn.seiler@windriver.com> wrote:
1- We need to move away from time-based releases 2- We need to do twice a year releases. This stmt, by definition, implies a time-gated release. Maybe it isn’t a specific date, but it is still time-gated.
The wording does need work, yes. After a short conversation with Bruce this afternoon (Bruce, correct me if I'm wrong here) I came away with the intention being more of increasing the lag from OpenStack releases rather than separating completely from the OpenStack release cycle which is likely to stay at approx 6 months for a while (that's a rabbit hole under the bike shed I'd like to avoid just now).
As a nascent project, I think we need to show gradual and consistent progress.
++
I did listen to much of the release team meeting today, and realize the trade-offs between big-rocks and timing are very difficult.
Given the difficult choice of functionality versus timing, I personally think we need to show progress in getting to Stein and a container based distribution as major milestones in 1H and perhaps defer the Distributed Cloud capability to a 2H release.
I don’t see anything intrinsically wrong with moving a specific date out; it happens all the time. But I also think a release should have some gate; i.e. we don’t move out of 1H. And if some functionality isn’t ready, then we move the functionality to another release in 2H.
We took a stab at estimating and missed, making adjustments now is normal and to be expected. I agree with considering pushing distcloud to the next release because it is a) new functionality, and b) devs overlap with the container work and I think making the k8s infrastructure rock solid is much more important. If we are too far off with system stability the ramifications will be harder to overcome than delaying a new feature.
Anyway, that would be my vote, if I have one.
You totally have a voice as part of the community, I would like to hear from more folks here... dt -- Dean Troyer dtroyer@gmail.com _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
OK, I understand now. Let's discuss at our meeting later today which other milestones we might want to include. brucej -----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Thursday, April 4, 2019 6:57 AM To: Jones, Bruce E <bruce.e.jones@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Subject: RE: [Starlingx-discuss] DRAFT release policy Hi Bruce - I guess I'm saying that we need to consider a wider churn policy than just what to do if stuff isn't ready at MS-3. -----Original Message----- From: Jones, Bruce E <bruce.e.jones@intel.com> Sent: Tuesday, April 2, 2019 3:38 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Subject: RE: [Starlingx-discuss] DRAFT release policy Bill, yes if all anchor features are unready, there will be a delay for MS-3. Regarding the how much, and which features, and the churn you describe, I think (hope!) that the topic is covered already in the draft. The proposal is that it is managed by the Release team who makes recommendations to the TSC. brucej -----Original Message----- From: Zvonar, Bill [mailto:Bill.Zvonar@windriver.com] Sent: Tuesday, April 2, 2019 11:24 AM To: Jones, Bruce E <bruce.e.jones@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Subject: RE: [Starlingx-discuss] DRAFT release policy Hi Bruce, If none of the anchor features will be ready by MS-3, then we have no choice but to reforecast. Then we get into the question of by how much, and which, if any anchor features can be excluded (which doesn't totally make sense to me - why would we have called it an anchor in the first place). This Release Churn is another heading that we need to add to the release policy, and whatever we come up with, we'll need to get it approved by the TSC, I think. Bill... -----Original Message----- From: Jones, Bruce E <bruce.e.jones@intel.com> Sent: Tuesday, April 2, 2019 1:02 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Subject: Re: [Starlingx-discuss] DRAFT release policy Good feedback, Ghada, thank you. I have updated the document accordingly. I've also made a few other changes like adding test case readiness as a release criteria and proposed a method for handing anchor features that aren't completed by MS-3. I've removed the previously resolved (and much appreciated) feedback in the interests of readability. Link and updated text below. Brucej This file is: https://etherpad.openstack.org/p/stx-release-policy-draft This is a draft document for release planning. Comments and feedback welcomed! Openstack Release Policy ==================== The StarlingX project follows the release model defined in https://docs.openstack.org/project-team-guide/release-management.html using the "Trailing the common cycle" due to our dependency on upstream OpenStack projects. Release Planning ============== Initial release planning starts at the Open Infrastructure PTG meetings, where the TSC and community members discuss candidate features for the next release. The TSC then will review and approve a feature list for the release, will identify any release gating "anchor" features, and set a target date. The recommended target date is the date of the next OpenStack release plus 6 weeks. The PTG meeting is also an opportunity to review the community's goals and to define goals for the release. The overall Release Plan is created and managed by the Release sub-team by combining the TSC's input on content and target dates with input from the feature developers in the community and the Test team. The plan will include a standard set of milestones as per the usual OpenStack release management process. The Release team will actively manage the plan over the course of the release, recommending any adjustments in content and dates to the community and to the TSC for approval. We recognize that we are a new community working in a highly dynamic technology and that changes in our plans over time are normal and expected. We will work as a community to be open and transparent about our release process, and to minimize change from the original plan. Open issue: We should consider changing our release naming convention to something that isn't a date. Defect Tracking ============ The release team shall review active and incoming bug reports and make an initial call as to whether or not the bug needs to be fixed in the next release. If so, the bugs shall be tagged and tracked as the work on the release progresses. The list of release gating bugs will be actively managed, reviewed and scrubbed by the Release team to ensure that bugs are properly categorized as release gating. Release Policy =========== The Release team, together with the Test team TLs/PLs, shall make a recommendation to the community and TSC that a release is ready to go. Upon TSC approval, the release branches are tagged and the release documented. That recommendation should be based on: * Whether or not all anchor features in the release are complete, as per the input of the team(s) implementing the features and the results of Test team testing of the features * Proposal: All features identified as anchor features for a release need to be completed by the feature freeze milestone (MS-3). In the event that an anchor feature is not complete before the release feature freeze milestone, the Release team will make a recommendation to the TSC to extend the milestone date or to defer the feature to the next release. * Whether or not all test cases planned for the release are complete and ready to run * Proposal: All planned test cases shall be ready before the start of formal release candidate testing (RC1 milestone) * The completion and results of formal Testing performed by the Test team, measured by the percentage of planned tests attempted and the test pass rate * Proposal: 100% test cases attempted and 95% test cases passing in all configurations * The status of release gating bugs * Proposal: All release gating bugs must be fixed prior to a StarlingX release, ideally before the RC1 milestone but certainly before the final release. Bugfix Releases ============= The Release team, in conjunction with the community, can create a plan for a bug fix release. This would be an update to a previous release to address important defects that are impacting our users. The content would be fixes backported from master to the release branch, and would be based on both community and developer input regarding which fixes should be included. Testing of a bug fix release should include at least verification testing of the fixes and any additional testing needed as determined by the Test team. -----Original Message----- From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Monday, April 1, 2019 5:19 PM To: Jones, Bruce E <bruce.e.jones@intel.com>; Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] DRAFT release policy Hi Bruce, I have a comment regarding this point: The severity and number of bugs open against the release Proposal: No open Critical or High severity bugs against the release candidate. Or maybe 1-3 Highs if we have a clear resolution plan (and a plan to release a patch against the release?) (ghada) We use an explicit tag to identify which bugs gate a particular release (regardless of severity). The whole list will have to be reviewed and scrubbed prior to reaching the release milestone. I don't feel it is sufficient to only review Critical / Major issues. [Example: On April 1/2019, there are 85 release gating bugs: only 13 are Critical/High. Yet it wouldn't be sufficient to only fix those 13 to ensure a quality release]. In the Release Planning wiki[1] , we have previously stated this policy: All release gating issues are addressed or reviewed/accepted for deferral I feel we need to keep this. [1] https://wiki.openstack.org/wiki/StarlingX/Release_Plan -----Original Message----- From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Friday, March 29, 2019 2:10 PM To: Jones, Bruce E; Dean Troyer; Seiler, Glenn Cc: starlingx-discuss Subject: Re: [Starlingx-discuss] DRAFT release policy I've updated the etherpad with changes that reflect the current feedback. Please review and add any additional feedback there. Thank you! https://etherpad.openstack.org/p/stx-release-policy-draft brucej -----Original Message----- From: Jones, Bruce E [mailto:bruce.e.jones@intel.com] Sent: Friday, March 29, 2019 10:06 AM To: Dean Troyer <dtroyer@gmail.com>; Seiler, Glenn <glenn.seiler@windriver.com> Cc: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy Dean, Glenn - thank you for the feedback. I agree with it. There is also some feedback in the etherpad. I'm going to respond to both sets in the etherpad and try to improve the policy and the wording. https://etherpad.openstack.org/p/stx-release-policy-draft Thanks! brucej -----Original Message----- From: Dean Troyer [mailto:dtroyer@gmail.com] Sent: Thursday, March 28, 2019 7:29 PM To: Seiler, Glenn <glenn.seiler@windriver.com> Cc: Jones, Bruce E <bruce.e.jones@intel.com>; starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] DRAFT release policy On Thu, Mar 28, 2019 at 6:39 PM Seiler, Glenn <glenn.seiler@windriver.com> wrote:
1- We need to move away from time-based releases 2- We need to do twice a year releases. This stmt, by definition, implies a time-gated release. Maybe it isn’t a specific date, but it is still time-gated.
The wording does need work, yes. After a short conversation with Bruce this afternoon (Bruce, correct me if I'm wrong here) I came away with the intention being more of increasing the lag from OpenStack releases rather than separating completely from the OpenStack release cycle which is likely to stay at approx 6 months for a while (that's a rabbit hole under the bike shed I'd like to avoid just now).
As a nascent project, I think we need to show gradual and consistent progress.
++
I did listen to much of the release team meeting today, and realize the trade-offs between big-rocks and timing are very difficult.
Given the difficult choice of functionality versus timing, I personally think we need to show progress in getting to Stein and a container based distribution as major milestones in 1H and perhaps defer the Distributed Cloud capability to a 2H release.
I don’t see anything intrinsically wrong with moving a specific date out; it happens all the time. But I also think a release should have some gate; i.e. we don’t move out of 1H. And if some functionality isn’t ready, then we move the functionality to another release in 2H.
We took a stab at estimating and missed, making adjustments now is normal and to be expected. I agree with considering pushing distcloud to the next release because it is a) new functionality, and b) devs overlap with the container work and I think making the k8s infrastructure rock solid is much more important. If we are too far off with system stability the ramifications will be harder to overcome than delaying a new feature.
Anyway, that would be my vote, if I have one.
You totally have a voice as part of the community, I would like to hear from more folks here... dt -- Dean Troyer dtroyer@gmail.com _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (5)
-
Dean Troyer
-
Jones, Bruce E
-
Khalil, Ghada
-
Seiler, Glenn
-
Zvonar, Bill