[Starlingx-discuss] Minutes: StarlingX Release Meeting - August 1/2019

Zvonar, Bill Bill.Zvonar at windriver.com
Thu Aug 1 20:51:43 UTC 2019


Notes & actions from today's release team meeting are below.

In summary...

- the test team has some actions around feature/regression test status - once we see those updates, we can decide on the RC1 milestone
- we will sort out the logistics of branch creation in the next day or so, starting with the email that Scott just sent 

Bill... 

https://etherpad.openstack.org/p/stx-releases

Release meeting notes August 1 2019

stx.2.0

Test Status

- Feature Testing and Regression Testing need to be complete by the RC1 milestone. 
  Both are still open. Need to discuss new forecast dates for both.

  - Two options to discuss:
    - Declare the milestone next week with an exception    << Preferred
    - Move the milestone

  - Feature Testing
    - Tracker:https://docs.google.com/spreadsheets/d/15us6HWgcb0dmHHZe2SyOR_UI5BvVpoU8TCST0rQd4zg/edit#gid=1717644237
      - Helm Overrides
         - still executing some of these, about 7 left to run - will try to get them done in the next couple of days 
         - ACTION: Elio send Helm Override testcase titles to Frank in case we can help prioritize 
      - Ironic
         - they were lacking the right HW to run these, they have it now, need to set it up first, so they'll need to provide a plan 
         - ACTION: Elio/Ada provide a plan, including the number of testcases 

  - Regression Testing
      - pass rate 94.82% 
      - Tracker: https://docs.google.com/spreadsheets/d/1MoBroFimeQjsJvCC_N_DSLYMQsNZVMzTwAVc5tPy_t0/edit#gid=838066175
      - 29 left to run - planning to finish tomorrow 
        - ACTION: Elio/Ada provide update on these 
      - 27 blocked
        - 4 - Security on IPv6 - blocked since IPv6 not set up yet
            - should be ok as an exception since IPv6 
        - 7 - SRIOV - ran out of time
            - ACTION: Elio/Ada provide plan for running through these 
        - 12 - System - still having issues pinging between instances on external configuration (2+2+2) 
             - other 2+2+2 testcases have been run, only the ones that require pinging
             - LAUNCHPADS
               - https://bugs.launchpad.net/starlingx/+bug/1835575
               - https://bugs.launchpad.net/starlingx/+bug/1830286
             - Elio to reach out to networking peers (i.e. Forrest) about switch issues 
             - likely would not hold up RC1 declaration for this set of blocked testcases 


RC1 Milestone -- Branch Creation Mechanics / Logistics

  - Request participation from Scott Little, Dean Troyer & Don Penney
  - Work Items
    - Stop code merges on master for a short period. Send an email to the community. Control through the core reviewers
    - Create release branch:  r/stx.2.0
      - Branch the main starlingx repos ...including docs (doc updates need to be cherrypicked as well
        - (dtroyer) did this change recently?  docs was not versioned for stx.1
      - What about the stx staging branches?  stx-nova / libvirt / qemu
        - At a min, libvirt / qemu need to 
    - Create new CENGN builds from the r/stx.2.0 branch.
      - Build daily between RC1 and Final Compile.
      - How often to build docker images? right now, it's weekly
    - At Final Compile / Release, label the release branch r/stx.2.0 as stx.2.0.0
      - How do we make sure it's clear which ISO on CENGN corresponds to this?   (also need to make sure it and the corresponding docker images are not deleted)
    - After Final Compile / Release (Aug 26-30)
      - Developers will continue to work on stx.2.0 High priority bugs - sourcing fixes in master and cherrypicking to the release branch
      - Candidate "maintenance" release builds will be done on demand  OR perhaps once a week(?) so that sanity and mini-regression can continue (helps ensure stability)
        - How do we distinguish on CENGN between those and the "offical" maintenance release
      - When we are ready to issue a maintenance release, label the release branch as stx.2.0.x
        - Again, need to figure out how to distinguish the official maintenance release on CENGN

- Scott's Details

  - General Requirements

    - Create a release candidate branch from master, preferably originating from the context of a green sanity.
      -  more on branching below
    - RC branch recieves cherry-picked patches from master until a final compile is declared.
    - Set SW_VERSION, aka PLATFORM_RELEASE, for RC branch to 19.08  (Format YY.MM)
      - Retain same SW_VERSION for dot releases
    - Set SW_VERSION, aka PLATFORM_RELEASE, for master branch to 19.09
    - Any helm chart changes to pick up release images, not the master images?
      - answer appears to be no.  Helm charts will list the images we build
    - Create a CENGN job to build the RC branch.  Daily builds until final compile is declared.
      - scripts are fairly generic.  In theory it's just need for a new master job to set customize parameters.
      - A little bit of work to set docker images tags correctly vs branching startagy... more below
        - Current format would default to  "r-${BRANCH}-centos-stable-${PUBLISH_TIMESTAMP}.0"
      - A release branch will test some new code paths in cengn scripts.  Will have to monitor closely.
    - Make sure the build/image retirement scripts are doing the right thing.
      - Already coded to support branching opt 2 (below)
      - Support for opt 1 will reqiore some new scripting in CENGN

    - Branching startagy and content lockdown
      - Desired properties of the branch strategy.
        - We can re-build the ISO release at a later date.
          - The exact context of StarlingX git trees is captured in some form.
          - Context of third party git repos is captured on a best effort basis.  e.g. capture tag or sha (an assume they are stable), but not cloning gits.
          - Leverage the 'revision' field for all repos in the manifest.
        - We can rebuild our docker images at a later date... I don't think we fully know how to do this yet.
          - Lock down the base centos image, and yum, if possible
            - Need tooling changes for this
            - Need to reference centos docker image by sha, not tag.
            - Probably need to hack the yum configuration as well, point it to cengn.
              - The build of stx-centos points to cengn repo for yum update. The loci build of images, however, also uses upstream sources. Otherwise, we would need to include all RPMs used for those images in the LST files
          - Lock down our inputs from PYPI as best we can
            - Find all files named *stable_docker_image, field 'PIP_PACKAGES=' needs to use syntax like ... e.g. panko==5.0.0
            - Inputs can be found in piplst files   e.g. $MY_WORKSPACE/std/build-images/tis-networking-avs-heat-centos-stable.piplst
            - All python modules (non-starlingx) installed would need to be in the base wheels.cfg, which also updates the upper-constraints.txt in the tarball to restrict the installed version.
          - Lock down rpms feeding into docker images if possible.
            - Might not be possible without lock down of the base image
            - Find all files named *stable_docker_image, field 'DIST_PACKAGES=' needs to use syntax like ... e.g. bash-4.2.46-31.el7.tis.4.x86_64
            - Inputs can be found in rpmlis files   e.g. $MY_WORKSPACE/std/build-images/tis-networking-avs-heat-centos-stable.rpmlst
            - ALL 'tis' packages found in rpmlst must be listed in DIST_PACKAGES
        - We don't have the power to branch and tag all repos.
          - Some of the work needs to be done by further locking down the manifest on specific tags/shas
          - Do we store the locked down manifest as tagged copy of default.xml, or use versioned file names xml?

    - Basic format of our branchs and tags
      - YYYY.MM   i.e. date like
      - 2.0       i.e. a release number 
      - Current CENGN scripting uses the date format, release format requires scripting changes.
      - I'm always eager that a release branch be clearly distinguished from a dev/feature branch.  Currently CENGN looks for YYYY.MM.  Is it safe enough to look for anything starting with a number? 

    - Opt 1 - Single RC branch. Tags mark dot releases - prefered
      - Branch name is r/stx.2.0 ... applies to starlingx
      - Tags for each dot release (v2.0.##). applies to starlingx and (v.stx.2.0.##) starlingx-staging
      - Git lockdown via creation of a manifest (v2019.08.##.xml  or  v2.0.##.xml) rather than default.xml.  In this manifest we specify tags or shas for each git.
      - New scripting required on CENGN for load and docker image retirement
      - we agreed on this option, with the v2.0  - SemVer format
      - we also agreed to tag the staging repos - may need to prefix those w/ stx if they're already using numeric tagging 
  
    - Opt 2 - Single RC branch. fork a branch to lock down a dot releases
      - Branch name is 2019.08-rc ... or 2.0-rc ... applies to starlingx and starlingx-staging
      - Branch for a dot release (2019.08.##  or  v2.0.##).
      - Git lockdown occures in default.xml.  In this manifest we specify tags or shas for each git.

    - Opt 3 - new RC branch for each dot release  (waterfall)
      - Branch name is 2019.08.##... or 2.0.## ... applies to starlingx and starlingx-staging
      - Final commit is to lock down the manifest 
      - Next dot release forks from the prior dot release (the commit prior to manifest lockdown).

    - Docker image labeling:
      - Release image options
      - r-2019.08.##-centos-stable.0
      - r-2.0.##-centos-stable.0
      - 2019.08.##-centos-stable.0
      - 2.0.##-centos-stable-${PUBLISH_TIMESTAMP}.0   --- prefered   <--- leaning towards this 

    - Cengn path:
      - Release path
      - starlingx/release/2018.10.##
      - starlingx/release/2018.10/2018.10.##
      - starlingx/release/2.0.##
      - starlingx/release/2.0/2.0.##

    - RC path
      - starlingx/rc/2018.10/timestamp
      - starlingx/rc/2018.10.##/timestamp
      - starlingx/rc/2018.10/2018.10.##/timestamp
      - starlingx/rc/2.0/timestamp
      - starlingx/rc/2.0.##/timestamp
      - starlingx/rc/2.0/2.0.##/timestamp

Didn't get to this... 

    - Maintenance Release Plan
      - From the Release Planning wiki: https://wiki.openstack.org/wiki/StarlingX/Release_Plan#Release_Maintenance
      - Formal releases are maintained for 12 months. This allows support for two previous StarlingX releases.
        - During the maintenance window, the release team will evaluate monthly whether a maintenance release is required.
        - The maintenance release will be tagged on the release branch.
        - The maintenance release will undergo a mini regression test cycle.
        - The corresponding binaries will be posted on the StarlingX CENGN mirror.
      - Plan for the first stx.2.0 maintenance release
        - recommend 4-6wks after the stx.2.0 release
        - Request Ada and Numan to put together the mini-regression plan 
      - Discuss labels



More information about the Starlingx-discuss mailing list