StarlingX Developers:

In the past 2 weeks we’ve had many days of build failures.  A full timeline is below but the majority of the issues were associated with delivery of a large number of commits for 2 large features.  And having this done just days after build layering was enabled for the project.  Looking through the variety of failures, the Build Team identified the following recommendations to prevent similar issues in the future.  These recommendations are for both the developers as well as the Cores.

Recommendations for Developers:

1. Every developer must do a final build before removing WFL -1 from their commit. 
2. For larger features or any commits that introduce or change an rpm, the developer build must be done in clean workspace and clean ‘output’ directories. This includes starting with a clean mirror download from CENGN.  Address any failures seen in your build when using a clean workspace.  Identify any mirror updates required and see [3] for how to handle.
3. Any changes to lst files need to be handled separate from rest of code/feature as follows:
• Deliver a commit that adds all entries to lst files required for the feature
• Wait for a successful CENGN build, confirm the new files appear in the CENGN mirror. 
• Rebuild the code/feature in a clean developer workspace and ensure a clean build with no additional lst changes needed.  Then merge the code/feature commits. 
• Once the next CENGN build is successful, then deliver an additional commit to remove any old lst file entries that are no longer needed.
4. Community needs to move over to use the layered build approach for their developer builds.  This is the production way of building and will save developer build times for most commits.

Recommendations for Cores:

1. Ensure large features have followed the above guidelines before allowing the code to merge.

If there are any other suggestions please provide your input.  Thank-you.

StarlingX Build Team

----------------------------------------

----- Timeline of failures --------

April 28

• Train commits merged, build failure
• Root cause: copy/paste error on the source repo config https://review.opendev.org/716883
• Don Penney found issue and provided fix: https://review.opendev.org/724127
• Preventative action: before merge do final test with build in developer workspace in clean workspace and clean ‘output’ directories. Address any failures seen in build from a clean workspace.

April 29

• Train commits causing additional build failure
• Root cause: missing dependency - need to add lst entry for python2-sphinxcontrib-apidoc. 
• Scott Little found issue and provided fix: https://review.opendev.org/#/c/724411/
• Preventative action: developers need to build before merge and start with clean build environment

April 30

• Train commits still causing build failure
• Root cause: missing dependency: need build dependency in python-neutronclient
• Scott Little found issue and provided fix: https://review.opendev.org/#/c/724792/

May 1-3

• 4.18 kernel commits causing build failures
• Root cause #1: CentOS 8 has released a newer kernel version than the one that was merged in our stx-kernel.  CentOS 8 has an “obsoletes” on the kernel we had merged.  The yumdownloader tool refused to download an obsoletes package.
• Scott Little found issue, Austin Sun & Shuicheng Lin provided fixes: [1][2][3]
• Preventative action: before merge do final test with build in clean workspace and clean ‘output’ directories. Address any failures seen in build from a clean workspace.
• Additional action: How are we going to prevent build failures due to obsoletes when next CentOS 8 version is released and our current version is obsoleted?

May 4

• 4.18 kernel commits causing additional build failures
• Root cause #2: missing dependencies for kmods & mellanox – lst files needed to be updated
• Scott Little provided fixes: [4][5]
• Shuicheng Lin provided an additional fix to add perl-generators to lst: [6]

May 5

• 4.18 kernel commits causing additional build failure during flock layer build
• Root cause #3: Issue with build layering mirror update script colliding with a generic script running on CENGN
• Scott Little provided fix in CENGN script

[1] https://review.opendev.org/#/c/725214/

[2] https://review.opendev.org/#/c/724951/

[3] https://review.opendev.org/#/c/724955/

[4] https://review.opendev.org/#/c/725379/

[5] https://review.opendev.org/#/c/725371/

[6] https://review.opendev.org/#/c/725447/