It was a really bad week for build issues on CENGN. Below I try to summarize the we've been working on. Scott PS: use a wide window to view! Branch OS Type Issue ====== == ==== ===== Master Centos Layered Build failures Oct 28-31 Issue: 'Repo' pulled down a non-backward compatible upgrade from upstream that breaks non-containerized uses of 'repo'. Fix: Updated the version of 'repo' on CENGN bare metal. Manual build successful: Oct 31 Master Centos Monolithic Build Failure Oct 29 Issue: Corruption of the jenkins build script. Cause of corruption unknown. Fix: Repaired build script and restarted jenkins Next build attempt Nov 5 Master Debian Monolithic Build Failures Oct 28-Nov 1 Issue 1: Attempt to optimize the 'publish packages' step had undetected bugs when merged Friday. Fix 1: corrected on path, and prevented script from throwing error when a grep fails to find a match. Benefit: 'publish packages' step takes minutes, not hours. Issue 2: STX_build_debian_master jenkins job was deleted. Unclear how/why but CENGN jenkins was very slow and acting strangely. Fix 2: restored STX_build_debian_master from git, but we've lost build history. Restarted Jenkins. Issue 3: Activation of 'reuse' accelerated package build, ~8 hrs -> ~2hrs. Works for ISO, but breaks container image build. Misunderstood the intent of one recent fix to this feature, thought it would address the container image build issues. Clearly it did not. Fix 3: Deactivated 'reuse' feature once again. Longer term, a LP will be raised vs docker images. Docker images need to pull debs from the aptly published repo, not the traditional repo. Issue 4: A jenkins job was create to test changes to jenkins pipeline scripts. That job had a name that was similar to but not identical to the main job. This exposed a defect in how minikube finds containers using fuzzy rather than exact matches. Cross-talk between main and dev minikube environments broke both builds. https://bugs.launchpad.net/starlingx/+bug/1995400 <https://bugs.launchpad.net/starlingx/+bug/1995400> Fix 4: Quick fix was to change the dev build name to avoid the 'fuzzy' match, and to nuke/rebuild both correpted minikube environments. Longer term, we need to use exect matches within minikube. Success: Build with docker images, TIMESTAMP 20221102T060000Z, Available 20221102T160000Z stx.6.0 Centos Layered Build Failed Oct 29-Nov 1 Issue: The builder container can't build. pip install pulled in unconstrained packages that have dropped py2.7 compatibility. https://bugs.launchpad.net/starlingx/+bug/1995484 Fix: Add constraints to maintain py2.7 compatibility in the pip installed packages. https://review.opendev.org/c/starlingx/tools/+/863265 Test build started stx.7.0 Centos Layered Build Failed Oct 30 Issue: Failed to build containers locationservice-base, notificationclient-base, notificationservice-base https://bugs.launchpad.net/starlingx/+bug/1995486 One candidate fix back-ported https://review.opendev.org/c/starlingx/ptp-notification-armada-app/+/856185 Awaiting a free time slot for a trial build.