It was a really bad week
for build issues on CENGN. Below I try to summarize the we've
been working on.
Branch OS Type Issue
====== == ==== =====
Master Centos Layered Build failures Oct 28-31
Issue: 'Repo' pulled down a non-backward compatible upgrade from upstream that breaks non-containerized uses of 'repo'.
Fix: Updated the version of 'repo' on CENGN bare metal.
Manual build successful: Oct 31
Master Centos Monolithic Build Failure Oct 29
Issue: Corruption of the jenkins build script. Cause of corruption unknown.
Fix: Repaired build script and restarted jenkins
Next build attempt Nov 5
Master Debian Monolithic Build Failures Oct 28-Nov 1
Issue 1: Attempt to optimize the 'publish packages' step had undetected bugs when merged Friday.
Fix 1: corrected on path, and prevented script from throwing error when a grep fails to find a match.
Benefit: 'publish packages' step takes minutes, not hours.
Issue 2: STX_build_debian_master jenkins job was deleted. Unclear how/why but CENGN jenkins was very slow and acting strangely.
Fix 2: restored STX_build_debian_master from git, but we've lost build history. Restarted Jenkins.
Issue 3: Activation of 'reuse' accelerated package build, ~8 hrs -> ~2hrs. Works for ISO, but breaks container image build.
Misunderstood the intent of one recent fix to this feature, thought it would address the container image build issues.
Clearly it did not.
Fix 3: Deactivated 'reuse' feature once again. Longer term, a LP will be raised vs docker images. Docker images
need to pull debs from the aptly published repo, not the traditional repo.
Issue 4: A jenkins job was create to test changes to jenkins pipeline scripts. That job had a name that was
similar to but not identical to the main job. This exposed a defect in how minikube finds containers using
fuzzy rather than exact matches. Cross-talk between main and dev minikube environments broke both builds.
https://bugs.launchpad.net/starlingx/+bug/1995400
Fix 4: Quick fix was to change the dev build name to avoid the 'fuzzy' match, and to nuke/rebuild both correpted
minikube environments. Longer term, we need to use exect matches within minikube.
Success: Build with docker images, TIMESTAMP 20221102T060000Z, Available 20221102T160000Z
stx.6.0 Centos Layered Build Failed Oct 29-Nov 1
Issue: The builder container can't build. pip install pulled in unconstrained packages that have dropped py2.7 compatibility.
https://bugs.launchpad.net/starlingx/+bug/1995484
Fix: Add constraints to maintain py2.7 compatibility in the pip installed packages.
https://review.opendev.org/c/starlingx/tools/+/863265
Test build started
stx.7.0 Centos Layered Build Failed Oct 30
Issue: Failed to build containers locationservice-base, notificationclient-base, notificationservice-base
https://bugs.launchpad.net/starlingx/+bug/1995486
One candidate fix back-ported
https://review.opendev.org/c/starlingx/ptp-notification-armada-app/+/856185
Awaiting a free time slot for a trial build.