[Starlingx-discuss] Summary of the numerous build issues to strike CENGN in the past week

Scott Little scott.little at windriver.com
Wed Nov 2 17:09:15 UTC 2022


It was a really bad week for build issues on CENGN.  Below I try to 
summarize the we've been working on.

Scott

PS: use a wide window to view!

Branch	OS	Type		Issue
======	==	====		=====

Master	Centos	Layered		Build failures Oct 28-31
				Issue: 'Repo' pulled down a non-backward compatible upgrade from upstream that breaks non-containerized uses of 'repo'.
				Fix: Updated the version of 'repo' on CENGN bare metal.
                                 Manual build successful: Oct 31

Master	Centos	Monolithic	Build Failure Oct 29
				Issue: Corruption of the jenkins build script.  Cause of corruption unknown.
                                 Fix: Repaired build script and restarted jenkins
				Next build attempt Nov 5

Master	Debian	Monolithic      Build Failures Oct 28-Nov 1
                                 Issue 1: Attempt to optimize the 'publish packages' step had undetected bugs when merged Friday.
				Fix 1: corrected on path, and prevented script from throwing error when a grep fails to find a match.
				Benefit: 'publish packages' step takes minutes, not hours.
				Issue 2: STX_build_debian_master jenkins job was deleted.  Unclear how/why but CENGN jenkins was very slow and acting strangely.
				Fix 2: restored STX_build_debian_master from git, but we've lost build history.  Restarted Jenkins.
				Issue 3: Activation of 'reuse' accelerated package build, ~8 hrs -> ~2hrs. Works for ISO, but breaks container image build.
					Misunderstood the intent of one recent fix to this feature, thought it would address the container image build issues.
					Clearly it did not.
				Fix 3: Deactivated 'reuse' feature once again.  Longer term, a LP will be raised vs docker images.  Docker images
					need to pull debs from the aptly published repo, not the traditional repo.
				Issue 4: A jenkins job was create to test changes to jenkins pipeline scripts. That job had a name that was
					similar to but not identical to the main job. This exposed a defect in how minikube finds containers using
					fuzzy rather than exact matches.  Cross-talk between main and dev minikube environments broke both builds.
					https://bugs.launchpad.net/starlingx/+bug/1995400  <https://bugs.launchpad.net/starlingx/+bug/1995400>
				Fix 4: Quick fix was to change the dev build name to avoid the 'fuzzy' match, and to nuke/rebuild both correpted
					minikube environments. Longer term, we need to use exect matches within minikube.
				Success: Build with docker images, TIMESTAMP 20221102T060000Z, Available 20221102T160000Z

stx.6.0	Centos  Layered		Build Failed Oct 29-Nov 1
				Issue: The builder container can't build.  pip install pulled in unconstrained packages that have dropped py2.7 compatibility.
					https://bugs.launchpad.net/starlingx/+bug/1995484
				Fix: Add constraints to maintain py2.7 compatibility in the pip installed packages.
					https://review.opendev.org/c/starlingx/tools/+/863265
				Test build started

stx.7.0	Centos	Layered		Build Failed Oct 30
				Issue: Failed to build containers locationservice-base, notificationclient-base, notificationservice-base
					https://bugs.launchpad.net/starlingx/+bug/1995486
				One candidate fix back-ported
					https://review.opendev.org/c/starlingx/ptp-notification-armada-app/+/856185
				Awaiting a free time slot for a trial build.


				
				


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20221102/fa5523b6/attachment-0001.htm>


More information about the Starlingx-discuss mailing list