Scott, Great Job! On 5/5/20 1:20 PM, Scott Little wrote:
ok, this was an interesting one.
The final step of a layer build is to publish the result. After the result is published, I call a script that uses yum and reposync commands to replicate the repo in the mirror. The flock layer build should then be able to download distro rpms from the mirror.
Concurrent with the distro build, the generic mirror update script was also running, and uses the same script to update all our upstream repos.
The two fought over possession of the pid file that serves as a lock file fir yum. It's a single yock per user. The distro publisher lost. Because the upstream repos are not always available, the script doesn't fail if one repo fails, it just moves on to the next. We'll just try again tomorrow.
Try again tomorrow is not an acceptable behavior when it comes to publishing our internal build results.
Good Explanation! So, we might have stumbled on this with other updates at somepoint.
So I need to make two fixes within CENGN
1) Use TMPDIR to try and set a per-mirrored-repo lock path, rather than sharing one lock for all yum activity.
2) Add a flag to the script requesting that failed yum commands are either retried, or fail the job entirely, rather than being silently ignored.
Let me know if you need something reviewed. Sau!
Scott
On 2020-05-05 1:24 p.m., Scott Little wrote:
CENGN build result ...
Distro layer passed. Flock layer failed.
I'm investigating the cause.
Scott
On 2020-05-05 8:15 a.m., Sun, Austin wrote:
Hi Frank. Thanks. just reminder 725447 change is not yet merged .
yes . I'm ok to revert kernel changes to unblock cengn build if any issue is still seeing .
Thanks a lot .
BR Austin Sun. -----Original Message----- From: Miller, Frank <Frank.Miller@windriver.com> Sent: Tuesday, May 5, 2020 8:12 PM To: Sun, Austin <austin.sun@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com>; Little, Scott <Scott.Little@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] Builds remain broken
Austin:
A member of the build team will trigger a CENGN build shortly. If this fails I'll ask one of the kernel Cores to revert so that any additional debug is not done on the master branch.
Frank
-----Original Message----- From: Sun, Austin <austin.sun@intel.com> Sent: Tuesday, May 05, 2020 8:05 AM To: Rowsell, Brent <Brent.Rowsell@windriver.com>; Miller, Frank <Frank.Miller@windriver.com>; Little, Scott <Scott.Little@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] Builds remain broken
Hi Scott & Frank: Shuicheng has uploaded changes [1] to fix distro layer builder error. With this change , layer build is ok in local env. Details following Scott's guide : 1) Compiler and distro layer build successfully in a clean and separately workspace. 2) For flock layer, using Cross layer to build as flock is using kernel upgrade distro output in local. The mirror/CentOS/stx-installer/ for flock layer is not generated after download_mirror.sh and generate-cgcs-centos-repo.sh, I manually cp those files to this folder. Except this step, builds-pkgs and build-iso were successfully for flock layer.
Currently we cannot trigger cengn build to verify in cengn env. @Scott , would you like to trigger cengn build to check if layer build is unblocked after [1] is merged.
If cengn layer build is still failure w/ change [1] , we are ok to revert kernel upgrade changes.
BTW: is there any guide how to trigger cengn build , are any additional permissions needed ? it might be useful different time zone can trigger cengn build .
[1] https://review.opendev.org/#/c/725447/
Thanks. BR Austin Sun.
-----Original Message----- From: Rowsell, Brent <Brent.Rowsell@windriver.com> Sent: Tuesday, May 5, 2020 6:38 AM To: Miller, Frank <Frank.Miller@windriver.com>; Little, Scott <Scott.Little@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Builds remain broken
+1
Brent
-----Original Message----- From: Miller, Frank [mailto:Frank.Miller@windriver.com] Sent: Monday, May 4, 2020 5:44 PM To: Little, Scott <Scott.Little@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Builds remain broken
Austin:
Thank-you for your fixes earlier today. But as you can see from Scott additional fixes are required. Since our builds have been broken for a long stretch (first due to the Train commits last week and now due to the kernel commits) the best thing to do is to revert the kernel commits so that the master branch can again start having successful builds. Please revert the commits when you start your day on Tuesday: https://review.opendev.org/#/q/topic:4.18_kernel+(status:open+OR+status:merg...)
Then we ask the team priming the 4.18 kernel to start with a clean workspace and mirror that matches exactly what is in the CENGN mirror as per Scott's instructions below. Work through the build and identify all dependent packages needed for the build to complete. These dependencies then need to be added first to the CENGN mirror before the kernel commits can then be merged back onto master.
Frank
-----Original Message----- From: Scott Little <scott.little@windriver.com> Sent: Monday, May 04, 2020 5:34 PM To: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Builds remain broken
What is critical here is
1) start with clean workspace and clean 'output' directories
2) build in a layered context
3) go through all the steps for each layer
download_mirror.sh generate-cgcs-centos-repo.sh populate_downloads.sh build-pkgs build-iso (flock only)
Scott
On 2020-05-04 5:22 p.m., Scott Little wrote:
I've fixed several issues with lst files in the wake of the 4.18 kernel submission to master.
https://review.opendev.org/725379
https://review.opendev.org/725371
So far this only addresses the build srmp phase of the distro layer build.
The builds remain broken on missing BuildRequires packages or their dependencies. e.g. the std kernel build is now asking for package 'perl-generators'. It's not listed in the distro layer lst files, although it is in the flock layer lst file, so a monolithic build might pass, but a layered build will not.
I'm tired of trying to fix this thing on the master branch. I think we need to revert all the 4.18 kernel related commits (including the two above). i.e. go back to the old kernel and get some successful builds working again.
The 4.18 kernel feature needs to be debugged in a developer environment. Particular attention needs to be put into the lst files, and to getting those entries in the correct layer.
I would suggest starting two totally new container build environments. The first for a distro layer build. Once that is working, use the second for a flock layer build (producing an iso). Reminder to study the Layered build description docs at https://docs.starlingx.io/developer_resources/Layered_Build.html
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss