[Starlingx-discuss] Builds remain broken

Scott Little scott.little at windriver.com
Tue May 5 20:41:52 UTC 2020 

Flock layer has built.  We have an ISO!

Container build underway

Scott


On 2020-05-05 4:20 p.m., Scott Little wrote:
> ok, this was an interesting one.
>
> The final step of a layer build is to publish the result.  After the 
> result is published, I call a script that uses yum and reposync 
> commands to replicate the repo in the mirror.  The flock layer build 
> should then be able to download distro rpms from the mirror.
>
> Concurrent with the distro build, the generic mirror update script was 
> also running, and uses the same script to update all our upstream repos.
>
> The two fought over possession of the pid file that serves as a lock 
> file fir yum. It's a single yock per user. The distro publisher lost.  
> Because the upstream repos are not always available, the script 
> doesn't fail if one repo fails, it just moves on to the next.  We'll 
> just try again tomorrow.
>
> Try again tomorrow is not an acceptable behavior when it comes to 
> publishing our internal build results.
>
> So I need to make two fixes within CENGN
>
> 1) Use TMPDIR to try and set a per-mirrored-repo lock path, rather 
> than sharing one lock for all yum activity.
>
> 2) Add a flag to the script requesting that failed yum commands are 
> either retried, or fail the job entirely, rather than being silently 
> ignored.
>
> Scott
>
>
> On 2020-05-05 1:24 p.m., Scott Little wrote:
>> CENGN build result ...
>>
>> Distro layer passed. Flock layer failed.
>>
>> I'm investigating the cause.
>>
>> Scott
>>
>>
>> On 2020-05-05 8:15 a.m., Sun, Austin wrote:
>>> Hi Frank.
>>>      Thanks. just reminder  725447 change is not yet merged .
>>>
>>>      yes . I'm ok to revert kernel changes to unblock cengn build if 
>>> any issue is still seeing .
>>>
>>> Thanks a lot .
>>>
>>> BR
>>> Austin Sun.
>>> -----Original Message-----
>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>> Sent: Tuesday, May 5, 2020 8:12 PM
>>> To: Sun, Austin <austin.sun at intel.com>; Rowsell, Brent 
>>> <Brent.Rowsell at windriver.com>; Little, Scott 
>>> <Scott.Little at windriver.com>; 'starlingx-discuss at lists.starlingx.io' 
>>> <starlingx-discuss at lists.starlingx.io>
>>> Subject: RE: [Starlingx-discuss] Builds remain broken
>>>
>>> Austin:
>>>
>>> A member of the build team will trigger a CENGN build shortly. If 
>>> this fails I'll ask one of the kernel Cores to revert so that any 
>>> additional debug is not done on the master branch.
>>>
>>> Frank
>>>
>>> -----Original Message-----
>>> From: Sun, Austin <austin.sun at intel.com>
>>> Sent: Tuesday, May 05, 2020 8:05 AM
>>> To: Rowsell, Brent <Brent.Rowsell at windriver.com>; Miller, Frank 
>>> <Frank.Miller at windriver.com>; Little, Scott 
>>> <Scott.Little at windriver.com>; 'starlingx-discuss at lists.starlingx.io' 
>>> <starlingx-discuss at lists.starlingx.io>
>>> Subject: RE: [Starlingx-discuss] Builds remain broken
>>>
>>> Hi Scott & Frank:
>>>          Shuicheng has uploaded changes [1] to fix distro layer 
>>> builder error.  With this change , layer build is ok in local env.
>>> Details following Scott's guide :
>>> 1) Compiler and distro layer build successfully in a clean and 
>>> separately workspace.
>>> 2) For flock layer,  using Cross layer to build as flock is using 
>>> kernel upgrade distro output in local.
>>>       The mirror/CentOS/stx-installer/ for flock layer is not 
>>> generated after download_mirror.sh and generate-cgcs-centos-repo.sh, 
>>> I manually cp those files to this folder.
>>>       Except this step,  builds-pkgs and build-iso were successfully 
>>> for flock layer.
>>>
>>> Currently we cannot trigger cengn build to verify in cengn env. 
>>> @Scott , would you like to trigger cengn build to check if layer 
>>> build is unblocked after [1] is merged.
>>>
>>> If cengn layer build is still failure w/ change [1] , we are ok to 
>>> revert kernel upgrade changes.
>>>
>>> BTW: is there any guide how to trigger cengn build , are any 
>>> additional permissions needed ?  it might be useful different time 
>>> zone can trigger cengn build .
>>>
>>> [1] https://review.opendev.org/#/c/725447/
>>>
>>>
>>> Thanks.
>>> BR
>>> Austin Sun.
>>>
>>> -----Original Message-----
>>> From: Rowsell, Brent <Brent.Rowsell at windriver.com>
>>> Sent: Tuesday, May 5, 2020 6:38 AM
>>> To: Miller, Frank <Frank.Miller at windriver.com>; Little, Scott 
>>> <Scott.Little at windriver.com>; 'starlingx-discuss at lists.starlingx.io' 
>>> <starlingx-discuss at lists.starlingx.io>
>>> Subject: Re: [Starlingx-discuss] Builds remain broken
>>>
>>> +1
>>>
>>> Brent
>>>
>>> -----Original Message-----
>>> From: Miller, Frank [mailto:Frank.Miller at windriver.com]
>>> Sent: Monday, May 4, 2020 5:44 PM
>>> To: Little, Scott <Scott.Little at windriver.com>; 
>>> 'starlingx-discuss at lists.starlingx.io' 
>>> <starlingx-discuss at lists.starlingx.io>
>>> Subject: Re: [Starlingx-discuss] Builds remain broken
>>>
>>> Austin:
>>>
>>> Thank-you for your fixes earlier today. But as you can see from 
>>> Scott additional fixes are required.  Since our builds have been 
>>> broken for a long stretch (first due to the Train commits last week 
>>> and now due to the kernel commits) the best thing to do is to revert 
>>> the kernel commits so that the master branch can again start having 
>>> successful builds.  Please revert the commits when you start your 
>>> day on Tuesday: 
>>> https://review.opendev.org/#/q/topic:4.18_kernel+(status:open+OR+status:merged)
>>>
>>> Then we ask the team priming the 4.18 kernel to start with a clean 
>>> workspace and mirror that matches exactly what is in the CENGN 
>>> mirror as per Scott's instructions below.  Work through the build 
>>> and identify all dependent packages needed for the build to 
>>> complete.  These dependencies then need to be added first to the 
>>> CENGN mirror before the kernel commits can then be merged back onto 
>>> master.
>>>
>>> Frank
>>>
>>> -----Original Message-----
>>> From: Scott Little <scott.little at windriver.com>
>>> Sent: Monday, May 04, 2020 5:34 PM
>>> To: 'starlingx-discuss at lists.starlingx.io' 
>>> <starlingx-discuss at lists.starlingx.io>
>>> Subject: Re: [Starlingx-discuss] Builds remain broken
>>>
>>> What is critical here is
>>>
>>> 1) start with clean workspace and clean 'output' directories
>>>
>>> 2) build in a layered context
>>>
>>> 3) go through all the steps for each layer
>>>
>>> download_mirror.sh
>>> generate-cgcs-centos-repo.sh
>>> populate_downloads.sh
>>> build-pkgs
>>> build-iso (flock only)
>>>
>>>
>>> Scott
>>>
>>>
>>> On 2020-05-04 5:22 p.m., Scott Little wrote:
>>>> I've fixed several issues with lst files in the wake of the 4.18
>>>> kernel submission to master.
>>>>
>>>>     https://review.opendev.org/725379
>>>>
>>>>     https://review.opendev.org/725371
>>>>
>>>> So far this only addresses the build srmp phase of the distro layer
>>>> build.
>>>>
>>>> The builds remain broken on missing BuildRequires packages or their
>>>> dependencies.  e.g. the std kernel build is now asking for package
>>>> 'perl-generators'.  It's not listed in the distro layer lst files,
>>>> although it is in the flock layer lst file, so a monolithic build
>>>> might pass, but a layered build will not.
>>>>
>>>> I'm tired of trying to fix this thing on the master branch. I think
>>>> we need to revert all the 4.18 kernel related commits (including the
>>>> two above).  i.e. go back to the old kernel and get some successful
>>>> builds working again.
>>>>
>>>> The 4.18 kernel feature needs to be debugged in a developer
>>>> environment.  Particular attention needs to be put into the lst files,
>>>> and to getting those entries in the correct layer.
>>>>
>>>> I would suggest starting two totally new container build environments.
>>>> The first for a distro layer build.  Once that is working, use the
>>>> second for a flock layer build (producing an iso).  Reminder to study
>>>> the Layered build description docs at
>>>> https://docs.starlingx.io/developer_resources/Layered_Build.html
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

More information about the Starlingx-discuss mailing list