[Starlingx-discuss] [Build] CENGN unable to build debian master since March 3

Scott Little scott.little at windriver.com
Fri Mar 10 14:57:08 UTC 2023


I could only ever reproduce this failure on CENGN.

I believe the problem was a corruption of the cached/buffered copy of a 
file in RAM.

After a reboot, *we built successfully.*

/An alternative to a reboot might have been ....
/

/sync; echo 1 > /proc/sys/vm/drop_caches/

Scott


On 2023-03-08 11:05, Scott Little wrote:
> Created a LaunchPad to track the issue:
> https://bugs.launchpad.net/starlingx/+bug/2009722
>
> Scott
>
> On 2023-03-08 10:56, Scott.Little wrote:
>> Hi all
>>
>>
>> CENGN has been unable to complete a build since March 3.
>>
>> Since then, four builds were attempted, and all builds hung within 
>> the post-build unit tests of python3.9_3.9.2-1.stx.1.
>>
>> One build was hung for nearly 48 hours.
>>
>> The logs are not specific on which test is hanging.
>>
>> Killing the hung unit test results in the overall build of the 
>> python3.9 package build failing, but a retry loop attempts to 
>> rebuild, and it again hangs in the unit tests.
>>
>> There is nothing in the change logs that directly affect this 
>> package.  The only build system changes relate to secure boot. 
>> signing and as the python package is not one that requires signing, 
>> I'm currently discounting those as a cause.
>>
>> An equivalent build using an internal WindRiver build machine has so 
>> far not hit this issue.  The main difference being that CENGN uses 
>> minikube, and the internal server is using kubernetes directly.
>>
>> One theory was that something change upstream that affects the 
>> content within the build containers.  However, both CENGN and the 
>> internal build server rebuild the build containers each time.  If 
>> there was a change upstream, I would expect both builds to see it.
>>
>> Most designers are likely using minikube, and so far I've seen no 
>> complaints from designers on this topic.  Perhaps designers are using 
>> a build environment created on or before March 3, and haven't seen it 
>> yet.  Have you encountered hung builds in the last week?   Please 
>> report the issue, and we would like to know when the build containers 
>> were last rebuilt.   We'll be trying to setup a fresh minikube build 
>> later today.
>>
>> At some point we may be forced to disable the python unit tests as a 
>> work around, until a better solution presents itself.
>>
>> Scott
>>
>>
>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20230310/0f38ca1a/attachment.htm>


More information about the Starlingx-discuss mailing list