[Starlingx-discuss] [Build] CENGN unable to build debian master since March 3

Scott Little scott.little at windriver.com
Fri Mar 17 14:06:34 UTC 2023


CENGN is again experiencing hangs in the Python3.9 unit tests.  We are 
also now reproducing the hang on other build machines.

LaunchPad: https://bugs.launchpad.net/starlingx/+bug/2009722

I recommend we disable unit tests in the python3.9 package, until we 
better understand the issue.

Scott


On 2023-03-10 09:57, Scott Little wrote:
>
> I could only ever reproduce this failure on CENGN.
>
> I believe the problem was a corruption of the cached/buffered copy of 
> a file in RAM.
>
> After a reboot, *we built successfully.*
>
> /An alternative to a reboot might have been ....
> /
>
> /sync; echo 1 > /proc/sys/vm/drop_caches/
>
> Scott
>
>
> On 2023-03-08 11:05, Scott Little wrote:
>> Created a LaunchPad to track the issue:
>> https://bugs.launchpad.net/starlingx/+bug/2009722
>>
>> Scott
>>
>> On 2023-03-08 10:56, Scott.Little wrote:
>>> Hi all
>>>
>>>
>>> CENGN has been unable to complete a build since March 3.
>>>
>>> Since then, four builds were attempted, and all builds hung within 
>>> the post-build unit tests of python3.9_3.9.2-1.stx.1.
>>>
>>> One build was hung for nearly 48 hours.
>>>
>>> The logs are not specific on which test is hanging.
>>>
>>> Killing the hung unit test results in the overall build of the 
>>> python3.9 package build failing, but a retry loop attempts to 
>>> rebuild, and it again hangs in the unit tests.
>>>
>>> There is nothing in the change logs that directly affect this 
>>> package.  The only build system changes relate to secure boot. 
>>> signing and as the python package is not one that requires signing, 
>>> I'm currently discounting those as a cause.
>>>
>>> An equivalent build using an internal WindRiver build machine has so 
>>> far not hit this issue.  The main difference being that CENGN uses 
>>> minikube, and the internal server is using kubernetes directly.
>>>
>>> One theory was that something change upstream that affects the 
>>> content within the build containers.  However, both CENGN and the 
>>> internal build server rebuild the build containers each time.  If 
>>> there was a change upstream, I would expect both builds to see it.
>>>
>>> Most designers are likely using minikube, and so far I've seen no 
>>> complaints from designers on this topic.  Perhaps designers are 
>>> using a build environment created on or before March 3, and haven't 
>>> seen it yet.  Have you encountered hung builds in the last week?   
>>> Please report the issue, and we would like to know when the build 
>>> containers were last rebuilt.   We'll be trying to setup a fresh 
>>> minikube build later today.
>>>
>>> At some point we may be forced to disable the python unit tests as a 
>>> work around, until a better solution presents itself.
>>>
>>> Scott
>>>
>>>
>>>
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>
>>
>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20230317/e8233c30/attachment-0001.htm>


More information about the Starlingx-discuss mailing list