[Starlingx-discuss] [Build] CENGN unable to build debian master since March 3
Scott Little
scott.little at windriver.com
Fri Mar 17 14:06:34 UTC 2023
CENGN is again experiencing hangs in the Python3.9 unit tests. We are
also now reproducing the hang on other build machines.
LaunchPad: https://bugs.launchpad.net/starlingx/+bug/2009722
I recommend we disable unit tests in the python3.9 package, until we
better understand the issue.
Scott
On 2023-03-10 09:57, Scott Little wrote:
>
> I could only ever reproduce this failure on CENGN.
>
> I believe the problem was a corruption of the cached/buffered copy of
> a file in RAM.
>
> After a reboot, *we built successfully.*
>
> /An alternative to a reboot might have been ....
> /
>
> /sync; echo 1 > /proc/sys/vm/drop_caches/
>
> Scott
>
>
> On 2023-03-08 11:05, Scott Little wrote:
>> Created a LaunchPad to track the issue:
>> https://bugs.launchpad.net/starlingx/+bug/2009722
>>
>> Scott
>>
>> On 2023-03-08 10:56, Scott.Little wrote:
>>> Hi all
>>>
>>>
>>> CENGN has been unable to complete a build since March 3.
>>>
>>> Since then, four builds were attempted, and all builds hung within
>>> the post-build unit tests of python3.9_3.9.2-1.stx.1.
>>>
>>> One build was hung for nearly 48 hours.
>>>
>>> The logs are not specific on which test is hanging.
>>>
>>> Killing the hung unit test results in the overall build of the
>>> python3.9 package build failing, but a retry loop attempts to
>>> rebuild, and it again hangs in the unit tests.
>>>
>>> There is nothing in the change logs that directly affect this
>>> package. The only build system changes relate to secure boot.
>>> signing and as the python package is not one that requires signing,
>>> I'm currently discounting those as a cause.
>>>
>>> An equivalent build using an internal WindRiver build machine has so
>>> far not hit this issue. The main difference being that CENGN uses
>>> minikube, and the internal server is using kubernetes directly.
>>>
>>> One theory was that something change upstream that affects the
>>> content within the build containers. However, both CENGN and the
>>> internal build server rebuild the build containers each time. If
>>> there was a change upstream, I would expect both builds to see it.
>>>
>>> Most designers are likely using minikube, and so far I've seen no
>>> complaints from designers on this topic. Perhaps designers are
>>> using a build environment created on or before March 3, and haven't
>>> seen it yet. Have you encountered hung builds in the last week?
>>> Please report the issue, and we would like to know when the build
>>> containers were last rebuilt. We'll be trying to setup a fresh
>>> minikube build later today.
>>>
>>> At some point we may be forced to disable the python unit tests as a
>>> work around, until a better solution presents itself.
>>>
>>> Scott
>>>
>>>
>>>
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>
>>
>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20230317/e8233c30/attachment-0001.htm>
More information about the Starlingx-discuss
mailing list