[Starlingx-discuss] [Build] CENGN unable to build debian master since March 3
scott.little at windriver.com
Fri Mar 17 14:06:34 UTC 2023
CENGN is again experiencing hangs in the Python3.9 unit tests. We are
also now reproducing the hang on other build machines.
I recommend we disable unit tests in the python3.9 package, until we
better understand the issue.
On 2023-03-10 09:57, Scott Little wrote:
> I could only ever reproduce this failure on CENGN.
> I believe the problem was a corruption of the cached/buffered copy of
> a file in RAM.
> After a reboot, *we built successfully.*
> /An alternative to a reboot might have been ....
> /sync; echo 1 > /proc/sys/vm/drop_caches/
> On 2023-03-08 11:05, Scott Little wrote:
>> Created a LaunchPad to track the issue:
>> On 2023-03-08 10:56, Scott.Little wrote:
>>> Hi all
>>> CENGN has been unable to complete a build since March 3.
>>> Since then, four builds were attempted, and all builds hung within
>>> the post-build unit tests of python3.9_3.9.2-1.stx.1.
>>> One build was hung for nearly 48 hours.
>>> The logs are not specific on which test is hanging.
>>> Killing the hung unit test results in the overall build of the
>>> python3.9 package build failing, but a retry loop attempts to
>>> rebuild, and it again hangs in the unit tests.
>>> There is nothing in the change logs that directly affect this
>>> package. The only build system changes relate to secure boot.
>>> signing and as the python package is not one that requires signing,
>>> I'm currently discounting those as a cause.
>>> An equivalent build using an internal WindRiver build machine has so
>>> far not hit this issue. The main difference being that CENGN uses
>>> minikube, and the internal server is using kubernetes directly.
>>> One theory was that something change upstream that affects the
>>> content within the build containers. However, both CENGN and the
>>> internal build server rebuild the build containers each time. If
>>> there was a change upstream, I would expect both builds to see it.
>>> Most designers are likely using minikube, and so far I've seen no
>>> complaints from designers on this topic. Perhaps designers are
>>> using a build environment created on or before March 3, and haven't
>>> seen it yet. Have you encountered hung builds in the last week?
>>> Please report the issue, and we would like to know when the build
>>> containers were last rebuilt. We'll be trying to setup a fresh
>>> minikube build later today.
>>> At some point we may be forced to disable the python unit tests as a
>>> work around, until a better solution presents itself.
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Starlingx-discuss