[Starlingx-discuss] [Build] CENGN unable to build debian master since March 3
Scott Little
scott.little at windriver.com
Fri Mar 10 14:57:08 UTC 2023
I could only ever reproduce this failure on CENGN.
I believe the problem was a corruption of the cached/buffered copy of a
file in RAM.
After a reboot, *we built successfully.*
/An alternative to a reboot might have been ....
/
/sync; echo 1 > /proc/sys/vm/drop_caches/
Scott
On 2023-03-08 11:05, Scott Little wrote:
> Created a LaunchPad to track the issue:
> https://bugs.launchpad.net/starlingx/+bug/2009722
>
> Scott
>
> On 2023-03-08 10:56, Scott.Little wrote:
>> Hi all
>>
>>
>> CENGN has been unable to complete a build since March 3.
>>
>> Since then, four builds were attempted, and all builds hung within
>> the post-build unit tests of python3.9_3.9.2-1.stx.1.
>>
>> One build was hung for nearly 48 hours.
>>
>> The logs are not specific on which test is hanging.
>>
>> Killing the hung unit test results in the overall build of the
>> python3.9 package build failing, but a retry loop attempts to
>> rebuild, and it again hangs in the unit tests.
>>
>> There is nothing in the change logs that directly affect this
>> package. The only build system changes relate to secure boot.
>> signing and as the python package is not one that requires signing,
>> I'm currently discounting those as a cause.
>>
>> An equivalent build using an internal WindRiver build machine has so
>> far not hit this issue. The main difference being that CENGN uses
>> minikube, and the internal server is using kubernetes directly.
>>
>> One theory was that something change upstream that affects the
>> content within the build containers. However, both CENGN and the
>> internal build server rebuild the build containers each time. If
>> there was a change upstream, I would expect both builds to see it.
>>
>> Most designers are likely using minikube, and so far I've seen no
>> complaints from designers on this topic. Perhaps designers are using
>> a build environment created on or before March 3, and haven't seen it
>> yet. Have you encountered hung builds in the last week? Please
>> report the issue, and we would like to know when the build containers
>> were last rebuilt. We'll be trying to setup a fresh minikube build
>> later today.
>>
>> At some point we may be forced to disable the python unit tests as a
>> work around, until a better solution presents itself.
>>
>> Scott
>>
>>
>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20230310/0f38ca1a/attachment.htm>
More information about the Starlingx-discuss
mailing list