[Starlingx-discuss] Slow repo sync times and the yocto kernel
Scott Little
scott.little at windriver.com
Mon Jun 28 14:54:11 UTC 2021
I have posted a better fix for review ...
https://review.opendev.org/c/starlingx/manifest/+/798340
Scott
On 2021-06-28 10:51 a.m., Scott Little wrote:
> The manifest change to implement option 2 has been merged. However a
> lot of folks are now having 'repo sync' issues with linux-yocto.
> Strangely it doesn't hit the first repo sync, but all subsequent repo
> sync's are affected.
>
> If you are seeing an error from repo sync, please use this work around
> ...
>
> repo init ...
> rm -rf .repo/project-objects/linux-yocto.git*
> .repo/projects/cgcs-root/stx/git/linux-yocto-*
> repo sync ...
>
> I've reported the bug upsteam...
> https://bugs.chromium.org/p/gerrit/issues/detail?id=14700&q=tyranscooter&can=2
> ... if you have a gmail account and are willing to do so, add a star
> to increase visibility of this strange error.
>
> I'll also be looking for a better fix on our end.
>
> Scott
>
>
>
> On 2021-06-21 4:10 p.m., Scott Little wrote:
>> Hi all
>>
>> The yocto kernel git was added to the StarlingX manifests late last
>> week. Since then I've heard a lot of grumbling about slow 'repo
>> sync' times. It affects folks setting up a new distro or monolithic
>> workspace for the first time. The repo-sync time can exceed an hour
>> as the entire history of the linux kernel is downloaded. You will
>> also notice an additional 5.5 GB of storage consumed to hold all this
>> history. Subsequent repo sync's should be fast.
>>
>> So the question is... what if anything do we do about it?
>>
>> Our options...
>>
>> 1) Leave it as is.
>>
>> Hope that folks are mostly working in the 'flock' or 'container'
>> layers, and NOT using monolithic builds, and so the number of folk
>> impacted is low. Folk working at the distro layer or using
>> monolithic can work on something else, or go to lunch, while they
>> wait for the initial repo sync to complete.
>>
>>
>> 2) Try to minimize the amount of kernel history we download through a
>> manifest change. Limiting the git history depth does the trick ...
>>
>> <!-- linux-yocto Upstream -->
>> <project remote="linux-yocto" sync-c="true" sync-tags="false"
>> clone-depth="100" upstream="v5.10/standard/intel-x86"
>> revision="refs/tags/v5.10.30" name="linux-yocto"
>> path="cgcs-root/stx/git/linux-yocto-std"/>
>> <project remote="linux-yocto" sync-c="true" sync-tags="false"
>> clone-depth="100" upstream="v5.10/standard/preempt-rt/intel-x86"
>> revision="2112f10d3d0b558c9ece3ab562c41b7f6d89cff4"
>> name="linux-yocto.git" path="cgcs-root/stx/git/linux-yocto-rt"/>
>>
>> The good ...
>>
>> - repo sync time drops from ~1 hr to ~5 min
>>
>> - storage drops from ~5.5 GB to ~ 5GB
>>
>> The bad ...
>>
>> - This is fragile. It assumes that the desired rt sha can be reached
>> from 100 commits from head of branch. However, the connection to the
>> upstream git server drops if we ask for much more than that. e.g.
>> depth=500 is a guaranteed fail. So upstream adds a few patches and
>> we might start failing our repo-sync.
>>
>> - The history is incomplete, this may hinder kernel developers. A
>> 'git fetch linux-yocto' should pull in the rest of the history, so
>> probably not a blocker.
>>
>>
>> 3) We could double the number of manifests at each layer. One would
>> only pull in the minimal kernel history, and other the full history.
>>
>>
>> 4) Create a mirror an a larger git server, like github, and hope that
>> significantly improves the download speed.
>>
>>
>> 5) Download a tarball of the yocto kernel, rather than pulling in
>> it's git tree. Yocto's git server doesn't seem to be set up to serve
>> custom tarballs based on a requested sha. We would have to set it
>> all up manually, and it's not remotely convenient to kernel developers.
>>
>>
>> Of these options, I'm leaning toward option 2, but look forward to
>> hearing from the community.
>>
>>
>> Scott
>>
>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
More information about the Starlingx-discuss
mailing list