[Starlingx-discuss] Slow repo sync times and the yocto kernel

Scott Little scott.little at windriver.com
Mon Jun 28 14:54:11 UTC 2021


I have posted a better fix for review ...

https://review.opendev.org/c/starlingx/manifest/+/798340

Scott


On 2021-06-28 10:51 a.m., Scott Little wrote:
> The manifest change to implement option 2 has been merged.  However a 
> lot of folks are now having 'repo sync' issues with linux-yocto. 
> Strangely it doesn't hit the first repo sync, but all subsequent repo 
> sync's are affected.
>
> If you are seeing an error from repo sync, please use this work around 
> ...
>
> repo init ...
> rm -rf .repo/project-objects/linux-yocto.git* 
> .repo/projects/cgcs-root/stx/git/linux-yocto-*
> repo sync ...
>
> I've reported the bug upsteam... 
> https://bugs.chromium.org/p/gerrit/issues/detail?id=14700&q=tyranscooter&can=2 
> ... if you have a gmail account and are willing to do so, add a star 
> to increase visibility of this strange error.
>
> I'll also be looking for a better fix on our end.
>
> Scott
>
>
>
> On 2021-06-21 4:10 p.m., Scott Little wrote:
>> Hi all
>>
>> The yocto kernel git was added to the StarlingX manifests late last 
>> week.  Since then I've heard a lot of grumbling about slow 'repo 
>> sync' times.  It affects folks setting up a new distro or monolithic 
>> workspace for the first time.  The repo-sync time can exceed an hour 
>> as the entire history of the linux kernel is downloaded.  You will 
>> also notice an additional 5.5 GB of storage consumed to hold all this 
>> history.  Subsequent repo sync's should be fast.
>>
>> So the question is... what if anything do we do about it?
>>
>> Our options...
>>
>> 1) Leave it as is.
>>
>> Hope that folks are mostly working in the 'flock' or 'container' 
>> layers, and NOT using monolithic builds, and so the number of folk 
>> impacted is low.   Folk working at the distro layer or using 
>> monolithic can work on something else, or go to lunch, while they 
>> wait for the initial repo sync to complete.
>>
>>
>> 2) Try to minimize the amount of kernel history we download through a 
>> manifest change.  Limiting the git history depth does the trick ...
>>
>>    <!-- linux-yocto Upstream -->
>>   <project remote="linux-yocto"    sync-c="true" sync-tags="false" 
>> clone-depth="100" upstream="v5.10/standard/intel-x86" 
>> revision="refs/tags/v5.10.30" name="linux-yocto" 
>> path="cgcs-root/stx/git/linux-yocto-std"/>
>>   <project remote="linux-yocto"    sync-c="true" sync-tags="false" 
>> clone-depth="100" upstream="v5.10/standard/preempt-rt/intel-x86" 
>> revision="2112f10d3d0b558c9ece3ab562c41b7f6d89cff4" 
>> name="linux-yocto.git" path="cgcs-root/stx/git/linux-yocto-rt"/>
>>
>> The good ...
>>
>>  - repo sync time drops from ~1 hr to ~5 min
>>
>>  - storage drops from ~5.5 GB to ~ 5GB
>>
>> The bad ...
>>
>> - This is fragile.  It assumes that the desired rt sha can be reached 
>> from 100 commits from head of branch.  However, the connection to the 
>> upstream git server drops if we ask for much more than that. e.g. 
>> depth=500 is a guaranteed fail.  So upstream adds a few patches and 
>> we might start failing our repo-sync.
>>
>> - The history is incomplete, this may hinder kernel developers. A 
>> 'git fetch linux-yocto' should pull in the rest of the history, so 
>> probably not a blocker.
>>
>>
>> 3) We could double the number of manifests at each layer. One would 
>> only pull in the minimal kernel history, and other the full history.
>>
>>
>> 4) Create a mirror an a larger git server, like github, and hope that 
>> significantly improves the download speed.
>>
>>
>> 5) Download a tarball of the yocto kernel, rather than pulling in 
>> it's git tree.  Yocto's git server doesn't seem to be set up to serve 
>> custom tarballs based on a requested sha.  We would have to set it 
>> all up manually, and it's not remotely convenient to kernel developers.
>>
>>
>> Of these options, I'm leaning toward option 2, but look forward to 
>> hearing from the community.
>>
>>
>> Scott
>>
>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss





More information about the Starlingx-discuss mailing list