[Starlingx-discuss] Slow repo sync times and the yocto kernel

Scott Little scott.little at windriver.com
Mon Jun 28 14:51:30 UTC 2021


The manifest change to implement option 2 has been merged.  However a 
lot of folks are now having 'repo sync' issues with linux-yocto.  
Strangely it doesn't hit the first repo sync, but all subsequent repo 
sync's are affected.

If you are seeing an error from repo sync, please use this work around ...

repo init ...
rm -rf .repo/project-objects/linux-yocto.git* 
.repo/projects/cgcs-root/stx/git/linux-yocto-*
repo sync ...

I've reported the bug upsteam... 
https://bugs.chromium.org/p/gerrit/issues/detail?id=14700&q=tyranscooter&can=2 
... if you have a gmail account and are willing to do so, add a star to 
increase visibility of this strange error.

I'll also be looking for a better fix on our end.

Scott



On 2021-06-21 4:10 p.m., Scott Little wrote:
> Hi all
>
> The yocto kernel git was added to the StarlingX manifests late last 
> week.  Since then I've heard a lot of grumbling about slow 'repo sync' 
> times.  It affects folks setting up a new distro or monolithic 
> workspace for the first time.  The repo-sync time can exceed an hour 
> as the entire history of the linux kernel is downloaded.  You will 
> also notice an additional 5.5 GB of storage consumed to hold all this 
> history.  Subsequent repo sync's should be fast.
>
> So the question is... what if anything do we do about it?
>
> Our options...
>
> 1) Leave it as is.
>
> Hope that folks are mostly working in the 'flock' or 'container' 
> layers, and NOT using monolithic builds, and so the number of folk 
> impacted is low.   Folk working at the distro layer or using 
> monolithic can work on something else, or go to lunch, while they wait 
> for the initial repo sync to complete.
>
>
> 2) Try to minimize the amount of kernel history we download through a 
> manifest change.  Limiting the git history depth does the trick ...
>
>    <!-- linux-yocto Upstream -->
>   <project remote="linux-yocto"    sync-c="true" sync-tags="false" 
> clone-depth="100" upstream="v5.10/standard/intel-x86" 
> revision="refs/tags/v5.10.30" name="linux-yocto" 
> path="cgcs-root/stx/git/linux-yocto-std"/>
>   <project remote="linux-yocto"    sync-c="true" sync-tags="false" 
> clone-depth="100" upstream="v5.10/standard/preempt-rt/intel-x86" 
> revision="2112f10d3d0b558c9ece3ab562c41b7f6d89cff4" 
> name="linux-yocto.git" path="cgcs-root/stx/git/linux-yocto-rt"/>
>
> The good ...
>
>  - repo sync time drops from ~1 hr to ~5 min
>
>  - storage drops from ~5.5 GB to ~ 5GB
>
> The bad ...
>
> - This is fragile.  It assumes that the desired rt sha can be reached 
> from 100 commits from head of branch.  However, the connection to the 
> upstream git server drops if we ask for much more than that. e.g. 
> depth=500 is a guaranteed fail.  So upstream adds a few patches and we 
> might start failing our repo-sync.
>
> - The history is incomplete, this may hinder kernel developers. A 'git 
> fetch linux-yocto' should pull in the rest of the history, so probably 
> not a blocker.
>
>
> 3) We could double the number of manifests at each layer. One would 
> only pull in the minimal kernel history, and other the full history.
>
>
> 4) Create a mirror an a larger git server, like github, and hope that 
> significantly improves the download speed.
>
>
> 5) Download a tarball of the yocto kernel, rather than pulling in it's 
> git tree.  Yocto's git server doesn't seem to be set up to serve 
> custom tarballs based on a requested sha.  We would have to set it all 
> up manually, and it's not remotely convenient to kernel developers.
>
>
> Of these options, I'm leaning toward option 2, but look forward to 
> hearing from the community.
>
>
> Scott
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss





More information about the Starlingx-discuss mailing list