[Starlingx-discuss] Slow repo sync times and the yocto kernel

Scott Little scott.little at windriver.com
Mon Jun 21 20:10:27 UTC 2021


Hi all

The yocto kernel git was added to the StarlingX manifests late last 
week.  Since then I've heard a lot of grumbling about slow 'repo sync' 
times.  It affects folks setting up a new distro or monolithic workspace 
for the first time.  The repo-sync time can exceed an hour as the entire 
history of the linux kernel is downloaded.  You will also notice an 
additional 5.5 GB of storage consumed to hold all this history.  
Subsequent repo sync's should be fast.

So the question is... what if anything do we do about it?

Our options...

1) Leave it as is.

Hope that folks are mostly working in the 'flock' or 'container' layers, 
and NOT using monolithic builds, and so the number of folk impacted is 
low.   Folk working at the distro layer or using monolithic can work on 
something else, or go to lunch, while they wait for the initial repo 
sync to complete.


2) Try to minimize the amount of kernel history we download through a 
manifest change.  Limiting the git history depth does the trick ...

    <!-- linux-yocto Upstream -->
   <project remote="linux-yocto"    sync-c="true" sync-tags="false" 
clone-depth="100" upstream="v5.10/standard/intel-x86" 
revision="refs/tags/v5.10.30" name="linux-yocto" 
path="cgcs-root/stx/git/linux-yocto-std"/>
   <project remote="linux-yocto"    sync-c="true" sync-tags="false" 
clone-depth="100" upstream="v5.10/standard/preempt-rt/intel-x86" 
revision="2112f10d3d0b558c9ece3ab562c41b7f6d89cff4" 
name="linux-yocto.git" path="cgcs-root/stx/git/linux-yocto-rt"/>

The good ...

  - repo sync time drops from ~1 hr to ~5 min

  - storage drops from ~5.5 GB to ~ 5GB

The bad ...

- This is fragile.  It assumes that the desired rt sha can be reached 
from 100 commits from head of branch.  However, the connection to the 
upstream git server drops if we ask for much more than that. e.g. 
depth=500 is a guaranteed fail.  So upstream adds a few patches and we 
might start failing our repo-sync.

- The history is incomplete, this may hinder kernel developers. A 'git 
fetch linux-yocto' should pull in the rest of the history, so probably 
not a blocker.


3) We could double the number of manifests at each layer. One would only 
pull in the minimal kernel history, and other the full history.


4) Create a mirror an a larger git server, like github, and hope that 
significantly improves the download speed.


5) Download a tarball of the yocto kernel, rather than pulling in it's 
git tree.  Yocto's git server doesn't seem to be set up to serve custom 
tarballs based on a requested sha.  We would have to set it all up 
manually, and it's not remotely convenient to kernel developers.


Of these options, I'm leaning toward option 2, but look forward to 
hearing from the community.


Scott




More information about the Starlingx-discuss mailing list