[Starlingx-discuss] Slow repo sync times and the yocto kernel
Scott Little
scott.little at windriver.com
Mon Jun 21 20:10:27 UTC 2021
Hi all
The yocto kernel git was added to the StarlingX manifests late last
week. Since then I've heard a lot of grumbling about slow 'repo sync'
times. It affects folks setting up a new distro or monolithic workspace
for the first time. The repo-sync time can exceed an hour as the entire
history of the linux kernel is downloaded. You will also notice an
additional 5.5 GB of storage consumed to hold all this history.
Subsequent repo sync's should be fast.
So the question is... what if anything do we do about it?
Our options...
1) Leave it as is.
Hope that folks are mostly working in the 'flock' or 'container' layers,
and NOT using monolithic builds, and so the number of folk impacted is
low. Folk working at the distro layer or using monolithic can work on
something else, or go to lunch, while they wait for the initial repo
sync to complete.
2) Try to minimize the amount of kernel history we download through a
manifest change. Limiting the git history depth does the trick ...
<!-- linux-yocto Upstream -->
<project remote="linux-yocto" sync-c="true" sync-tags="false"
clone-depth="100" upstream="v5.10/standard/intel-x86"
revision="refs/tags/v5.10.30" name="linux-yocto"
path="cgcs-root/stx/git/linux-yocto-std"/>
<project remote="linux-yocto" sync-c="true" sync-tags="false"
clone-depth="100" upstream="v5.10/standard/preempt-rt/intel-x86"
revision="2112f10d3d0b558c9ece3ab562c41b7f6d89cff4"
name="linux-yocto.git" path="cgcs-root/stx/git/linux-yocto-rt"/>
The good ...
- repo sync time drops from ~1 hr to ~5 min
- storage drops from ~5.5 GB to ~ 5GB
The bad ...
- This is fragile. It assumes that the desired rt sha can be reached
from 100 commits from head of branch. However, the connection to the
upstream git server drops if we ask for much more than that. e.g.
depth=500 is a guaranteed fail. So upstream adds a few patches and we
might start failing our repo-sync.
- The history is incomplete, this may hinder kernel developers. A 'git
fetch linux-yocto' should pull in the rest of the history, so probably
not a blocker.
3) We could double the number of manifests at each layer. One would only
pull in the minimal kernel history, and other the full history.
4) Create a mirror an a larger git server, like github, and hope that
significantly improves the download speed.
5) Download a tarball of the yocto kernel, rather than pulling in it's
git tree. Yocto's git server doesn't seem to be set up to serve custom
tarballs based on a requested sha. We would have to set it all up
manually, and it's not remotely convenient to kernel developers.
Of these options, I'm leaning toward option 2, but look forward to
hearing from the community.
Scott
More information about the Starlingx-discuss
mailing list