[Starlingx-discuss] Slow repo sync times and the yocto kernel
Hi all The yocto kernel git was added to the StarlingX manifests late last week. Since then I've heard a lot of grumbling about slow 'repo sync' times. It affects folks setting up a new distro or monolithic workspace for the first time. The repo-sync time can exceed an hour as the entire history of the linux kernel is downloaded. You will also notice an additional 5.5 GB of storage consumed to hold all this history. Subsequent repo sync's should be fast. So the question is... what if anything do we do about it? Our options... 1) Leave it as is. Hope that folks are mostly working in the 'flock' or 'container' layers, and NOT using monolithic builds, and so the number of folk impacted is low. Folk working at the distro layer or using monolithic can work on something else, or go to lunch, while they wait for the initial repo sync to complete. 2) Try to minimize the amount of kernel history we download through a manifest change. Limiting the git history depth does the trick ... <!-- linux-yocto Upstream --> <project remote="linux-yocto" sync-c="true" sync-tags="false" clone-depth="100" upstream="v5.10/standard/intel-x86" revision="refs/tags/v5.10.30" name="linux-yocto" path="cgcs-root/stx/git/linux-yocto-std"/> <project remote="linux-yocto" sync-c="true" sync-tags="false" clone-depth="100" upstream="v5.10/standard/preempt-rt/intel-x86" revision="2112f10d3d0b558c9ece3ab562c41b7f6d89cff4" name="linux-yocto.git" path="cgcs-root/stx/git/linux-yocto-rt"/> The good ... - repo sync time drops from ~1 hr to ~5 min - storage drops from ~5.5 GB to ~ 5GB The bad ... - This is fragile. It assumes that the desired rt sha can be reached from 100 commits from head of branch. However, the connection to the upstream git server drops if we ask for much more than that. e.g. depth=500 is a guaranteed fail. So upstream adds a few patches and we might start failing our repo-sync. - The history is incomplete, this may hinder kernel developers. A 'git fetch linux-yocto' should pull in the rest of the history, so probably not a blocker. 3) We could double the number of manifests at each layer. One would only pull in the minimal kernel history, and other the full history. 4) Create a mirror an a larger git server, like github, and hope that significantly improves the download speed. 5) Download a tarball of the yocto kernel, rather than pulling in it's git tree. Yocto's git server doesn't seem to be set up to serve custom tarballs based on a requested sha. We would have to set it all up manually, and it's not remotely convenient to kernel developers. Of these options, I'm leaning toward option 2, but look forward to hearing from the community. Scott
The manifest change to implement option 2 has been merged. However a lot of folks are now having 'repo sync' issues with linux-yocto. Strangely it doesn't hit the first repo sync, but all subsequent repo sync's are affected. If you are seeing an error from repo sync, please use this work around ... repo init ... rm -rf .repo/project-objects/linux-yocto.git* .repo/projects/cgcs-root/stx/git/linux-yocto-* repo sync ... I've reported the bug upsteam... https://bugs.chromium.org/p/gerrit/issues/detail?id=14700&q=tyranscooter&can=2 ... if you have a gmail account and are willing to do so, add a star to increase visibility of this strange error. I'll also be looking for a better fix on our end. Scott On 2021-06-21 4:10 p.m., Scott Little wrote:
Hi all
The yocto kernel git was added to the StarlingX manifests late last week. Since then I've heard a lot of grumbling about slow 'repo sync' times. It affects folks setting up a new distro or monolithic workspace for the first time. The repo-sync time can exceed an hour as the entire history of the linux kernel is downloaded. You will also notice an additional 5.5 GB of storage consumed to hold all this history. Subsequent repo sync's should be fast.
So the question is... what if anything do we do about it?
Our options...
1) Leave it as is.
Hope that folks are mostly working in the 'flock' or 'container' layers, and NOT using monolithic builds, and so the number of folk impacted is low. Folk working at the distro layer or using monolithic can work on something else, or go to lunch, while they wait for the initial repo sync to complete.
2) Try to minimize the amount of kernel history we download through a manifest change. Limiting the git history depth does the trick ...
<!-- linux-yocto Upstream --> <project remote="linux-yocto" sync-c="true" sync-tags="false" clone-depth="100" upstream="v5.10/standard/intel-x86" revision="refs/tags/v5.10.30" name="linux-yocto" path="cgcs-root/stx/git/linux-yocto-std"/> <project remote="linux-yocto" sync-c="true" sync-tags="false" clone-depth="100" upstream="v5.10/standard/preempt-rt/intel-x86" revision="2112f10d3d0b558c9ece3ab562c41b7f6d89cff4" name="linux-yocto.git" path="cgcs-root/stx/git/linux-yocto-rt"/>
The good ...
- repo sync time drops from ~1 hr to ~5 min
- storage drops from ~5.5 GB to ~ 5GB
The bad ...
- This is fragile. It assumes that the desired rt sha can be reached from 100 commits from head of branch. However, the connection to the upstream git server drops if we ask for much more than that. e.g. depth=500 is a guaranteed fail. So upstream adds a few patches and we might start failing our repo-sync.
- The history is incomplete, this may hinder kernel developers. A 'git fetch linux-yocto' should pull in the rest of the history, so probably not a blocker.
3) We could double the number of manifests at each layer. One would only pull in the minimal kernel history, and other the full history.
4) Create a mirror an a larger git server, like github, and hope that significantly improves the download speed.
5) Download a tarball of the yocto kernel, rather than pulling in it's git tree. Yocto's git server doesn't seem to be set up to serve custom tarballs based on a requested sha. We would have to set it all up manually, and it's not remotely convenient to kernel developers.
Of these options, I'm leaning toward option 2, but look forward to hearing from the community.
Scott
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
I have posted a better fix for review ... https://review.opendev.org/c/starlingx/manifest/+/798340 Scott On 2021-06-28 10:51 a.m., Scott Little wrote:
The manifest change to implement option 2 has been merged. However a lot of folks are now having 'repo sync' issues with linux-yocto. Strangely it doesn't hit the first repo sync, but all subsequent repo sync's are affected.
If you are seeing an error from repo sync, please use this work around ...
repo init ... rm -rf .repo/project-objects/linux-yocto.git* .repo/projects/cgcs-root/stx/git/linux-yocto-* repo sync ...
I've reported the bug upsteam... https://bugs.chromium.org/p/gerrit/issues/detail?id=14700&q=tyranscooter&can=2 ... if you have a gmail account and are willing to do so, add a star to increase visibility of this strange error.
I'll also be looking for a better fix on our end.
Scott
On 2021-06-21 4:10 p.m., Scott Little wrote:
Hi all
The yocto kernel git was added to the StarlingX manifests late last week. Since then I've heard a lot of grumbling about slow 'repo sync' times. It affects folks setting up a new distro or monolithic workspace for the first time. The repo-sync time can exceed an hour as the entire history of the linux kernel is downloaded. You will also notice an additional 5.5 GB of storage consumed to hold all this history. Subsequent repo sync's should be fast.
So the question is... what if anything do we do about it?
Our options...
1) Leave it as is.
Hope that folks are mostly working in the 'flock' or 'container' layers, and NOT using monolithic builds, and so the number of folk impacted is low. Folk working at the distro layer or using monolithic can work on something else, or go to lunch, while they wait for the initial repo sync to complete.
2) Try to minimize the amount of kernel history we download through a manifest change. Limiting the git history depth does the trick ...
<!-- linux-yocto Upstream --> <project remote="linux-yocto" sync-c="true" sync-tags="false" clone-depth="100" upstream="v5.10/standard/intel-x86" revision="refs/tags/v5.10.30" name="linux-yocto" path="cgcs-root/stx/git/linux-yocto-std"/> <project remote="linux-yocto" sync-c="true" sync-tags="false" clone-depth="100" upstream="v5.10/standard/preempt-rt/intel-x86" revision="2112f10d3d0b558c9ece3ab562c41b7f6d89cff4" name="linux-yocto.git" path="cgcs-root/stx/git/linux-yocto-rt"/>
The good ...
- repo sync time drops from ~1 hr to ~5 min
- storage drops from ~5.5 GB to ~ 5GB
The bad ...
- This is fragile. It assumes that the desired rt sha can be reached from 100 commits from head of branch. However, the connection to the upstream git server drops if we ask for much more than that. e.g. depth=500 is a guaranteed fail. So upstream adds a few patches and we might start failing our repo-sync.
- The history is incomplete, this may hinder kernel developers. A 'git fetch linux-yocto' should pull in the rest of the history, so probably not a blocker.
3) We could double the number of manifests at each layer. One would only pull in the minimal kernel history, and other the full history.
4) Create a mirror an a larger git server, like github, and hope that significantly improves the download speed.
5) Download a tarball of the yocto kernel, rather than pulling in it's git tree. Yocto's git server doesn't seem to be set up to serve custom tarballs based on a requested sha. We would have to set it all up manually, and it's not remotely convenient to kernel developers.
Of these options, I'm leaning toward option 2, but look forward to hearing from the community.
Scott
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Concerns have been raised with the security implications of the git protocol. We really should return to https if at all possible. Options: 1) Use no-clone-bundle, as in repo sync -j20 --no-clone-bundle However this hurts performance and transfers a significant load onto the git servers. Slightly better would be ... repo sync -j20 || repo sync -j20 --no-clone-bundle Where the first commands is efficient, but fails on yocto, followed by the second that handles yocto. 2) Return to downloading the full history of linux-yocto. It eats an hour on the initial clone, but after that it's not too bad. 3) Set up a mirror of linux-yocto on a more powerful hosting site. git hub? 4) Switch to tarball snapshots of linux-yocto. Scott On 2021-06-28 10:54 a.m., Scott Little wrote:
I have posted a better fix for review ...
https://review.opendev.org/c/starlingx/manifest/+/798340
Scott
On 2021-06-28 10:51 a.m., Scott Little wrote:
The manifest change to implement option 2 has been merged. However a lot of folks are now having 'repo sync' issues with linux-yocto. Strangely it doesn't hit the first repo sync, but all subsequent repo sync's are affected.
If you are seeing an error from repo sync, please use this work around ...
repo init ... rm -rf .repo/project-objects/linux-yocto.git* .repo/projects/cgcs-root/stx/git/linux-yocto-* repo sync ...
I've reported the bug upsteam... https://bugs.chromium.org/p/gerrit/issues/detail?id=14700&q=tyranscooter&can=2 ... if you have a gmail account and are willing to do so, add a star to increase visibility of this strange error.
I'll also be looking for a better fix on our end.
Scott
On 2021-06-21 4:10 p.m., Scott Little wrote:
Hi all
The yocto kernel git was added to the StarlingX manifests late last week. Since then I've heard a lot of grumbling about slow 'repo sync' times. It affects folks setting up a new distro or monolithic workspace for the first time. The repo-sync time can exceed an hour as the entire history of the linux kernel is downloaded. You will also notice an additional 5.5 GB of storage consumed to hold all this history. Subsequent repo sync's should be fast.
So the question is... what if anything do we do about it?
Our options...
1) Leave it as is.
Hope that folks are mostly working in the 'flock' or 'container' layers, and NOT using monolithic builds, and so the number of folk impacted is low. Folk working at the distro layer or using monolithic can work on something else, or go to lunch, while they wait for the initial repo sync to complete.
2) Try to minimize the amount of kernel history we download through a manifest change. Limiting the git history depth does the trick ...
<!-- linux-yocto Upstream --> <project remote="linux-yocto" sync-c="true" sync-tags="false" clone-depth="100" upstream="v5.10/standard/intel-x86" revision="refs/tags/v5.10.30" name="linux-yocto" path="cgcs-root/stx/git/linux-yocto-std"/> <project remote="linux-yocto" sync-c="true" sync-tags="false" clone-depth="100" upstream="v5.10/standard/preempt-rt/intel-x86" revision="2112f10d3d0b558c9ece3ab562c41b7f6d89cff4" name="linux-yocto.git" path="cgcs-root/stx/git/linux-yocto-rt"/>
The good ...
- repo sync time drops from ~1 hr to ~5 min
- storage drops from ~5.5 GB to ~ 5GB
The bad ...
- This is fragile. It assumes that the desired rt sha can be reached from 100 commits from head of branch. However, the connection to the upstream git server drops if we ask for much more than that. e.g. depth=500 is a guaranteed fail. So upstream adds a few patches and we might start failing our repo-sync.
- The history is incomplete, this may hinder kernel developers. A 'git fetch linux-yocto' should pull in the rest of the history, so probably not a blocker.
3) We could double the number of manifests at each layer. One would only pull in the minimal kernel history, and other the full history.
4) Create a mirror an a larger git server, like github, and hope that significantly improves the download speed.
5) Download a tarball of the yocto kernel, rather than pulling in it's git tree. Yocto's git server doesn't seem to be set up to serve custom tarballs based on a requested sha. We would have to set it all up manually, and it's not remotely convenient to kernel developers.
Of these options, I'm leaning toward option 2, but look forward to hearing from the community.
Scott
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (1)
-
Scott Little