[Starlingx-discuss] [ Build ] Layered build challenges

[Scott Little]

Layered builds relies on an authoritative source (e.g. CENGN) publishing 
an RPM repo with all the RPMS produced by a single build, and an lst 
file enumerating those RPMS.  There are new config files describing the 
url's for those repos that use the latest_build symlink.  The client, a 
higher layer we wish to build, must download the new lst file and repo 
data, and use those to download all new/updates rpms.

Issue identified early on where.

1) The outputs of a specific build might not endure on CENGN as long as 
a designers build context.

2) The latest_build symlink might change in the middle of a download 
season.  Packages version may have changes during between builds.  The 
old version specified by the pre-update lst file might no longer be 
available.

So the next step is for CENGN to mirror those build outputs in a 
cumulative fashion, like any other repo.  It's a default for 
download_mirror.sh to look for a CENGN mirror of any url it uses, and 
its a default behavior for CENGN to mirror it.  The mirror would 
accumulate all the versions from all the builds.  So changing symlinks 
and deletion of old builds shouldn't impact the end user.

All this hinged on two assumptions.

1) If a package was not upversioned between builds, there would not be 
significant changes to the content of the RPM.

2) yum repodata has checksums and timestamps for all packages.  I 
expected yum to use that data to flag a changed rpm for re-download.

Recent experience has proven both assumptions turned out to be wrong.

1) Kernel modules were the first example of a violation of this 
asumption.  The mkod implicitly changes when the kernel changes. The 
most obvious is that the installation path to the kmod change, embedding 
the kernel version it was compiled against.  There are deeper 
differences not worth enumerating.

2) Yum will throw error over inconsistancies between new repodata and 
old, our new repodata and rpms already downloaded, but will not 
re-download the updated packages.  I have not been able to override this 
behavious with command line options, config files, or even cleaning all 
yum cache.  It appears to be a defense against mirrors that are slow to 
update, and perhaps against malicious hacks of an upstream repo.

The broken assumptions affected both the CENGN mirroring process, and 
the final download by upper layer builders.  Neither was updating rpms 
that have changed if the file name did not also change.

Is this only an important matter for kmods.  Exploration with rpmdiff 
reveals that the majority of RPMS change when rebuilt, even with zero 
source code changes to any package.  Excluding the timestamps on files 
embedded within the rpms lefts a list greater than 25% of the repo 
reporting differences.  Spent some time trying to figure out a way to 
filter out paths that embed the build time stamp, but the list remained 
stubbornly large.

So how to fix it.

Several methods have been explored.  Some solutions only apply if the 
downloader retains the file structure of the upstream repo, or if the 
download capture the entire repo.   True for the cengn mirror, not for 
the output of download_mirror.sh.

1) reposync: An rpm repo mirroring tool.  It's built on yum, and has the 
same behaviour as yum.  Can't use that.

2) verifytree:  This tool validates the checksums in the repodata vs the 
current set of rpms.  It's a tool aimed at rpm mirror operators. It 
requires that the download directory structure is a faithful replica of 
upstream.  So I can manually download the repodata, then validate it vs 
the previously downloaded set of rpms.  RPM's that it flags as having 
the wrong checksum need to be re-downloaded.  The actual re-download 
could be accomplished in two ways.

2a) Replicate the current repo in a temporary working directory, delete 
the updated rpms, use yum (either reposync or yumdownloader) with clean 
cache to re-download the missing files. Finally exchange the temporary 
directory with regular one.  This gives an atomic cutover to the new 
content.  This is what CENGN mirror is using now.

   This doesn't entirely solve the problem for downloaders that were 
actively updated their local snapshot when such a mirror update goes 
live.  Ideally the download_mirror tool has a retry loop.  If the last 
pass downloaded files, try again until a pass reports there are no 
further files to update.

2b) curl:  Use curl rather than yum.  Requires that we somehow know the 
URL for the rpm we intend to download.  'yumdownloader --url' can tell 
use the upstream url.

3) curl:  Use curl rather than yum, but do so directly without 
verifytree.  Rely entirely on timestamps.  If we can give curl the 
download url for the rpm, it can be given options that compare the local 
and remote file timestamps and only downloads if the remote copy is 
newer.  Also it preserves the file timestamp on download. We can use 
'yumdownloader --url' to tell us which repo to use, and what the 
download url is from that repo.

This was my solution for the download_mirror.sh users.

4) rsync: Requires CENGN to expose an rsync port to the world. Not my 
first choice.  File sharing pirates would love to compromise a machine 
with an open rsync port.


New issues.

1) 'yumdownloader --url' is SLOW.  It takes more than a second per 
package to produce an answer.  since lower layer rpms are handled by the 
same tools as all other rpms, we have over 2000 packages to check.  This 
slowness largely persists in batch mode where I query a large set of 
packages at once (3/4 sec per package).  It'll take an hour to process 
them all, even if the result is 'nothing to update'

2) As noted above, most rpms are showing subtle changes between builds 
even though the source has not changed.  This is undesirable, triggering 
a huge number of needless downloads.


New solutions?

1) Download rpms from lower layers using a different tool than 
'download_mirror.sh' or at least a different sub-function than dl-rpms.  
First, this mean that we only need to run 'yumdownloader -url' vs a 
smaller list of rpms.  Also this would allow us to use reposync and 
verify tree for the download functions, removing 'yumdownloader -url' 
entirely.

2) Use build avoidance.  Copies rpms from previous build into current 
build environment and then use an incremental build.  This avoids 
needlessly rebuilding rpms. RPMS retain the same checksum and timestamp 
if not rebuilt.  Cut the problem off at the source. Concern with this 
solution is that I still like to force a non-build avoidance build from 
time to time to ensure we haven't diverged from reality in some new and 
unexpected way.