[Starlingx-discuss] [ Build ] Layered build challenges
Scott Little
scott.little at windriver.com
Thu Apr 2 03:18:08 UTC 2020
Some follow up data to give a sense of the diverse types of content
changes seen within the rpms of tow consecutive distro layer builds
where the only source code delta was in the kernel.
The crude format here is '*<category*> *<number of rpms within the
category>*'. I add a layer of indent when I further break down a group
into sub categories.
total rpms 1847
no diff 1445
diff 402
kmod 22
timestamp only 136
other 244
REQUIRES 2 (dropped requires)
add/remove files 37
.build-id 35
date in path 1
other 3
checksum 242
pyc pyo 26
debug 31
bin 62
lib 126
whl 21
object 2
gz 15
html 12
.mo 1
.c .h 4
other 15
I have the lists of rpms falling into each category if that interests folks.
And here are a few examples of the categories. Format is <rpm>:<a few
lines of rpmdiff output>
e.g. add remove other
rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed /usr/lib/systemd/system/rdma-ndd.service
rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed /usr/lib/udev/rules.d/60-rdma-ndd.rules
rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed /usr/sbin/rdma-ndd
rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed /usr/share/man/man8/rdma-ndd.8.gz
e.g. date in path
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/PKG-INFO
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/SOURCES.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/dependency_links.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/entry_points.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/requires.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/top_level.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/zip-safe
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/PKG-INFO
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/SOURCES.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/dependency_links.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/entry_points.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/requires.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/top_level.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/zip-safe
e.g. REQUIERS
../data/ibacm-45mlnx1-1.45101.tis.1.x86_64.rpm:removed REQUIRES libsystemd.so.0()(64bit)
../data/ibacm-45mlnx1-1.45101.tis.1.x86_64.rpm:removed REQUIRES libsystemd.so.0(LIBSYSTEMD_209)(64bit)
e.g. pyc pyo
./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/vlog.pyc
./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/vlog.pyo
./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/winutils.pyc
./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/winutils.pyo
e.g. debug
./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-controller-2.11.0-0.tis.1.x86_64.debug
./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-controller-vtep-2.11.0-0.tis.1.x86_64.debug
./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-nbctl-2.11.0-0.tis.1.x86_64.debug
./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-northd-2.11.0-0.tis.1.x86_64.debug
./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:S.5.......T /usr/lib/debug/usr/bin/ovn-sbctl-2.11.0-0.tis.1.x86_64.debug
./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:S.5.......T /usr/lib/debug/usr/bin/ovn-trace-2.11.0-0.tis.1.x86_64.debug
e.g. bin
ntpdate-4.2.6p5-29.el7.centos.tis.1.x86_64.rpm:..5.......T /usr/sbin/ntpdate
e.g. lib
lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_access.so
lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_accesslog.so
lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_alias.so
lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_auth.so
e.g. object
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/xz_dec_lzma2.o
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/xz_dec_stream.o
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/xz_dec_syms.o
e.g. gz
resource-agents-4.1.1-12.el7_6.7.tis.1.x86_64.rpm:S.5.......T /usr/share/man/man7/ocf_heartbeat_symlink.7.gz
resource-agents-4.1.1-12.el7_6.7.tis.1.x86_64.rpm:..5.......T /usr/share/man/man7/ocf_heartbeat_tomcat.7.gz
resource-agents-4.1.1-12.el7_6.7.tis.1.x86_64.rpm:S.5.......T /usr/share/man/man7/ocf_heartbeat_vdo-vol.7.gz
e.g. html
grub2-common-2.02-0.76.el7.centos.tis.12.noarch.rpm:..5.......T /usr/share/doc/grub2-common-2.02/grub-dev.html
grub2-common-2.02-0.76.el7.centos.tis.12.noarch.rpm:..5.......T /usr/share/doc/grub2-common-2.02/grub.html
e.g. .mo
libvirt-libs-4.7.0-1.tis.101.x86_64.rpm:..5.......T /usr/share/locale/af/LC_MESSAGES/libvirt.mo
libvirt-libs-4.7.0-1.tis.101.x86_64.rpm:..5.......T /usr/share/locale/am/LC_MESSAGES/libvirt.mo
libvirt-libs-4.7.0-1.tis.101.x86_64.rpm:..5.......T /usr/share/locale/anp/LC_MESSAGES/libvirt.mo
e.g. .c .h
shim-unsigned-x64-debuginfo-15-1.el7.tis.2.noarch.rpm:..5.......T /usr/src/debug/shim-15-1.el7.tis.2/shim-15-ia32/version.c
shim-unsigned-x64-debuginfo-15-1.el7.tis.2.noarch.rpm:..5.......T /usr/src/debug/shim-15-1.el7.tis.2/shim-15-x64/version.c
e.g. other
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:S.5.......T /usr/src/ofa_kernel/default/compat/config.log
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/config.status
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/mlx_compat.ko
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/configure.mk.kernel
mlnx-ofa_kernel-rt-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /etc/infiniband/info
grub2-efi-x64-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/grubx64.efi
shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/BOOT/fallback.efi
shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/BOOT/fbx64.efi
shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/centos/MokManager.efi
shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/centos/mmx64.efi
grub2-efi-x64-unsigned-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/gcdx64.efi.unsigned
grub2-efi-x64-unsigned-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/grubx64.efi.unsigned
shim-unsigned-x64-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/x64-15-1.el7.tis.2/shimx64.efi
shim-unsigned-x64-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/x64-15-1.el7.tis.2/shimx64.hash
python-keystoneauth1-doc-3.13.1-1.tis.0.noarch.rpm:S.5.......T /usr/share/doc/python-keystoneauth1-doc-3.13.1/html/searchindex.js
shim-ia32-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/BOOT/fbia32.efi
shim-ia32-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/centos/mmia32.efi
setup-2.8.71-10.el7.tis.16.noarch.rpm:..5.......T /etc/shadow
python-gnocchiclient-doc-7.0.4-1.tis.14.noarch.rpm:..5.......T /usr/share/doc/python-gnocchiclient-doc-7.0.4/html/searchindex.js
mlnx-ofa_kernel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /etc/infiniband/info
shim-unsigned-ia32-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/ia32-15-1.el7.tis.2/shimia32.efi
shim-unsigned-ia32-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/ia32-15-1.el7.tis.2/shimia32.hash
mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/config.log
mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/config.status
mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/mlx_compat.ko
mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/configure.mk.kernel
grub2-efi-x64-pxeboot-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /pxeboot/EFI/grubx64.efi
grub2-efi-x64-cdboot-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/gcdx64.efi
On 2020-03-31 2:21 p.m., Scott Little wrote:
> Layered builds relies on an authoritative source (e.g. CENGN)
> publishing an RPM repo with all the RPMS produced by a single build,
> and an lst file enumerating those RPMS. There are new config files
> describing the url's for those repos that use the latest_build
> symlink. The client, a higher layer we wish to build, must download
> the new lst file and repo data, and use those to download all
> new/updates rpms.
>
> Issue identified early on where.
>
> 1) The outputs of a specific build might not endure on CENGN as long
> as a designers build context.
>
> 2) The latest_build symlink might change in the middle of a download
> season. Packages version may have changes during between builds. The
> old version specified by the pre-update lst file might no longer be
> available.
>
> So the next step is for CENGN to mirror those build outputs in a
> cumulative fashion, like any other repo. It's a default for
> download_mirror.sh to look for a CENGN mirror of any url it uses, and
> its a default behavior for CENGN to mirror it. The mirror would
> accumulate all the versions from all the builds. So changing symlinks
> and deletion of old builds shouldn't impact the end user.
>
> All this hinged on two assumptions.
>
> 1) If a package was not upversioned between builds, there would not be
> significant changes to the content of the RPM.
>
> 2) yum repodata has checksums and timestamps for all packages. I
> expected yum to use that data to flag a changed rpm for re-download.
>
> Recent experience has proven both assumptions turned out to be wrong.
>
> 1) Kernel modules were the first example of a violation of this
> asumption. The mkod implicitly changes when the kernel changes. The
> most obvious is that the installation path to the kmod change,
> embedding the kernel version it was compiled against. There are
> deeper differences not worth enumerating.
>
> 2) Yum will throw error over inconsistancies between new repodata and
> old, our new repodata and rpms already downloaded, but will not
> re-download the updated packages. I have not been able to override
> this behavious with command line options, config files, or even
> cleaning all yum cache. It appears to be a defense against mirrors
> that are slow to update, and perhaps against malicious hacks of an
> upstream repo.
>
> The broken assumptions affected both the CENGN mirroring process, and
> the final download by upper layer builders. Neither was updating rpms
> that have changed if the file name did not also change.
>
> Is this only an important matter for kmods. Exploration with rpmdiff
> reveals that the majority of RPMS change when rebuilt, even with zero
> source code changes to any package. Excluding the timestamps on files
> embedded within the rpms lefts a list greater than 25% of the repo
> reporting differences. Spent some time trying to figure out a way to
> filter out paths that embed the build time stamp, but the list
> remained stubbornly large.
>
> So how to fix it.
>
> Several methods have been explored. Some solutions only apply if the
> downloader retains the file structure of the upstream repo, or if the
> download capture the entire repo. True for the cengn mirror, not for
> the output of download_mirror.sh.
>
> 1) reposync: An rpm repo mirroring tool. It's built on yum, and has
> the same behaviour as yum. Can't use that.
>
> 2) verifytree: This tool validates the checksums in the repodata vs
> the current set of rpms. It's a tool aimed at rpm mirror operators.
> It requires that the download directory structure is a faithful
> replica of upstream. So I can manually download the repodata, then
> validate it vs the previously downloaded set of rpms. RPM's that it
> flags as having the wrong checksum need to be re-downloaded. The
> actual re-download could be accomplished in two ways.
>
> 2a) Replicate the current repo in a temporary working directory,
> delete the updated rpms, use yum (either reposync or yumdownloader)
> with clean cache to re-download the missing files. Finally exchange
> the temporary directory with regular one. This gives an atomic
> cutover to the new content. This is what CENGN mirror is using now.
>
> This doesn't entirely solve the problem for downloaders that were
> actively updated their local snapshot when such a mirror update goes
> live. Ideally the download_mirror tool has a retry loop. If the last
> pass downloaded files, try again until a pass reports there are no
> further files to update.
>
> 2b) curl: Use curl rather than yum. Requires that we somehow know
> the URL for the rpm we intend to download. 'yumdownloader --url' can
> tell use the upstream url.
>
> 3) curl: Use curl rather than yum, but do so directly without
> verifytree. Rely entirely on timestamps. If we can give curl the
> download url for the rpm, it can be given options that compare the
> local and remote file timestamps and only downloads if the remote copy
> is newer. Also it preserves the file timestamp on download. We can
> use 'yumdownloader --url' to tell us which repo to use, and what the
> download url is from that repo.
>
> This was my solution for the download_mirror.sh users.
>
> 4) rsync: Requires CENGN to expose an rsync port to the world. Not my
> first choice. File sharing pirates would love to compromise a machine
> with an open rsync port.
>
>
> New issues.
>
> 1) 'yumdownloader --url' is SLOW. It takes more than a second per
> package to produce an answer. since lower layer rpms are handled by
> the same tools as all other rpms, we have over 2000 packages to
> check. This slowness largely persists in batch mode where I query a
> large set of packages at once (3/4 sec per package). It'll take an
> hour to process them all, even if the result is 'nothing to update'
>
> 2) As noted above, most rpms are showing subtle changes between builds
> even though the source has not changed. This is undesirable,
> triggering a huge number of needless downloads.
>
>
> New solutions?
>
> 1) Download rpms from lower layers using a different tool than
> 'download_mirror.sh' or at least a different sub-function than
> dl-rpms. First, this mean that we only need to run 'yumdownloader
> -url' vs a smaller list of rpms. Also this would allow us to use
> reposync and verify tree for the download functions, removing
> 'yumdownloader -url' entirely.
>
> 2) Use build avoidance. Copies rpms from previous build into current
> build environment and then use an incremental build. This avoids
> needlessly rebuilding rpms. RPMS retain the same checksum and
> timestamp if not rebuilt. Cut the problem off at the source. Concern
> with this solution is that I still like to force a non-build avoidance
> build from time to time to ensure we haven't diverged from reality in
> some new and unexpected way.
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20200401/e3b5a032/attachment-0001.html>
More information about the Starlingx-discuss
mailing list