Some follow up data to give a sense of the diverse types of content changes seen within the rpms of tow consecutive distro layer builds where the only source code delta was in the kernel.
The crude format here is '<category> <number
of rpms within the category>'. I add a layer of indent
when I further break down a group into sub categories.
total rpms 1847
no diff 1445
diff 402
kmod 22
timestamp only 136
other 244
REQUIRES 2 (dropped requires)
add/remove files 37
.build-id 35
date in path 1
other 3
checksum 242
pyc pyo 26
debug 31
bin 62
lib 126
whl 21
object 2
gz 15
html 12
.mo 1
.c .h 4
other 15
I have the lists of rpms falling into each category if that
interests folks.
And here are a few examples of the categories. Format is
<rpm>:<a few lines of rpmdiff output>
e.g. add remove other
rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed /usr/lib/systemd/system/rdma-ndd.service
rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed /usr/lib/udev/rules.d/60-rdma-ndd.rules
rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed /usr/sbin/rdma-ndd
rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed /usr/share/man/man8/rdma-ndd.8.gz
e.g. date in path
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/PKG-INFO
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/SOURCES.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/dependency_links.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/entry_points.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/requires.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/top_level.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/zip-safe
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/PKG-INFO
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/SOURCES.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/dependency_links.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/entry_points.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/requires.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/top_level.txt
python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/zip-safe
e.g. REQUIERS
../data/ibacm-45mlnx1-1.45101.tis.1.x86_64.rpm:removed REQUIRES libsystemd.so.0()(64bit)
../data/ibacm-45mlnx1-1.45101.tis.1.x86_64.rpm:removed REQUIRES libsystemd.so.0(LIBSYSTEMD_209)(64bit)
e.g. pyc pyo
./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/vlog.pyc
./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/vlog.pyo
./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/winutils.pyc
./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/winutils.pyo
e.g. debug
./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-controller-2.11.0-0.tis.1.x86_64.debug
./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-controller-vtep-2.11.0-0.tis.1.x86_64.debug
./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-nbctl-2.11.0-0.tis.1.x86_64.debug
./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-northd-2.11.0-0.tis.1.x86_64.debug
./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:S.5.......T /usr/lib/debug/usr/bin/ovn-sbctl-2.11.0-0.tis.1.x86_64.debug
./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:S.5.......T /usr/lib/debug/usr/bin/ovn-trace-2.11.0-0.tis.1.x86_64.debug
e.g. bin
ntpdate-4.2.6p5-29.el7.centos.tis.1.x86_64.rpm:..5.......T /usr/sbin/ntpdate
e.g. lib
lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_access.so
lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_accesslog.so
lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_alias.so
lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_auth.so
e.g. object
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/xz_dec_lzma2.o
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/xz_dec_stream.o
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/xz_dec_syms.o
e.g. gz
resource-agents-4.1.1-12.el7_6.7.tis.1.x86_64.rpm:S.5.......T /usr/share/man/man7/ocf_heartbeat_symlink.7.gz
resource-agents-4.1.1-12.el7_6.7.tis.1.x86_64.rpm:..5.......T /usr/share/man/man7/ocf_heartbeat_tomcat.7.gz
resource-agents-4.1.1-12.el7_6.7.tis.1.x86_64.rpm:S.5.......T /usr/share/man/man7/ocf_heartbeat_vdo-vol.7.gz
e.g. html
grub2-common-2.02-0.76.el7.centos.tis.12.noarch.rpm:..5.......T /usr/share/doc/grub2-common-2.02/grub-dev.html
grub2-common-2.02-0.76.el7.centos.tis.12.noarch.rpm:..5.......T /usr/share/doc/grub2-common-2.02/grub.html
e.g. .mo
libvirt-libs-4.7.0-1.tis.101.x86_64.rpm:..5.......T /usr/share/locale/af/LC_MESSAGES/libvirt.mo
libvirt-libs-4.7.0-1.tis.101.x86_64.rpm:..5.......T /usr/share/locale/am/LC_MESSAGES/libvirt.mo
libvirt-libs-4.7.0-1.tis.101.x86_64.rpm:..5.......T /usr/share/locale/anp/LC_MESSAGES/libvirt.mo
e.g. .c .h
shim-unsigned-x64-debuginfo-15-1.el7.tis.2.noarch.rpm:..5.......T /usr/src/debug/shim-15-1.el7.tis.2/shim-15-ia32/version.c
shim-unsigned-x64-debuginfo-15-1.el7.tis.2.noarch.rpm:..5.......T /usr/src/debug/shim-15-1.el7.tis.2/shim-15-x64/version.c
e.g. other
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:S.5.......T /usr/src/ofa_kernel/default/compat/config.log
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/config.status
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/mlx_compat.ko
mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/configure.mk.kernel
mlnx-ofa_kernel-rt-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /etc/infiniband/info
grub2-efi-x64-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/grubx64.efi
shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/BOOT/fallback.efi
shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/BOOT/fbx64.efi
shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/centos/MokManager.efi
shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/centos/mmx64.efi
grub2-efi-x64-unsigned-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/gcdx64.efi.unsigned
grub2-efi-x64-unsigned-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/grubx64.efi.unsigned
shim-unsigned-x64-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/x64-15-1.el7.tis.2/shimx64.efi
shim-unsigned-x64-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/x64-15-1.el7.tis.2/shimx64.hash
python-keystoneauth1-doc-3.13.1-1.tis.0.noarch.rpm:S.5.......T /usr/share/doc/python-keystoneauth1-doc-3.13.1/html/searchindex.js
shim-ia32-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/BOOT/fbia32.efi
shim-ia32-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/centos/mmia32.efi
setup-2.8.71-10.el7.tis.16.noarch.rpm:..5.......T /etc/shadow
python-gnocchiclient-doc-7.0.4-1.tis.14.noarch.rpm:..5.......T /usr/share/doc/python-gnocchiclient-doc-7.0.4/html/searchindex.js
mlnx-ofa_kernel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /etc/infiniband/info
shim-unsigned-ia32-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/ia32-15-1.el7.tis.2/shimia32.efi
shim-unsigned-ia32-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/ia32-15-1.el7.tis.2/shimia32.hash
mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/config.log
mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/config.status
mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/mlx_compat.ko
mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/configure.mk.kernel
grub2-efi-x64-pxeboot-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /pxeboot/EFI/grubx64.efi
grub2-efi-x64-cdboot-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/gcdx64.efi
Layered builds relies on an authoritative source (e.g. CENGN) publishing an RPM repo with all the RPMS produced by a single build, and an lst file enumerating those RPMS. There are new config files describing the url's for those repos that use the latest_build symlink. The client, a higher layer we wish to build, must download the new lst file and repo data, and use those to download all new/updates rpms.
Issue identified early on where.
1) The outputs of a specific build might not endure on CENGN as long as a designers build context.
2) The latest_build symlink might change in the middle of a download season. Packages version may have changes during between builds. The old version specified by the pre-update lst file might no longer be available.
So the next step is for CENGN to mirror those build outputs in a cumulative fashion, like any other repo. It's a default for download_mirror.sh to look for a CENGN mirror of any url it uses, and its a default behavior for CENGN to mirror it. The mirror would accumulate all the versions from all the builds. So changing symlinks and deletion of old builds shouldn't impact the end user.
All this hinged on two assumptions.
1) If a package was not upversioned between builds, there would not be significant changes to the content of the RPM.
2) yum repodata has checksums and timestamps for all packages. I expected yum to use that data to flag a changed rpm for re-download.
Recent experience has proven both assumptions turned out to be wrong.
1) Kernel modules were the first example of a violation of this asumption. The mkod implicitly changes when the kernel changes. The most obvious is that the installation path to the kmod change, embedding the kernel version it was compiled against. There are deeper differences not worth enumerating.
2) Yum will throw error over inconsistancies between new repodata and old, our new repodata and rpms already downloaded, but will not re-download the updated packages. I have not been able to override this behavious with command line options, config files, or even cleaning all yum cache. It appears to be a defense against mirrors that are slow to update, and perhaps against malicious hacks of an upstream repo.
The broken assumptions affected both the CENGN mirroring process, and the final download by upper layer builders. Neither was updating rpms that have changed if the file name did not also change.
Is this only an important matter for kmods. Exploration with rpmdiff reveals that the majority of RPMS change when rebuilt, even with zero source code changes to any package. Excluding the timestamps on files embedded within the rpms lefts a list greater than 25% of the repo reporting differences. Spent some time trying to figure out a way to filter out paths that embed the build time stamp, but the list remained stubbornly large.
So how to fix it.
Several methods have been explored. Some solutions only apply if the downloader retains the file structure of the upstream repo, or if the download capture the entire repo. True for the cengn mirror, not for the output of download_mirror.sh.
1) reposync: An rpm repo mirroring tool. It's built on yum, and has the same behaviour as yum. Can't use that.
2) verifytree: This tool validates the checksums in the repodata vs the current set of rpms. It's a tool aimed at rpm mirror operators. It requires that the download directory structure is a faithful replica of upstream. So I can manually download the repodata, then validate it vs the previously downloaded set of rpms. RPM's that it flags as having the wrong checksum need to be re-downloaded. The actual re-download could be accomplished in two ways.
2a) Replicate the current repo in a temporary working directory, delete the updated rpms, use yum (either reposync or yumdownloader) with clean cache to re-download the missing files. Finally exchange the temporary directory with regular one. This gives an atomic cutover to the new content. This is what CENGN mirror is using now.
This doesn't entirely solve the problem for downloaders that were actively updated their local snapshot when such a mirror update goes live. Ideally the download_mirror tool has a retry loop. If the last pass downloaded files, try again until a pass reports there are no further files to update.
2b) curl: Use curl rather than yum. Requires that we somehow know the URL for the rpm we intend to download. 'yumdownloader --url' can tell use the upstream url.
3) curl: Use curl rather than yum, but do so directly without verifytree. Rely entirely on timestamps. If we can give curl the download url for the rpm, it can be given options that compare the local and remote file timestamps and only downloads if the remote copy is newer. Also it preserves the file timestamp on download. We can use 'yumdownloader --url' to tell us which repo to use, and what the download url is from that repo.
This was my solution for the download_mirror.sh users.
4) rsync: Requires CENGN to expose an rsync port to the world. Not my first choice. File sharing pirates would love to compromise a machine with an open rsync port.
New issues.
1) 'yumdownloader --url' is SLOW. It takes more than a second per package to produce an answer. since lower layer rpms are handled by the same tools as all other rpms, we have over 2000 packages to check. This slowness largely persists in batch mode where I query a large set of packages at once (3/4 sec per package). It'll take an hour to process them all, even if the result is 'nothing to update'
2) As noted above, most rpms are showing subtle changes between builds even though the source has not changed. This is undesirable, triggering a huge number of needless downloads.
New solutions?
1) Download rpms from lower layers using a different tool than 'download_mirror.sh' or at least a different sub-function than dl-rpms. First, this mean that we only need to run 'yumdownloader -url' vs a smaller list of rpms. Also this would allow us to use reposync and verify tree for the download functions, removing 'yumdownloader -url' entirely.
2) Use build avoidance. Copies rpms from previous build into current build environment and then use an incremental build. This avoids needlessly rebuilding rpms. RPMS retain the same checksum and timestamp if not rebuilt. Cut the problem off at the source. Concern with this solution is that I still like to force a non-build avoidance build from time to time to ensure we haven't diverged from reality in some new and unexpected way.
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss