<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Some follow up data to give a sense of the diverse types of
content changes seen within the rpms of tow consecutive distro
layer builds where the only source code delta was in the kernel.</p>
<p>The crude format here is '<b><category</b>> <b><number
of rpms within the category></b>'. I add a layer of indent
when I further break down a group into sub categories.<br>
</p>
<pre>total rpms 1847</pre>
<pre> no diff 1445</pre>
<pre> diff 402</pre>
<pre> kmod 22</pre>
<pre> timestamp only 136</pre>
<pre> other 244</pre>
<pre> REQUIRES 2 (dropped requires)</pre>
<pre> add/remove files 37</pre>
<pre> .build-id 35</pre>
<pre> date in path 1</pre>
<pre> other 3</pre>
<pre> checksum 242</pre>
<pre> pyc pyo 26</pre>
<pre> debug 31</pre>
<pre> bin 62</pre>
<pre> lib 126</pre>
<pre> whl 21</pre>
<pre> object 2</pre>
<pre> gz 15</pre>
<pre> html 12</pre>
<pre> .mo 1</pre>
<pre> .c .h 4</pre>
<pre> other 15</pre>
<p> <br>
I have the lists of rpms falling into each category if that
interests folks.<br>
<br>
And here are a few examples of the categories. Format is
<rpm>:<a few lines of rpmdiff output><br>
<br>
<br>
</p>
<pre>e.g. add remove other</pre>
<pre>rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed /usr/lib/systemd/system/rdma-ndd.service</pre>
<pre>rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed /usr/lib/udev/rules.d/60-rdma-ndd.rules</pre>
<pre>rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed /usr/sbin/rdma-ndd</pre>
<pre>rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed /usr/share/man/man8/rdma-ndd.8.gz</pre>
<pre>
e.g. date in path</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/PKG-INFO</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/SOURCES.txt</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/dependency_links.txt</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/entry_points.txt</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/requires.txt</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/top_level.txt</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/zip-safe</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/PKG-INFO</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/SOURCES.txt</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/dependency_links.txt</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/entry_points.txt</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/requires.txt</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/top_level.txt</pre>
<pre>python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/zip-safe</pre>
<pre>
e.g. REQUIERS</pre>
<pre>../data/ibacm-45mlnx1-1.45101.tis.1.x86_64.rpm:removed REQUIRES libsystemd.so.0()(64bit) </pre>
<pre>../data/ibacm-45mlnx1-1.45101.tis.1.x86_64.rpm:removed REQUIRES libsystemd.so.0(LIBSYSTEMD_209)(64bit) </pre>
<pre>
e.g. pyc pyo</pre>
<pre>./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/vlog.pyc</pre>
<pre>./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/vlog.pyo</pre>
<pre>./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/winutils.pyc</pre>
<pre>./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/winutils.pyo</pre>
<pre>
e.g. debug</pre>
<pre>./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-controller-2.11.0-0.tis.1.x86_64.debug</pre>
<pre>./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-controller-vtep-2.11.0-0.tis.1.x86_64.debug</pre>
<pre>./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-nbctl-2.11.0-0.tis.1.x86_64.debug</pre>
<pre>./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-northd-2.11.0-0.tis.1.x86_64.debug</pre>
<pre>./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:S.5.......T /usr/lib/debug/usr/bin/ovn-sbctl-2.11.0-0.tis.1.x86_64.debug</pre>
<pre>./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:S.5.......T /usr/lib/debug/usr/bin/ovn-trace-2.11.0-0.tis.1.x86_64.debug</pre>
<pre>
</pre>
<pre>e.g. bin</pre>
<pre>ntpdate-4.2.6p5-29.el7.centos.tis.1.x86_64.rpm:..5.......T /usr/sbin/ntpdate</pre>
<pre>
e.g. lib </pre>
<pre>lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_access.so</pre>
<pre>lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_accesslog.so</pre>
<pre>lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_alias.so</pre>
<pre>lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_auth.so</pre>
<pre>
e.g. object</pre>
<pre>mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/xz_dec_lzma2.o</pre>
<pre>mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/xz_dec_stream.o</pre>
<pre>mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/xz_dec_syms.o</pre>
<pre>
e.g. gz</pre>
<pre>resource-agents-4.1.1-12.el7_6.7.tis.1.x86_64.rpm:S.5.......T /usr/share/man/man7/ocf_heartbeat_symlink.7.gz</pre>
<pre>resource-agents-4.1.1-12.el7_6.7.tis.1.x86_64.rpm:..5.......T /usr/share/man/man7/ocf_heartbeat_tomcat.7.gz</pre>
<pre>resource-agents-4.1.1-12.el7_6.7.tis.1.x86_64.rpm:S.5.......T /usr/share/man/man7/ocf_heartbeat_vdo-vol.7.gz</pre>
<pre>
e.g. html</pre>
<pre>grub2-common-2.02-0.76.el7.centos.tis.12.noarch.rpm:..5.......T /usr/share/doc/grub2-common-2.02/grub-dev.html</pre>
<pre>grub2-common-2.02-0.76.el7.centos.tis.12.noarch.rpm:..5.......T /usr/share/doc/grub2-common-2.02/grub.html</pre>
<pre>
e.g. .mo</pre>
<pre>libvirt-libs-4.7.0-1.tis.101.x86_64.rpm:..5.......T /usr/share/locale/af/LC_MESSAGES/libvirt.mo</pre>
<pre>libvirt-libs-4.7.0-1.tis.101.x86_64.rpm:..5.......T /usr/share/locale/am/LC_MESSAGES/libvirt.mo</pre>
<pre>libvirt-libs-4.7.0-1.tis.101.x86_64.rpm:..5.......T /usr/share/locale/anp/LC_MESSAGES/libvirt.mo</pre>
<pre>
e.g. .c .h</pre>
<pre>shim-unsigned-x64-debuginfo-15-1.el7.tis.2.noarch.rpm:..5.......T /usr/src/debug/shim-15-1.el7.tis.2/shim-15-ia32/version.c</pre>
<pre>shim-unsigned-x64-debuginfo-15-1.el7.tis.2.noarch.rpm:..5.......T /usr/src/debug/shim-15-1.el7.tis.2/shim-15-x64/version.c</pre>
<pre>
e.g. other</pre>
<pre>mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:S.5.......T /usr/src/ofa_kernel/default/compat/config.log</pre>
<pre>mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/config.status</pre>
<pre>mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/mlx_compat.ko</pre>
<pre>mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/configure.mk.kernel</pre>
<pre>mlnx-ofa_kernel-rt-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /etc/infiniband/info</pre>
<pre>grub2-efi-x64-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/grubx64.efi</pre>
<pre>shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/BOOT/fallback.efi</pre>
<pre>shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/BOOT/fbx64.efi</pre>
<pre>shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/centos/MokManager.efi</pre>
<pre>shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/centos/mmx64.efi</pre>
<pre>grub2-efi-x64-unsigned-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/gcdx64.efi.unsigned</pre>
<pre>grub2-efi-x64-unsigned-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/grubx64.efi.unsigned</pre>
<pre>shim-unsigned-x64-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/x64-15-1.el7.tis.2/shimx64.efi</pre>
<pre>shim-unsigned-x64-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/x64-15-1.el7.tis.2/shimx64.hash</pre>
<pre>python-keystoneauth1-doc-3.13.1-1.tis.0.noarch.rpm:S.5.......T /usr/share/doc/python-keystoneauth1-doc-3.13.1/html/searchindex.js</pre>
<pre>shim-ia32-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/BOOT/fbia32.efi</pre>
<pre>shim-ia32-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/centos/mmia32.efi</pre>
<pre>setup-2.8.71-10.el7.tis.16.noarch.rpm:..5.......T /etc/shadow</pre>
<pre>python-gnocchiclient-doc-7.0.4-1.tis.14.noarch.rpm:..5.......T /usr/share/doc/python-gnocchiclient-doc-7.0.4/html/searchindex.js</pre>
<pre>mlnx-ofa_kernel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /etc/infiniband/info</pre>
<pre>shim-unsigned-ia32-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/ia32-15-1.el7.tis.2/shimia32.efi</pre>
<pre>shim-unsigned-ia32-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/ia32-15-1.el7.tis.2/shimia32.hash</pre>
<pre>mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/config.log</pre>
<pre>mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/config.status</pre>
<pre>mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/mlx_compat.ko</pre>
<pre>mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/configure.mk.kernel</pre>
<pre>grub2-efi-x64-pxeboot-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /pxeboot/EFI/grubx64.efi</pre>
<pre>grub2-efi-x64-cdboot-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/gcdx64.efi</pre>
<p></p>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 2020-03-31 2:21 p.m., Scott Little
wrote:<br>
</div>
<blockquote type="cite" cite="mid:87354fc1-c64d-7b84-7934-8b5785abf13f@windriver.com">Layered
builds relies on an authoritative source (e.g. CENGN) publishing
an RPM repo with all the RPMS produced by a single build, and an
lst file enumerating those RPMS. There are new config files
describing the url's for those repos that use the latest_build
symlink. The client, a higher layer we wish to build, must
download the new lst file and repo data, and use those to download
all new/updates rpms.
<br>
<br>
Issue identified early on where.
<br>
<br>
1) The outputs of a specific build might not endure on CENGN as
long as a designers build context.
<br>
<br>
2) The latest_build symlink might change in the middle of a
download season. Packages version may have changes during between
builds. The old version specified by the pre-update lst file
might no longer be available.
<br>
<br>
So the next step is for CENGN to mirror those build outputs in a
cumulative fashion, like any other repo. It's a default for
download_mirror.sh to look for a CENGN mirror of any url it uses,
and its a default behavior for CENGN to mirror it. The mirror
would accumulate all the versions from all the builds. So
changing symlinks and deletion of old builds shouldn't impact the
end user.
<br>
<br>
All this hinged on two assumptions.
<br>
<br>
1) If a package was not upversioned between builds, there would
not be significant changes to the content of the RPM.
<br>
<br>
2) yum repodata has checksums and timestamps for all packages. I
expected yum to use that data to flag a changed rpm for
re-download.
<br>
<br>
Recent experience has proven both assumptions turned out to be
wrong.
<br>
<br>
1) Kernel modules were the first example of a violation of this
asumption. The mkod implicitly changes when the kernel changes.
The most obvious is that the installation path to the kmod change,
embedding the kernel version it was compiled against. There are
deeper differences not worth enumerating.
<br>
<br>
2) Yum will throw error over inconsistancies between new repodata
and old, our new repodata and rpms already downloaded, but will
not re-download the updated packages. I have not been able to
override this behavious with command line options, config files,
or even cleaning all yum cache. It appears to be a defense
against mirrors that are slow to update, and perhaps against
malicious hacks of an upstream repo.
<br>
<br>
The broken assumptions affected both the CENGN mirroring process,
and the final download by upper layer builders. Neither was
updating rpms that have changed if the file name did not also
change.
<br>
<br>
Is this only an important matter for kmods. Exploration with
rpmdiff reveals that the majority of RPMS change when rebuilt,
even with zero source code changes to any package. Excluding the
timestamps on files embedded within the rpms lefts a list greater
than 25% of the repo reporting differences. Spent some time
trying to figure out a way to filter out paths that embed the
build time stamp, but the list remained stubbornly large.
<br>
<br>
So how to fix it.
<br>
<br>
Several methods have been explored. Some solutions only apply if
the downloader retains the file structure of the upstream repo, or
if the download capture the entire repo. True for the cengn
mirror, not for the output of download_mirror.sh.
<br>
<br>
1) reposync: An rpm repo mirroring tool. It's built on yum, and
has the same behaviour as yum. Can't use that.
<br>
<br>
2) verifytree: This tool validates the checksums in the repodata
vs the current set of rpms. It's a tool aimed at rpm mirror
operators. It requires that the download directory structure is a
faithful replica of upstream. So I can manually download the
repodata, then validate it vs the previously downloaded set of
rpms. RPM's that it flags as having the wrong checksum need to be
re-downloaded. The actual re-download could be accomplished in
two ways.
<br>
<br>
2a) Replicate the current repo in a temporary working directory,
delete the updated rpms, use yum (either reposync or
yumdownloader) with clean cache to re-download the missing files.
Finally exchange the temporary directory with regular one. This
gives an atomic cutover to the new content. This is what CENGN
mirror is using now.
<br>
<br>
This doesn't entirely solve the problem for downloaders that
were actively updated their local snapshot when such a mirror
update goes live. Ideally the download_mirror tool has a retry
loop. If the last pass downloaded files, try again until a pass
reports there are no further files to update.
<br>
<br>
2b) curl: Use curl rather than yum. Requires that we somehow
know the URL for the rpm we intend to download. 'yumdownloader
--url' can tell use the upstream url.
<br>
<br>
3) curl: Use curl rather than yum, but do so directly without
verifytree. Rely entirely on timestamps. If we can give curl the
download url for the rpm, it can be given options that compare the
local and remote file timestamps and only downloads if the remote
copy is newer. Also it preserves the file timestamp on download.
We can use 'yumdownloader --url' to tell us which repo to use, and
what the download url is from that repo.
<br>
<br>
This was my solution for the download_mirror.sh users.
<br>
<br>
4) rsync: Requires CENGN to expose an rsync port to the world. Not
my first choice. File sharing pirates would love to compromise a
machine with an open rsync port.
<br>
<br>
<br>
New issues.
<br>
<br>
1) 'yumdownloader --url' is SLOW. It takes more than a second per
package to produce an answer. since lower layer rpms are handled
by the same tools as all other rpms, we have over 2000 packages to
check. This slowness largely persists in batch mode where I query
a large set of packages at once (3/4 sec per package). It'll take
an hour to process them all, even if the result is 'nothing to
update'
<br>
<br>
2) As noted above, most rpms are showing subtle changes between
builds even though the source has not changed. This is
undesirable, triggering a huge number of needless downloads.
<br>
<br>
<br>
New solutions?
<br>
<br>
1) Download rpms from lower layers using a different tool than
'download_mirror.sh' or at least a different sub-function than
dl-rpms. First, this mean that we only need to run 'yumdownloader
-url' vs a smaller list of rpms. Also this would allow us to use
reposync and verify tree for the download functions, removing
'yumdownloader -url' entirely.
<br>
<br>
2) Use build avoidance. Copies rpms from previous build into
current build environment and then use an incremental build. This
avoids needlessly rebuilding rpms. RPMS retain the same checksum
and timestamp if not rebuilt. Cut the problem off at the source.
Concern with this solution is that I still like to force a
non-build avoidance build from time to time to ensure we haven't
diverged from reality in some new and unexpected way.
<br>
<br>
<br>
_______________________________________________
<br>
Starlingx-discuss mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Starlingx-discuss@lists.starlingx.io">Starlingx-discuss@lists.starlingx.io</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a>
<br>
</blockquote>
</body>
</html>