[Starlingx-discuss] [ Build ] Layered build challenges

Scott Little scott.little at windriver.com
Thu Apr 2 03:18:08 UTC 2020


Some follow up data to give a sense of the diverse types of content 
changes seen within the rpms of tow consecutive distro layer builds 
where the only source code delta was in the kernel.

The crude format here is '*<category*> *<number of rpms within the 
category>*'. I add a layer of indent when I further break down a group 
into sub categories.

total rpms 1847

    no diff 1445

    diff     402

        kmod            22

        timestamp only 136

        other          244

            REQUIRES           2    (dropped requires)

            add/remove files  37

                .build-id     35

                date in path   1

                other          3

            checksum     242

                pyc pyo   26

                debug     31

                bin       62

                lib      126

                whl       21

                object     2

                gz        15

                html      12

                .mo        1

                .c .h      4

                other     15


I have the lists of rpms falling into each category if that interests folks.

And here are a few examples of the categories.  Format is <rpm>:<a few 
lines of rpmdiff output>


e.g. add remove other

rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed     /usr/lib/systemd/system/rdma-ndd.service

rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed     /usr/lib/udev/rules.d/60-rdma-ndd.rules

rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed     /usr/sbin/rdma-ndd

rdma-core-45mlnx1-1.45101.tis.1.x86_64.rpm:removed     /usr/share/man/man8/rdma-ndd.8.gz

e.g. date in path

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added       /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added       /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/PKG-INFO

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added       /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/SOURCES.txt

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added       /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/dependency_links.txt

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added       /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/entry_points.txt

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added       /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/requires.txt

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added       /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/top_level.txt

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:added       /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200320-py2.7.egg-info/zip-safe

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed     /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed     /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/PKG-INFO

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed     /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/SOURCES.txt

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed     /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/dependency_links.txt

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed     /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/entry_points.txt

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed     /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/requires.txt

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed     /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/top_level.txt

python2-setuptools-38.5.1-1.el7.tis.0.noarch.rpm:removed     /usr/lib/python2.7/site-packages/setuptools-38.5.1.post20200321-py2.7.egg-info/zip-safe

e.g. REQUIERS

../data/ibacm-45mlnx1-1.45101.tis.1.x86_64.rpm:removed     REQUIRES libsystemd.so.0()(64bit)

../data/ibacm-45mlnx1-1.45101.tis.1.x86_64.rpm:removed     REQUIRES libsystemd.so.0(LIBSYSTEMD_209)(64bit)

e.g. pyc pyo

./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/vlog.pyc

./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/vlog.pyo

./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/winutils.pyc

./python2-openvswitch-2.11.0-0.tis.1.noarch.rpm:..5.......T /usr/lib/python2.7/site-packages/ovs/winutils.pyo

e.g. debug

./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-controller-2.11.0-0.tis.1.x86_64.debug

./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-controller-vtep-2.11.0-0.tis.1.x86_64.debug

./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-nbctl-2.11.0-0.tis.1.x86_64.debug

./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:..5.......T /usr/lib/debug/usr/bin/ovn-northd-2.11.0-0.tis.1.x86_64.debug

./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:S.5.......T /usr/lib/debug/usr/bin/ovn-sbctl-2.11.0-0.tis.1.x86_64.debug

./openvswitch-debuginfo-2.11.0-0.tis.1.x86_64.rpm:S.5.......T /usr/lib/debug/usr/bin/ovn-trace-2.11.0-0.tis.1.x86_64.debug

e.g. bin

ntpdate-4.2.6p5-29.el7.centos.tis.1.x86_64.rpm:..5.......T /usr/sbin/ntpdate

e.g. lib

lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_access.so

lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_accesslog.so

lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_alias.so

lighttpd-1.4.54-1.el7.tis.7.x86_64.rpm:..5.......T /usr/lib64/lighttpd/mod_auth.so

e.g. object

mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/xz_dec_lzma2.o

mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/xz_dec_stream.o

mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/xz_dec_syms.o

e.g. gz

resource-agents-4.1.1-12.el7_6.7.tis.1.x86_64.rpm:S.5.......T /usr/share/man/man7/ocf_heartbeat_symlink.7.gz

resource-agents-4.1.1-12.el7_6.7.tis.1.x86_64.rpm:..5.......T /usr/share/man/man7/ocf_heartbeat_tomcat.7.gz

resource-agents-4.1.1-12.el7_6.7.tis.1.x86_64.rpm:S.5.......T /usr/share/man/man7/ocf_heartbeat_vdo-vol.7.gz

e.g. html

grub2-common-2.02-0.76.el7.centos.tis.12.noarch.rpm:..5.......T /usr/share/doc/grub2-common-2.02/grub-dev.html

grub2-common-2.02-0.76.el7.centos.tis.12.noarch.rpm:..5.......T /usr/share/doc/grub2-common-2.02/grub.html

e.g. .mo

libvirt-libs-4.7.0-1.tis.101.x86_64.rpm:..5.......T /usr/share/locale/af/LC_MESSAGES/libvirt.mo

libvirt-libs-4.7.0-1.tis.101.x86_64.rpm:..5.......T /usr/share/locale/am/LC_MESSAGES/libvirt.mo

libvirt-libs-4.7.0-1.tis.101.x86_64.rpm:..5.......T /usr/share/locale/anp/LC_MESSAGES/libvirt.mo

e.g. .c .h

shim-unsigned-x64-debuginfo-15-1.el7.tis.2.noarch.rpm:..5.......T /usr/src/debug/shim-15-1.el7.tis.2/shim-15-ia32/version.c

shim-unsigned-x64-debuginfo-15-1.el7.tis.2.noarch.rpm:..5.......T /usr/src/debug/shim-15-1.el7.tis.2/shim-15-x64/version.c

e.g. other

mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:S.5.......T /usr/src/ofa_kernel/default/compat/config.log

mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/config.status

mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/mlx_compat.ko

mlnx-ofa_kernel-rt-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/configure.mk.kernel

mlnx-ofa_kernel-rt-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /etc/infiniband/info

grub2-efi-x64-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/grubx64.efi

shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/BOOT/fallback.efi

shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/BOOT/fbx64.efi

shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/centos/MokManager.efi

shim-x64-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/centos/mmx64.efi

grub2-efi-x64-unsigned-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/gcdx64.efi.unsigned

grub2-efi-x64-unsigned-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/grubx64.efi.unsigned

shim-unsigned-x64-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/x64-15-1.el7.tis.2/shimx64.efi

shim-unsigned-x64-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/x64-15-1.el7.tis.2/shimx64.hash

python-keystoneauth1-doc-3.13.1-1.tis.0.noarch.rpm:S.5.......T /usr/share/doc/python-keystoneauth1-doc-3.13.1/html/searchindex.js

shim-ia32-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/BOOT/fbia32.efi

shim-ia32-15-1.tis.2.x86_64.rpm:..5.......T /boot/efi/EFI/centos/mmia32.efi

setup-2.8.71-10.el7.tis.16.noarch.rpm:..5.......T /etc/shadow

python-gnocchiclient-doc-7.0.4-1.tis.14.noarch.rpm:..5.......T /usr/share/doc/python-gnocchiclient-doc-7.0.4/html/searchindex.js

mlnx-ofa_kernel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /etc/infiniband/info

shim-unsigned-ia32-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/ia32-15-1.el7.tis.2/shimia32.efi

shim-unsigned-ia32-15-1.el7.tis.2.x86_64.rpm:..5.......T /usr/share/shim/ia32-15-1.el7.tis.2/shimia32.hash

mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/config.log

mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/config.status

mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/compat/mlx_compat.ko

mlnx-ofa_kernel-devel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.tis.2.x86_64.rpm:..5.......T /usr/src/ofa_kernel/default/configure.mk.kernel

grub2-efi-x64-pxeboot-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /pxeboot/EFI/grubx64.efi

grub2-efi-x64-cdboot-2.02-0.76.el7.centos.tis.12.x86_64.rpm:..5.......T /boot/efi/EFI/centos/gcdx64.efi



On 2020-03-31 2:21 p.m., Scott Little wrote:
> Layered builds relies on an authoritative source (e.g. CENGN) 
> publishing an RPM repo with all the RPMS produced by a single build, 
> and an lst file enumerating those RPMS.  There are new config files 
> describing the url's for those repos that use the latest_build 
> symlink.  The client, a higher layer we wish to build, must download 
> the new lst file and repo data, and use those to download all 
> new/updates rpms.
>
> Issue identified early on where.
>
> 1) The outputs of a specific build might not endure on CENGN as long 
> as a designers build context.
>
> 2) The latest_build symlink might change in the middle of a download 
> season.  Packages version may have changes during between builds.  The 
> old version specified by the pre-update lst file might no longer be 
> available.
>
> So the next step is for CENGN to mirror those build outputs in a 
> cumulative fashion, like any other repo.  It's a default for 
> download_mirror.sh to look for a CENGN mirror of any url it uses, and 
> its a default behavior for CENGN to mirror it.  The mirror would 
> accumulate all the versions from all the builds.  So changing symlinks 
> and deletion of old builds shouldn't impact the end user.
>
> All this hinged on two assumptions.
>
> 1) If a package was not upversioned between builds, there would not be 
> significant changes to the content of the RPM.
>
> 2) yum repodata has checksums and timestamps for all packages.  I 
> expected yum to use that data to flag a changed rpm for re-download.
>
> Recent experience has proven both assumptions turned out to be wrong.
>
> 1) Kernel modules were the first example of a violation of this 
> asumption.  The mkod implicitly changes when the kernel changes. The 
> most obvious is that the installation path to the kmod change, 
> embedding the kernel version it was compiled against.  There are 
> deeper differences not worth enumerating.
>
> 2) Yum will throw error over inconsistancies between new repodata and 
> old, our new repodata and rpms already downloaded, but will not 
> re-download the updated packages.  I have not been able to override 
> this behavious with command line options, config files, or even 
> cleaning all yum cache.  It appears to be a defense against mirrors 
> that are slow to update, and perhaps against malicious hacks of an 
> upstream repo.
>
> The broken assumptions affected both the CENGN mirroring process, and 
> the final download by upper layer builders.  Neither was updating rpms 
> that have changed if the file name did not also change.
>
> Is this only an important matter for kmods.  Exploration with rpmdiff 
> reveals that the majority of RPMS change when rebuilt, even with zero 
> source code changes to any package.  Excluding the timestamps on files 
> embedded within the rpms lefts a list greater than 25% of the repo 
> reporting differences.  Spent some time trying to figure out a way to 
> filter out paths that embed the build time stamp, but the list 
> remained stubbornly large.
>
> So how to fix it.
>
> Several methods have been explored.  Some solutions only apply if the 
> downloader retains the file structure of the upstream repo, or if the 
> download capture the entire repo.   True for the cengn mirror, not for 
> the output of download_mirror.sh.
>
> 1) reposync: An rpm repo mirroring tool.  It's built on yum, and has 
> the same behaviour as yum.  Can't use that.
>
> 2) verifytree:  This tool validates the checksums in the repodata vs 
> the current set of rpms.  It's a tool aimed at rpm mirror operators. 
> It requires that the download directory structure is a faithful 
> replica of upstream.  So I can manually download the repodata, then 
> validate it vs the previously downloaded set of rpms.  RPM's that it 
> flags as having the wrong checksum need to be re-downloaded.  The 
> actual re-download could be accomplished in two ways.
>
> 2a) Replicate the current repo in a temporary working directory, 
> delete the updated rpms, use yum (either reposync or yumdownloader) 
> with clean cache to re-download the missing files. Finally exchange 
> the temporary directory with regular one.  This gives an atomic 
> cutover to the new content.  This is what CENGN mirror is using now.
>
>   This doesn't entirely solve the problem for downloaders that were 
> actively updated their local snapshot when such a mirror update goes 
> live.  Ideally the download_mirror tool has a retry loop.  If the last 
> pass downloaded files, try again until a pass reports there are no 
> further files to update.
>
> 2b) curl:  Use curl rather than yum.  Requires that we somehow know 
> the URL for the rpm we intend to download.  'yumdownloader --url' can 
> tell use the upstream url.
>
> 3) curl:  Use curl rather than yum, but do so directly without 
> verifytree.  Rely entirely on timestamps.  If we can give curl the 
> download url for the rpm, it can be given options that compare the 
> local and remote file timestamps and only downloads if the remote copy 
> is newer.  Also it preserves the file timestamp on download. We can 
> use 'yumdownloader --url' to tell us which repo to use, and what the 
> download url is from that repo.
>
> This was my solution for the download_mirror.sh users.
>
> 4) rsync: Requires CENGN to expose an rsync port to the world. Not my 
> first choice.  File sharing pirates would love to compromise a machine 
> with an open rsync port.
>
>
> New issues.
>
> 1) 'yumdownloader --url' is SLOW.  It takes more than a second per 
> package to produce an answer.  since lower layer rpms are handled by 
> the same tools as all other rpms, we have over 2000 packages to 
> check.  This slowness largely persists in batch mode where I query a 
> large set of packages at once (3/4 sec per package).  It'll take an 
> hour to process them all, even if the result is 'nothing to update'
>
> 2) As noted above, most rpms are showing subtle changes between builds 
> even though the source has not changed.  This is undesirable, 
> triggering a huge number of needless downloads.
>
>
> New solutions?
>
> 1) Download rpms from lower layers using a different tool than 
> 'download_mirror.sh' or at least a different sub-function than 
> dl-rpms.  First, this mean that we only need to run 'yumdownloader 
> -url' vs a smaller list of rpms.  Also this would allow us to use 
> reposync and verify tree for the download functions, removing 
> 'yumdownloader -url' entirely.
>
> 2) Use build avoidance.  Copies rpms from previous build into current 
> build environment and then use an incremental build.  This avoids 
> needlessly rebuilding rpms. RPMS retain the same checksum and 
> timestamp if not rebuilt.  Cut the problem off at the source. Concern 
> with this solution is that I still like to force a non-build avoidance 
> build from time to time to ensure we haven't diverged from reality in 
> some new and unexpected way.
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20200401/e3b5a032/attachment-0001.html>


More information about the Starlingx-discuss mailing list