[Starlingx-discuss] build-pkgs cannot complete std build

Thu Sep 27 16:10:59 UTC 2018

On 18-09-26 08:05 PM, Cordoba Malibran, Erich wrote:
> On Wed, 2018-09-26 at 16:44 -0700, Saul Wold wrote:
>> On 09/26/2018 02:08 PM, Scott Little wrote:
>>> I have also been investigating another intermittent build error
>>> affecting initscripts.
>>>
>>> https://bugs.launchpad.net/starlingx/+bug/1794611
>>>
>>> So any given package might be built by rpm-4.11.3, or by rpm-
>>> 4.14.0.
>>> It's a race with many inputs.  Ideally it shouldn't matter, but it
>>> does.  build-pkgs --serial might reduce the odds, but likely won't
>>> solve
>>> it entirely.
>>>
>> Why is this the case in the first place, can't we ensure we only
>> build
>> rpm-4.14.0?
>>
>>> In this case it is a new option that rpm passes to the patch
>>> command,
>>> --no-backup-if-mismatch, first introduced by rpm-4.13.
>>> This option has the effect of suppressing the creation of '.orig'
>>> files.
>>>
>>> Creation of .orig files are a consequence of applying patches that
>>> are
>>> not clean. Unclean patches are ones that require 'fuzzing', i.e.
>>> treat
>>> the patch line numbers as approximate, rather than a strict
>>> requirement,
>>> just so long as the before/after context seems to be correct.
>>>
>>> Prior to StarlingX, my policy for rebasing patches was that no fuzz
>>> is
>>> tolerated in our patches.
>>>
>>> All the work to upgrade to 7.5 has created a lot of fuzzy patches.
>>>
>> Wow, sorry I was not aware that they were all fuzzy patches.
>>
> How did you got this list? There were .orig files in those folder?
My query was ...

for f in $(grep -l 'Hunk' $MY_WORKSPACE/std/results/*/*/build.log); do 
basename $(dirname $f); done

As noted in another thread, we might get false positives in the form of 
CentOS/EL7 patches that are fuzzy, not our own.

>
>>> audit-2.8.1-3.el7.tis.2
>>> bash-4.2.46-30.el7.tis.3
>>> dhcp-4.2.5-68.el7.centos.1.tis.8
>>> dnsmasq-2.76-5.el7.tis.6
>>> drbd-8.4.3-0.tis.6
>>> facter-2.4.4-4.el7.tis.4
>>> haproxy-1.5.18-7.el7.tis.7
>>> initscripts-9.49.41-1.el7.tis.16
>>> iptables-1.4.21-24.1.el7_5.tis.3
>>> kubernetes-1.10.0-1.tis.1
>>> libevent-2.0.21-4.el7.tis.2
>>> lighttpd-1.4.50-1.el7.tis.6
>>> logrotate-3.8.6-15.el7.tis.3
>>> netpbm-10.79.00-7.el7.tis.2
>>> net-snmp-5.7.2-33.el7_5.2.tis.10
>>> net-tools-2.0-0.22.20131004git.el7.tis.2
>>> nfs-utils-1.3.0-0.54.el7.tis.4
>>> nss-pam-ldapd-0.8.13-16.el7.tis.4
>>> ntp-4.2.6p5-28.el7.centos.tis.3
>>> openldap-2.4.44-15.el7_5.tis.8
>>> openssh-7.4p1-16.el7_4.tis.9
>>> pam-1.1.8-22.el7.tis.4
>>> puppet-4.8.2-1.el7.tis.2
>>> puppet-ceph-2.4.1-1.el7.tis.4
>>> puppet-horizon-11.5.0-1.el7.tis.1
>>> python-2.7.5-69.el7_5.tis.3
>>> python-keyring-5.7.1-1.tis.2
>>> python-wsme-0.9.2-1.el7.tis.3
>>> resource-agents-3.9.5-124.el7.tis.12
>>> rsync-3.1.2-4.el7.tis.2
>>> shadow-utils-4.1.5.1-24.el7.tis.4
>>> sudo-1.8.19p2-14.el7_5.tis.3
>>> watchdog-5.13-11.el7.tis.2
>>>
>>> So any of these packages might or might not produce unwanted .orig
>>> files.
>>> The .orig files might or might not be packaged, or break packaging.
>>>
>>> So there are a number of threads to pull at here.
>>> 1) Are fuzzy patches tolerated.   I vote no.  Sooner or later a
>>> fuzzy
>>> patch will be mis-applied and cause us problems.
>> Agreed, when updating patches should be rebased and de-fuzzed.
>>
>>> 2) Can we force rpmbuild within mock to use a consistent policy
>>> with
>>> respect to creation of orig files?
>> It would be good to investigate that.
>>
> Seems we can pass _default_patch_fuzz 0 as a variable to rpmbuild.
>
> What could be the course of action here?
> - Wire rpmbuild to not allow fuzzy patches.
> - Then iteratively try to build until all the index are correct
>
>
> -Erich
I usually start by using a combination of 'build-pkgs --edit', 'git 
format-patch', and meld (a graphical diff/merge tool).
e.g.

# If 'build-pkgs --edit' works, it's rather fragile as it's more or less 
parsing spec files, it produces
# two gits, on for working on the tarball content, and a second for the 
spec file and other files outside the tarball.
# Patches are seen as commits, applied with fuzz allowed.  STX patches 
will get a WRS prefix  (sorry, I'll make that STX in future)
build-pkgs --edit --clean initscripts
build-pkgs --edit initscripts
...
   11:36:45 b7: ===== 'initscripts' has been extracted for editing. =====
   11:36:45 b7: ===== Metadata can be found at: 
/localdisk/loadbuild/slittle1/starlingx-4/std/srpm_work/initscripts/rpmbuild
   11:36:45 b7: ===== Source code can be found at: 
/localdisk/loadbuild/slittle1/starlingx-4/std/srpm_work/initscripts/gits/initscripts.spec/initscripts-9.49.41

cd 
/localdisk/loadbuild/slittle1/starlingx-4/std/srpm_work/initscripts/gits/initscripts.spec/initscripts-9.49.41
git log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s 
%Cgreen(%ci) %C(bold blue)<%an>%Creset' --abbrev-commit
   * 3aaf0ca - (HEAD, initscripts-9.49.41-1.el7) WRS: Patch15: 
run-dhclient-as-daemon-for-ipv6.patch (2018-09-27 11:36:45 -0400) <Scott 
Little>
   * 126e810 - WRS: Patch14: ifup-eth-stop-waiting-if-link-is-up.patch 
(2018-09-27 11:36:44 -0400) <Scott Little>
   * df98ada - WRS: Patch13: ipv6-static-route-support.patch (2018-09-27 
11:36:44 -0400) <Scott Little>
   * cc391dd - WRS: Patch12: 
sysconfig-unsafe-usage-of-linkdelay-variable.patch (2018-09-27 11:36:44 
-0400) <Scott Little>
   * 398cdf6 - WRS: Patch11: 
sysconfig-affirmative-check-for-link-carrier.patch (2018-09-27 11:36:43 
-0400) <Scott Little>
   * a3ebe9e - WRS: Patch10: run-ifdown-on-all-interfaces.patch 
(2018-09-27 11:36:43 -0400) <Scott Little>
   * 0a6e1fc - WRS: Patch9: 0001-force-delay-check-link-down.patch 
(2018-09-27 11:36:43 -0400) <Scott Little>
   * 45075af - WRS: Patch8: 0001-dhclient-remove-1-arg.patch (2018-09-27 
11:36:42 -0400) <Scott Little>
   * 02f8215 - WRS: Patch7: support-interface-promisc.patch (2018-09-27 
11:36:42 -0400) <Scott Little>
   * c5b8eab - WRS: Patch6: 
dhclient-restrict-interfaces-to-those-on-c.patch (2018-09-27 11:36:41 
-0400) <Scott Little>
   * 6d8bcb5 - WRS: Patch5: relocate-dhclient-leases-to-var-run.patch 
(2018-09-27 11:36:41 -0400) <Scott Little>
   * 480e9a5 - WRS: Patch4: support-interface-scriptlets.patch 
(2018-09-27 11:36:41 -0400) <Scott Little>
   * 487fdfa - (tag: pre_wrs_initscripts-9.49.41-1.el7) ORIGINAL: 
extracted archive (2018-09-27 11:36:39 -0400) <Scott Little>
   * 7ca8368 - (master) ORIGINAL: initial commit (2018-09-27 11:36:39 
-0400) <Scott Little>

# Create new patches without fuzz.  Note: patch comment text is often 
discarded
# git format-patch <SHA_prior_to_first_WRS_patch>
git format-patch $(git tag | grep pre_wrs_)

# use meld or other diff/merge tool to transfer the revised line 
numbers.  A bit of coding magic is required to translate patch names.
cd $MY_REPO/stx/stx-integ/base/initscripts/centos/patches/
for src in $(ls -1 
/localdisk/loadbuild/slittle1/starlingx-4/std/srpm_work/initscripts/gits/initscripts.spec/initscripts-9.49.41/00*patch); 
do dest=$(ls -1 $(basename $src | cut -d '-' -f 4- | sed 
's#.patch$#*#')); meld $src $dest; done

cd 
/localdisk/loadbuild/slittle1/starlingx-4/std/srpm_work/initscripts/rpmbuild
git log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s 
%Cgreen(%ci) %C(bold blue)<%an>%Creset' --abbrev-commit
   * 4ecfd77 - (HEAD, initscripts-9.49.41-1.el7) WRS: 
spec-run-dhclient-as-daemon-for-ipv6.patch (2018-09-27 11:36:40 -0400) 
<Scott Little>
   * 69d0bb4 - WRS: spec-ifup-eth-stop-waiting-if-link-is-up.patch 
(2018-09-27 11:36:40 -0400) <Scott Little>
   * e2942ca - WRS: ipv6-static-route-support.patch (2018-09-27 11:36:40 
-0400) <Scott Little>
   * 4e0368a - WRS: fix-build-failures-due-to-unwanted-sgid.patch 
(2018-09-27 11:36:40 -0400) <Scott Little>
   * bc43676 - WRS: 
spec-sysconfig-unsafe-usage-of-linkdelay-variable.patch (2018-09-27 
11:36:40 -0400) <Scott Little>
   * 6a38a1d - WRS: 
spec-sysconfig-affirmative-check-for-link-carrier.patch (2018-09-27 
11:36:40 -0400) <Scott Little>
   * 730fe3c - WRS: 0001-Update-package-versioning-for-TIS-format.patch 
(2018-09-27 11:36:39 -0400) <Scott Little>
   * a083a54 - WRS: run-ifdown-on-all-interfaces.patch (2018-09-27 
11:36:39 -0400) <Scott Little>
   * 7f02bc2 - WRS: force-delay-check-link-down.patch (2018-09-27 
11:36:39 -0400) <Scott Little>
   * dbdb232 - WRS: 
0001-Fix-Interfaces-intermittenly-do-not-come-up-on-boot.patch 
(2018-09-27 11:36:39 -0400) <Scott Little>
   * c51c427 - WRS: spec-add-mountnfs-init-script.patch (2018-09-27 
11:36:39 -0400) <Scott Little>
   * 0a5a833 - WRS: 0001-Support-PROMISC-for-interfaces-config.patch 
(2018-09-27 11:36:39 -0400) <Scott Little>
   * e0bbac3 - WRS: 0001-Disable-zeroconf-route.patch (2018-09-27 
11:36:39 -0400) <Scott Little>
   * 6485d13 - WRS: stop-creating-shared-dirs.patch (2018-09-27 11:36:39 
-0400) <Scott Little>
   * 7223930 - WRS: spec-include-TiS-changes.patch (2018-09-27 11:36:39 
-0400) <Scott Little>
   * b63eb3d - (tag: wrs_post_copy_list_initscripts-9.49.41-1.el7) WRS: 
COPY_LIST content (2018-09-27 11:36:39 -0400) <Scott Little>
   * 8e1e97a - (tag: pre_wrs_initscripts-9.49.41-1.el7) ORIGINAL: srpm 
extract (2018-09-27 11:36:38 -0400) <Scott Little>
   * a6928fd - (master) ORIGINAL: initial commit (2018-09-27 11:36:38 
-0400) <Scott Little>

# Create new meta-patches without fuzz.  Note: patch comment text is 
often discarded
# git format-patch <SHA_prior_to_first_WRS_patch>
git format-patch $(git tag | grep wrs_post_copy)

# use meld or other diff/merge tool to transfer the revised line 
numbers.  A bit of coding magic is required to translate patch names.
cd $MY_REPO/stx/stx-integ/base/initscripts/centos/meta_patches/
for src in $(ls -1 
/localdisk/loadbuild/slittle1/starlingx-4/std/srpm_work/initscripts/rpmbuild/00*patch); 
do dest=$(ls -1 $(basename $src | cut -d '-' -f 3- | sed 
's#.patch$#*#')); meld $src $dest; done

>
>>> 3) Can we pre-build rpm, such that all packages build against the
>>> same
>>> rpm version?
>> Yes, as mentioned above we should use a consistent version.
>>
>>> 4) Are we patching any other low level build tools that have
>>> similar
>>> issues?  Possibly explaining ceph?  TBD
>>>
>> Yup more info is needed
>>
>> Sau!
>>
>>
>>>
>>> On 18-09-26 12:16 PM, Scott Little wrote:
>>>> aclocal 'too many loops' has been popping up sporadically for a
>>>> week
>>>> or two now.  Possibly 7.5 related.
>>>>
>>>> I suspect that there is a build order and/or race condition
>>>> element to
>>>> this.   It often goes away if you just run build-pkgs a second
>>>> time.
>>>>
>>>> The second possible element is that build-pkgs is using flags
>>>> that
>>>> preserve the mock environment between packages.  The goal was to
>>>> avoid
>>>> reinstalling required packages that are often common across the
>>>> packages we build.   It was a build time speedup that has been in
>>>> use
>>>> for 2-3 years now without incident.
>>>>
>>>> Google shows that other folks have hit 'aclocal too many loops'
>>>> as
>>>> well, and there are suggestions that it might be fixed in a
>>>> recent
>>>> update to automake.  Scanning the changelog for the latest and
>>>> greatest from gnu.org doesn't show any obvious fixes addressing
>>>> this
>>>> issue.
>>>>
>>>> Scott
>>>>
>>>>
>>>> On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
>>>>> BUILDSTDERR: aclocal: error: too many loops
>>>>> BUILDSTDERR: aclocal: Please contact<bug-automake at gnu.org>.
>>>>> BUILDSTDERR:  at /usr/share/automake-1.13/Automake/Channels.pm
>>>>> line 662.
>>>>> BUILDSTDERR: 	Automake::Channels::msg('automake', '',
>>>>> 'too many loops') called at /usr/share/automake-
>>>>> 1.13/Automake/ChannelDefs.pm line 212
>>>>> BUILDSTDERR: 	Automake::ChannelDefs::prog_error('too
>>>>> many loops') called at /usr/bin/aclocal line 1187
>>>>> BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-
>>>>> tmp.Sj0E7c (%build)
>>>>> BUILDSTDERR:     Macro expanded in comment on line 214: %global
>>>>> _libexecdir %{_exec_prefix}/lib
>>>>> BUILDSTDERR:     Bad exit status from /var/tmp/rpm-tmp.Sj0E7c
>>>>> (%build)
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Starlingx-discuss mailing list
>>>> Starlingx-discuss at lists.starlingx.io
>>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-disc
>>>> uss
>>>
>>>
>>>
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
>>> s
>>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss