Re: [Starlingx-discuss] de-fuzz centos7.5 upgraded src rpm
Hi all, Here is the patch list to de-fuzz the patches in CentOS7.5 upgrade task. https://review.openstack.org/#/q/topic:patch_fuzz+(status:open+OR+status:mer...) I notice there are some other src rpms also have the fuzzy issue. Will handle them later. Also, I find some SRC RPM itself has the fuzzy issue, such as openldap/rsync/netpbm/openssh etc. So we cannot eliminate all fuzzy issue just by rebase ourselves patches. Best Regards Shuicheng From: Scott Little [mailto:scott.little@windriver.com] Sent: Thursday, September 27, 2018 5:08 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build I have also been investigating another intermittent build error affecting initscripts. https://bugs.launchpad.net/starlingx/+bug/1794611 So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely. In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files. Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct. Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches. All the work to upgrade to 7.5 has created a lot of fuzzy patches. audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2 So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging. So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems. 2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files? 3) Can we pre-build rpm, such that all packages build against the same rpm version? 4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD On 18-09-26 12:16 PM, Scott Little wrote: aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related. I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time. The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident. Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue. Scott On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote: BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact <bug-automake@gnu.org><mailto:bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Yes, I've seen some upstream packages that ship from CentOS or EL7 with fuzzy patches. Can't do much about that. My list might include packages where the only fuzzy patches are those coming from CentOS. We'll have to sort through them by hand I think. Scott On 18-09-27 10:24 AM, Lin, Shuicheng wrote:
Hi all,
Here is the patch list to de-fuzz the patches in CentOS7.5 upgrade task.
https://review.openstack.org/#/q/topic:patch_fuzz+(status:open+OR+status:mer...) <https://review.openstack.org/#/q/topic:patch_fuzz+%28status:open+OR+status:merged%29>
I notice there are some other src rpms also have the fuzzy issue. Will handle them later.
Also, I find some SRC RPM itself has the fuzzy issue, such as openldap/rsync/netpbm/openssh etc.
So we cannot eliminate all fuzzy issue just by rebase ourselves patches.
Best Regards
Shuicheng
*From:*Scott Little [mailto:scott.little@windriver.com] *Sent:* Thursday, September 27, 2018 5:08 AM *To:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] build-pkgs cannot complete std build
I have also been investigating another intermittent build error affecting initscripts.
https://bugs.launchpad.net/starlingx/+bug/1794611
So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely.
In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files.
Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct.
Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches.
All the work to upgrade to 7.5 has created a lot of fuzzy patches.
audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2
So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging.
So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems. 2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files? 3) Can we pre-build rpm, such that all packages build against the same rpm version? 4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD
On 18-09-26 12:16 PM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops
BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org> <mailto:bug-automake@gnu.org>.
BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662.
BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212
BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187
BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib
BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io>
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Scott, What do you think about upgrade rpm in mirror lst to 4.14 version? So we could get rid of the fuzz issue. To do this upgrade, rpm will be moved from centos repo lst to tarball lst. Here is the release notes of rpm-4.13 to fix this fuzz issue: Enable –no-backup-if-mismatch by default in %patch macro (RhBug:884755<https://bugzilla.redhat.com/show_bug.cgi?id=884755>) Best Regards Shuicheng From: Scott Little [mailto:scott.little@windriver.com] Sent: Thursday, September 27, 2018 11:24 PM To: Lin, Shuicheng <shuicheng.lin@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] de-fuzz centos7.5 upgraded src rpm Yes, I've seen some upstream packages that ship from CentOS or EL7 with fuzzy patches. Can't do much about that. My list might include packages where the only fuzzy patches are those coming from CentOS. We'll have to sort through them by hand I think. Scott On 18-09-27 10:24 AM, Lin, Shuicheng wrote: Hi all, Here is the patch list to de-fuzz the patches in CentOS7.5 upgrade task. https://review.openstack.org/#/q/topic:patch_fuzz+(status:open+OR+status:merged)<https://review.openstack.org/#/q/topic:patch_fuzz+%28status:open+OR+status:merged%29> I notice there are some other src rpms also have the fuzzy issue. Will handle them later. Also, I find some SRC RPM itself has the fuzzy issue, such as openldap/rsync/netpbm/openssh etc. So we cannot eliminate all fuzzy issue just by rebase ourselves patches. Best Regards Shuicheng From: Scott Little [mailto:scott.little@windriver.com] Sent: Thursday, September 27, 2018 5:08 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build I have also been investigating another intermittent build error affecting initscripts. https://bugs.launchpad.net/starlingx/+bug/1794611 So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely. In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files. Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct. Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches. All the work to upgrade to 7.5 has created a lot of fuzzy patches. audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2 So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging. So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems. 2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files? 3) Can we pre-build rpm, such that all packages build against the same rpm version? 4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD On 18-09-26 12:16 PM, Scott Little wrote: aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related. I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time. The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident. Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue. Scott On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote: BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact <bug-automake@gnu.org><mailto:bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Certainly that will prevent creation of .orig files, so the risk of build failures in the packaging phase will be eliminated. However I have strong distrust of fuzzy patches. It suggests to me that folks are upgrading packages without doing the due diligence to make sure the patches are still needed, and are being applied correctly. Also I have seen twice in my career, a fuzzy patch get improperly applied to a neighboring bit of code that happened to look similar to the proper target. It's rare, but it can occur. On the plus side, moving to 4.14 would give us a consistent version of rpm throughout the build. That I like. I'm thinking we should do it, but we need a no fuzzy patches policy, and perhaps we create an audit that looks for fuzzy patches. Scott On 18-09-28 02:27 AM, Lin, Shuicheng wrote:
Hi Scott,
What do you think about upgrade rpm in mirror lst to 4.14 version? So we could get rid of the fuzz issue.
To do this upgrade, rpm will be moved from centos repo lst to tarball lst.
Here is the release notes of rpm-4.13 to fix this fuzz issue:
Enable –no-backup-if-mismatch by default in %patch macro (RhBug:884755 <https://bugzilla.redhat.com/show_bug.cgi?id=884755>)
Best Regards
Shuicheng
*From:*Scott Little [mailto:scott.little@windriver.com] *Sent:* Thursday, September 27, 2018 11:24 PM *To:* Lin, Shuicheng <shuicheng.lin@intel.com>; starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] de-fuzz centos7.5 upgraded src rpm
Yes, I've seen some upstream packages that ship from CentOS or EL7 with fuzzy patches. Can't do much about that.
My list might include packages where the only fuzzy patches are those coming from CentOS. We'll have to sort through them by hand I think.
Scott
On 18-09-27 10:24 AM, Lin, Shuicheng wrote:
Hi all,
Here is the patch list to de-fuzz the patches in CentOS7.5 upgrade task.
https://review.openstack.org/#/q/topic:patch_fuzz+(status:open+OR+status:mer...) <https://review.openstack.org/#/q/topic:patch_fuzz+%28status:open+OR+status:merged%29>
I notice there are some other src rpms also have the fuzzy issue. Will handle them later.
Also, I find some SRC RPM itself has the fuzzy issue, such as openldap/rsync/netpbm/openssh etc.
So we cannot eliminate all fuzzy issue just by rebase ourselves patches.
Best Regards
Shuicheng
*From:*Scott Little [mailto:scott.little@windriver.com] *Sent:* Thursday, September 27, 2018 5:08 AM *To:* starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* Re: [Starlingx-discuss] build-pkgs cannot complete std build
I have also been investigating another intermittent build error affecting initscripts.
https://bugs.launchpad.net/starlingx/+bug/1794611
So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely.
In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files.
Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct.
Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches.
All the work to upgrade to 7.5 has created a lot of fuzzy patches.
audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2
So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging.
So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems. 2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files? 3) Can we pre-build rpm, such that all packages build against the same rpm version? 4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD
On 18-09-26 12:16 PM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops
BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org> <mailto:bug-automake@gnu.org>.
BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662.
BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212
BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187
BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib
BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io>
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 09/28/2018 06:54 AM, Scott Little wrote:
Certainly that will prevent creation of .orig files, so the risk of build failures in the packaging phase will be eliminated. However I have strong distrust of fuzzy patches. It suggests to me that folks are upgrading packages without doing the due diligence to make sure the patches are still needed, and are being applied correctly.
Also I have seen twice in my career, a fuzzy patch get improperly applied to a neighboring bit of code that happened to look similar to the proper target. It's rare, but it can occur.
I agree with this, I have also been "bitten" by a mis-applied fuzz patch.
On the plus side, moving to 4.14 would give us a consistent version of rpm throughout the build. That I like.
I'm thinking we should do it, but we need a no fuzzy patches policy, and perhaps we create an audit that looks for fuzzy patches.
This sounds like a good plan for post-October release Sau!
Scott
On 18-09-28 02:27 AM, Lin, Shuicheng wrote:
Hi Scott,
What do you think about upgrade rpm in mirror lst to 4.14 version? So we could get rid of the fuzz issue.
To do this upgrade, rpm will be moved from centos repo lst to tarball lst.
Here is the release notes of rpm-4.13 to fix this fuzz issue:
Enable –no-backup-if-mismatch by default in %patch macro (RhBug:884755 <https://bugzilla.redhat.com/show_bug.cgi?id=884755>)
Best Regards
Shuicheng
*From:*Scott Little [mailto:scott.little@windriver.com] *Sent:* Thursday, September 27, 2018 11:24 PM *To:* Lin, Shuicheng <shuicheng.lin@intel.com>; starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] de-fuzz centos7.5 upgraded src rpm
Yes, I've seen some upstream packages that ship from CentOS or EL7 with fuzzy patches. Can't do much about that.
My list might include packages where the only fuzzy patches are those coming from CentOS. We'll have to sort through them by hand I think.
Scott
On 18-09-27 10:24 AM, Lin, Shuicheng wrote:
Hi all,
Here is the patch list to de-fuzz the patches in CentOS7.5 upgrade task.
https://review.openstack.org/#/q/topic:patch_fuzz+(status:open+OR+status:mer...) <https://review.openstack.org/#/q/topic:patch_fuzz+%28status:open+OR+status:merged%29>
I notice there are some other src rpms also have the fuzzy issue. Will handle them later.
Also, I find some SRC RPM itself has the fuzzy issue, such as openldap/rsync/netpbm/openssh etc.
So we cannot eliminate all fuzzy issue just by rebase ourselves patches.
Best Regards
Shuicheng
*From:*Scott Little [mailto:scott.little@windriver.com] *Sent:* Thursday, September 27, 2018 5:08 AM *To:* starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* Re: [Starlingx-discuss] build-pkgs cannot complete std build
I have also been investigating another intermittent build error affecting initscripts.
https://bugs.launchpad.net/starlingx/+bug/1794611
So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely.
In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files.
Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct.
Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches.
All the work to upgrade to 7.5 has created a lot of fuzzy patches.
audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2
So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging.
So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems. 2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files? 3) Can we pre-build rpm, such that all packages build against the same rpm version? 4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD
On 18-09-26 12:16 PM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops
BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org> <mailto:bug-automake@gnu.org>.
BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662.
BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212
BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187
BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib
BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io>
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Agree with Scott Brent From: Scott Little [mailto:scott.little@windriver.com] Sent: Friday, September 28, 2018 9:54 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] de-fuzz centos7.5 upgraded src rpm Certainly that will prevent creation of .orig files, so the risk of build failures in the packaging phase will be eliminated. However I have strong distrust of fuzzy patches. It suggests to me that folks are upgrading packages without doing the due diligence to make sure the patches are still needed, and are being applied correctly. Also I have seen twice in my career, a fuzzy patch get improperly applied to a neighboring bit of code that happened to look similar to the proper target. It's rare, but it can occur. On the plus side, moving to 4.14 would give us a consistent version of rpm throughout the build. That I like. I'm thinking we should do it, but we need a no fuzzy patches policy, and perhaps we create an audit that looks for fuzzy patches. Scott On 18-09-28 02:27 AM, Lin, Shuicheng wrote: Hi Scott, What do you think about upgrade rpm in mirror lst to 4.14 version? So we could get rid of the fuzz issue. To do this upgrade, rpm will be moved from centos repo lst to tarball lst. Here is the release notes of rpm-4.13 to fix this fuzz issue: Enable –no-backup-if-mismatch by default in %patch macro (RhBug:884755<https://bugzilla.redhat.com/show_bug.cgi?id=884755>) Best Regards Shuicheng From: Scott Little [mailto:scott.little@windriver.com] Sent: Thursday, September 27, 2018 11:24 PM To: Lin, Shuicheng <shuicheng.lin@intel.com><mailto:shuicheng.lin@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] de-fuzz centos7.5 upgraded src rpm Yes, I've seen some upstream packages that ship from CentOS or EL7 with fuzzy patches. Can't do much about that. My list might include packages where the only fuzzy patches are those coming from CentOS. We'll have to sort through them by hand I think. Scott On 18-09-27 10:24 AM, Lin, Shuicheng wrote: Hi all, Here is the patch list to de-fuzz the patches in CentOS7.5 upgrade task. https://review.openstack.org/#/q/topic:patch_fuzz+(status:open+OR+status:merged)<https://review.openstack.org/#/q/topic:patch_fuzz+%28status:open+OR+status:merged%29> I notice there are some other src rpms also have the fuzzy issue. Will handle them later. Also, I find some SRC RPM itself has the fuzzy issue, such as openldap/rsync/netpbm/openssh etc. So we cannot eliminate all fuzzy issue just by rebase ourselves patches. Best Regards Shuicheng From: Scott Little [mailto:scott.little@windriver.com] Sent: Thursday, September 27, 2018 5:08 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build I have also been investigating another intermittent build error affecting initscripts. https://bugs.launchpad.net/starlingx/+bug/1794611 So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely. In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files. Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct. Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches. All the work to upgrade to 7.5 has created a lot of fuzzy patches. audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2 So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging. So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems. 2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files? 3) Can we pre-build rpm, such that all packages build against the same rpm version? 4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD On 18-09-26 12:16 PM, Scott Little wrote: aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related. I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time. The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident. Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue. Scott On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote: BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact <bug-automake@gnu.org><mailto:bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi all, Two story are created to track rpm upgrade and fuzzy patch elimination. [Feature] Upgrade rpm to 4.14.0 version<https://storyboard.openstack.org/#!/story/2003916> [Enhancement] De-fuzz fuzzy patch<https://storyboard.openstack.org/#!/story/2003917> Best Regards Shuicheng From: Rowsell, Brent [mailto:Brent.Rowsell@windriver.com] Sent: Saturday, September 29, 2018 7:39 AM To: Little, Scott <Scott.Little@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] de-fuzz centos7.5 upgraded src rpm Agree with Scott Brent From: Scott Little [mailto:scott.little@windriver.com] Sent: Friday, September 28, 2018 9:54 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] de-fuzz centos7.5 upgraded src rpm Certainly that will prevent creation of .orig files, so the risk of build failures in the packaging phase will be eliminated. However I have strong distrust of fuzzy patches. It suggests to me that folks are upgrading packages without doing the due diligence to make sure the patches are still needed, and are being applied correctly. Also I have seen twice in my career, a fuzzy patch get improperly applied to a neighboring bit of code that happened to look similar to the proper target. It's rare, but it can occur. On the plus side, moving to 4.14 would give us a consistent version of rpm throughout the build. That I like. I'm thinking we should do it, but we need a no fuzzy patches policy, and perhaps we create an audit that looks for fuzzy patches. Scott On 18-09-28 02:27 AM, Lin, Shuicheng wrote: Hi Scott, What do you think about upgrade rpm in mirror lst to 4.14 version? So we could get rid of the fuzz issue. To do this upgrade, rpm will be moved from centos repo lst to tarball lst. Here is the release notes of rpm-4.13 to fix this fuzz issue: Enable –no-backup-if-mismatch by default in %patch macro (RhBug:884755<https://bugzilla.redhat.com/show_bug.cgi?id=884755>) Best Regards Shuicheng From: Scott Little [mailto:scott.little@windriver.com] Sent: Thursday, September 27, 2018 11:24 PM To: Lin, Shuicheng <shuicheng.lin@intel.com><mailto:shuicheng.lin@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] de-fuzz centos7.5 upgraded src rpm Yes, I've seen some upstream packages that ship from CentOS or EL7 with fuzzy patches. Can't do much about that. My list might include packages where the only fuzzy patches are those coming from CentOS. We'll have to sort through them by hand I think. Scott On 18-09-27 10:24 AM, Lin, Shuicheng wrote: Hi all, Here is the patch list to de-fuzz the patches in CentOS7.5 upgrade task. https://review.openstack.org/#/q/topic:patch_fuzz+(status:open+OR+status:merged)<https://review.openstack.org/#/q/topic:patch_fuzz+%28status:open+OR+status:merged%29> I notice there are some other src rpms also have the fuzzy issue. Will handle them later. Also, I find some SRC RPM itself has the fuzzy issue, such as openldap/rsync/netpbm/openssh etc. So we cannot eliminate all fuzzy issue just by rebase ourselves patches. Best Regards Shuicheng From: Scott Little [mailto:scott.little@windriver.com] Sent: Thursday, September 27, 2018 5:08 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build I have also been investigating another intermittent build error affecting initscripts. https://bugs.launchpad.net/starlingx/+bug/1794611 So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely. In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files. Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct. Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches. All the work to upgrade to 7.5 has created a lot of fuzzy patches. audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2 So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging. So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems. 2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files? 3) Can we pre-build rpm, such that all packages build against the same rpm version? 4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD On 18-09-26 12:16 PM, Scott Little wrote: aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related. I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time. The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident. Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue. Scott On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote: BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact <bug-automake@gnu.org><mailto:bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi all, Here is the latest status of the two story. You could get detail info in the story. [Feature] Upgrade rpm to 4.14.0 version<https://storyboard.openstack.org/#!/story/2003916> This story is abandoned due to there is required package version conflict between rpm4.14 and ima-evm-utils. [Enhancement] De-fuzz fuzzy patch<https://storyboard.openstack.org/#!/story/2003917> All fuzzy patch during build-srpm stage are eliminated. And most fuzzy patch from StarlingX during build-rpm stage are eliminated also. Most of the remained fuzz are from src rpm itself. Please help review below patches: https://review.openstack.org/#/q/topic:patch_fuzz+(status:open) Best Regards Shuicheng From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Saturday, September 29, 2018 10:26 AM To: Rowsell, Brent <Brent.Rowsell@windriver.com>; Little, Scott <Scott.Little@windriver.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] de-fuzz centos7.5 upgraded src rpm Hi all, Two story are created to track rpm upgrade and fuzzy patch elimination. [Feature] Upgrade rpm to 4.14.0 version<https://storyboard.openstack.org/#!/story/2003916> [Enhancement] De-fuzz fuzzy patch<https://storyboard.openstack.org/#!/story/2003917> Best Regards Shuicheng From: Rowsell, Brent [mailto:Brent.Rowsell@windriver.com] Sent: Saturday, September 29, 2018 7:39 AM To: Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] de-fuzz centos7.5 upgraded src rpm Agree with Scott Brent From: Scott Little [mailto:scott.little@windriver.com] Sent: Friday, September 28, 2018 9:54 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] de-fuzz centos7.5 upgraded src rpm Certainly that will prevent creation of .orig files, so the risk of build failures in the packaging phase will be eliminated. However I have strong distrust of fuzzy patches. It suggests to me that folks are upgrading packages without doing the due diligence to make sure the patches are still needed, and are being applied correctly. Also I have seen twice in my career, a fuzzy patch get improperly applied to a neighboring bit of code that happened to look similar to the proper target. It's rare, but it can occur. On the plus side, moving to 4.14 would give us a consistent version of rpm throughout the build. That I like. I'm thinking we should do it, but we need a no fuzzy patches policy, and perhaps we create an audit that looks for fuzzy patches. Scott On 18-09-28 02:27 AM, Lin, Shuicheng wrote: Hi Scott, What do you think about upgrade rpm in mirror lst to 4.14 version? So we could get rid of the fuzz issue. To do this upgrade, rpm will be moved from centos repo lst to tarball lst. Here is the release notes of rpm-4.13 to fix this fuzz issue: Enable –no-backup-if-mismatch by default in %patch macro (RhBug:884755<https://bugzilla.redhat.com/show_bug.cgi?id=884755>) Best Regards Shuicheng From: Scott Little [mailto:scott.little@windriver.com] Sent: Thursday, September 27, 2018 11:24 PM To: Lin, Shuicheng <shuicheng.lin@intel.com><mailto:shuicheng.lin@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] de-fuzz centos7.5 upgraded src rpm Yes, I've seen some upstream packages that ship from CentOS or EL7 with fuzzy patches. Can't do much about that. My list might include packages where the only fuzzy patches are those coming from CentOS. We'll have to sort through them by hand I think. Scott On 18-09-27 10:24 AM, Lin, Shuicheng wrote: Hi all, Here is the patch list to de-fuzz the patches in CentOS7.5 upgrade task. https://review.openstack.org/#/q/topic:patch_fuzz+(status:open+OR+status:merged)<https://review.openstack.org/#/q/topic:patch_fuzz+%28status:open+OR+status:merged%29> I notice there are some other src rpms also have the fuzzy issue. Will handle them later. Also, I find some SRC RPM itself has the fuzzy issue, such as openldap/rsync/netpbm/openssh etc. So we cannot eliminate all fuzzy issue just by rebase ourselves patches. Best Regards Shuicheng From: Scott Little [mailto:scott.little@windriver.com] Sent: Thursday, September 27, 2018 5:08 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build I have also been investigating another intermittent build error affecting initscripts. https://bugs.launchpad.net/starlingx/+bug/1794611 So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely. In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files. Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct. Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches. All the work to upgrade to 7.5 has created a lot of fuzzy patches. audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2 So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging. So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems. 2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files? 3) Can we pre-build rpm, such that all packages build against the same rpm version? 4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD On 18-09-26 12:16 PM, Scott Little wrote: aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related. I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time. The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident. Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue. Scott On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote: BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact <bug-automake@gnu.org><mailto:bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (4)
-
Lin, Shuicheng
-
Rowsell, Brent
-
Saul Wold
-
Scott Little