[Starlingx-discuss] build-pkgs cannot complete std build
Hi, This is the build issue mentioned in the meeting this morning https://bugs.launchpad.net/starlingx/+bug/1794415 <- full details are here. In summary what happens is that ceph cannot be build, the build.log for ceph says: BUILDSTDERR: aclocal: overwriting 'm4/ax_c_var_func.m4' with '/usr/share/aclocal/ax_c_var_func.m4' BUILDSTDERR: aclocal: overwriting 'm4/ax_check_compile_flag.m4' with '/usr/share/aclocal/ax_check_compile_flag.m4' BUILDSTDERR: aclocal: installing 'm4/ax_cxx_compile_stdcxx.m4' from '/usr/share/aclocal/ax_cxx_compile_stdcxx.m4' BUILDSTDERR: aclocal: overwriting 'm4/ax_cxx_compile_stdcxx_11.m4' with '/usr/share/aclocal/ax_cxx_compile_stdcxx_11.m4' BUILDSTDERR: aclocal: overwriting 'm4/ax_cxx_static_cast.m4' with '/usr/share/aclocal/ax_cxx_static_cast.m4' BUILDSTDERR: aclocal: overwriting 'm4/pkg.m4' with '/usr/share/aclocal/pkg.m4' BUILDSTDERR: aclocal: installing 'm4/ax_require_defined.m4' from '/usr/share/aclocal/ax_require_defined.m4' BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact <bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) RPM build errors: Child return code was: 1 EXCEPTION: [Error()] However, I'm able to build ceph in mock and I don't see this problem there. We'll continue investigating -Erich
Erich, Did you ever try serial build in your Jenkins? It takes time but if serial build can be successful, it may narrow the problem. Thx. - cindy -----Original Message----- From: Cordoba Malibran, Erich [mailto:erich.cordoba.malibran@intel.com] Sent: Wednesday, September 26, 2018 11:32 PM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] build-pkgs cannot complete std build Hi, This is the build issue mentioned in the meeting this morning https://bugs.launchpad.net/starlingx/+bug/1794415 <- full details are here. In summary what happens is that ceph cannot be build, the build.log for ceph says: BUILDSTDERR: aclocal: overwriting 'm4/ax_c_var_func.m4' with '/usr/share/aclocal/ax_c_var_func.m4' BUILDSTDERR: aclocal: overwriting 'm4/ax_check_compile_flag.m4' with '/usr/share/aclocal/ax_check_compile_flag.m4' BUILDSTDERR: aclocal: installing 'm4/ax_cxx_compile_stdcxx.m4' from '/usr/share/aclocal/ax_cxx_compile_stdcxx.m4' BUILDSTDERR: aclocal: overwriting 'm4/ax_cxx_compile_stdcxx_11.m4' with '/usr/share/aclocal/ax_cxx_compile_stdcxx_11.m4' BUILDSTDERR: aclocal: overwriting 'm4/ax_cxx_static_cast.m4' with '/usr/share/aclocal/ax_cxx_static_cast.m4' BUILDSTDERR: aclocal: overwriting 'm4/pkg.m4' with '/usr/share/aclocal/pkg.m4' BUILDSTDERR: aclocal: installing 'm4/ax_require_defined.m4' from '/usr/share/aclocal/ax_require_defined.m4' BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact <bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) RPM build errors: Child return code was: 1 EXCEPTION: [Error()] However, I'm able to build ceph in mock and I don't see this problem there. We'll continue investigating -Erich _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related. I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time. The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident. Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue. Scott On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
I have also been investigating another intermittent build error affecting initscripts. https://bugs.launchpad.net/starlingx/+bug/1794611 So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely. In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files. Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct. Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches. All the work to upgrade to 7.5 has created a lot of fuzzy patches. audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2 So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging. So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems. 2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files? 3) Can we pre-build rpm, such that all packages build against the same rpm version? 4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD On 18-09-26 12:16 PM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 09/26/2018 02:08 PM, Scott Little wrote:
I have also been investigating another intermittent build error affecting initscripts.
https://bugs.launchpad.net/starlingx/+bug/1794611
So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely.
Why is this the case in the first place, can't we ensure we only build rpm-4.14.0?
In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files.
Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct.
Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches.
All the work to upgrade to 7.5 has created a lot of fuzzy patches.
Wow, sorry I was not aware that they were all fuzzy patches.
audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2
So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging.
So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems.
Agreed, when updating patches should be rebased and de-fuzzed.
2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files?
It would be good to investigate that.
3) Can we pre-build rpm, such that all packages build against the same rpm version?
Yes, as mentioned above we should use a consistent version.
4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD
Yup more info is needed Sau!
On 18-09-26 12:16 PM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Wed, 2018-09-26 at 16:44 -0700, Saul Wold wrote:
On 09/26/2018 02:08 PM, Scott Little wrote:
I have also been investigating another intermittent build error affecting initscripts.
https://bugs.launchpad.net/starlingx/+bug/1794611
So any given package might be built by rpm-4.11.3, or by rpm- 4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely.
Why is this the case in the first place, can't we ensure we only build rpm-4.14.0?
In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files.
Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct.
Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches.
All the work to upgrade to 7.5 has created a lot of fuzzy patches.
Wow, sorry I was not aware that they were all fuzzy patches.
How did you got this list? There were .orig files in those folder?
audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2
So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging.
So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems.
Agreed, when updating patches should be rebased and de-fuzzed.
2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files?
It would be good to investigate that.
Seems we can pass _default_patch_fuzz 0 as a variable to rpmbuild. What could be the course of action here? - Wire rpmbuild to not allow fuzzy patches. - Then iteratively try to build until all the index are correct -Erich
3) Can we pre-build rpm, such that all packages build against the same rpm version?
Yes, as mentioned above we should use a consistent version.
4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD
Yup more info is needed
Sau!
On 18-09-26 12:16 PM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake- 1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm- tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-disc uss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus s
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 18-09-26 08:05 PM, Cordoba Malibran, Erich wrote:
On Wed, 2018-09-26 at 16:44 -0700, Saul Wold wrote:
On 09/26/2018 02:08 PM, Scott Little wrote:
I have also been investigating another intermittent build error affecting initscripts.
https://bugs.launchpad.net/starlingx/+bug/1794611
So any given package might be built by rpm-4.11.3, or by rpm- 4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely.
Why is this the case in the first place, can't we ensure we only build rpm-4.14.0?
In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files.
Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct.
Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches.
All the work to upgrade to 7.5 has created a lot of fuzzy patches.
Wow, sorry I was not aware that they were all fuzzy patches.
How did you got this list? There were .orig files in those folder? My query was ...
for f in $(grep -l 'Hunk' $MY_WORKSPACE/std/results/*/*/build.log); do basename $(dirname $f); done As noted in another thread, we might get false positives in the form of CentOS/EL7 patches that are fuzzy, not our own.
audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2
So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging.
So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems. Agreed, when updating patches should be rebased and de-fuzzed.
2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files? It would be good to investigate that.
Seems we can pass _default_patch_fuzz 0 as a variable to rpmbuild.
What could be the course of action here? - Wire rpmbuild to not allow fuzzy patches. - Then iteratively try to build until all the index are correct
-Erich
I usually start by using a combination of 'build-pkgs --edit', 'git format-patch', and meld (a graphical diff/merge tool). e.g. # If 'build-pkgs --edit' works, it's rather fragile as it's more or less parsing spec files, it produces # two gits, on for working on the tarball content, and a second for the spec file and other files outside the tarball. # Patches are seen as commits, applied with fuzz allowed. STX patches will get a WRS prefix (sorry, I'll make that STX in future) build-pkgs --edit --clean initscripts build-pkgs --edit initscripts ... 11:36:45 b7: ===== 'initscripts' has been extracted for editing. ===== 11:36:45 b7: ===== Metadata can be found at: /localdisk/loadbuild/slittle1/starlingx-4/std/srpm_work/initscripts/rpmbuild 11:36:45 b7: ===== Source code can be found at: /localdisk/loadbuild/slittle1/starlingx-4/std/srpm_work/initscripts/gits/initscripts.spec/initscripts-9.49.41 cd /localdisk/loadbuild/slittle1/starlingx-4/std/srpm_work/initscripts/gits/initscripts.spec/initscripts-9.49.41 git log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%ci) %C(bold blue)<%an>%Creset' --abbrev-commit * 3aaf0ca - (HEAD, initscripts-9.49.41-1.el7) WRS: Patch15: run-dhclient-as-daemon-for-ipv6.patch (2018-09-27 11:36:45 -0400) <Scott Little> * 126e810 - WRS: Patch14: ifup-eth-stop-waiting-if-link-is-up.patch (2018-09-27 11:36:44 -0400) <Scott Little> * df98ada - WRS: Patch13: ipv6-static-route-support.patch (2018-09-27 11:36:44 -0400) <Scott Little> * cc391dd - WRS: Patch12: sysconfig-unsafe-usage-of-linkdelay-variable.patch (2018-09-27 11:36:44 -0400) <Scott Little> * 398cdf6 - WRS: Patch11: sysconfig-affirmative-check-for-link-carrier.patch (2018-09-27 11:36:43 -0400) <Scott Little> * a3ebe9e - WRS: Patch10: run-ifdown-on-all-interfaces.patch (2018-09-27 11:36:43 -0400) <Scott Little> * 0a6e1fc - WRS: Patch9: 0001-force-delay-check-link-down.patch (2018-09-27 11:36:43 -0400) <Scott Little> * 45075af - WRS: Patch8: 0001-dhclient-remove-1-arg.patch (2018-09-27 11:36:42 -0400) <Scott Little> * 02f8215 - WRS: Patch7: support-interface-promisc.patch (2018-09-27 11:36:42 -0400) <Scott Little> * c5b8eab - WRS: Patch6: dhclient-restrict-interfaces-to-those-on-c.patch (2018-09-27 11:36:41 -0400) <Scott Little> * 6d8bcb5 - WRS: Patch5: relocate-dhclient-leases-to-var-run.patch (2018-09-27 11:36:41 -0400) <Scott Little> * 480e9a5 - WRS: Patch4: support-interface-scriptlets.patch (2018-09-27 11:36:41 -0400) <Scott Little> * 487fdfa - (tag: pre_wrs_initscripts-9.49.41-1.el7) ORIGINAL: extracted archive (2018-09-27 11:36:39 -0400) <Scott Little> * 7ca8368 - (master) ORIGINAL: initial commit (2018-09-27 11:36:39 -0400) <Scott Little> # Create new patches without fuzz. Note: patch comment text is often discarded # git format-patch <SHA_prior_to_first_WRS_patch> git format-patch $(git tag | grep pre_wrs_) # use meld or other diff/merge tool to transfer the revised line numbers. A bit of coding magic is required to translate patch names. cd $MY_REPO/stx/stx-integ/base/initscripts/centos/patches/ for src in $(ls -1 /localdisk/loadbuild/slittle1/starlingx-4/std/srpm_work/initscripts/gits/initscripts.spec/initscripts-9.49.41/00*patch); do dest=$(ls -1 $(basename $src | cut -d '-' -f 4- | sed 's#.patch$#*#')); meld $src $dest; done cd /localdisk/loadbuild/slittle1/starlingx-4/std/srpm_work/initscripts/rpmbuild git log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%ci) %C(bold blue)<%an>%Creset' --abbrev-commit * 4ecfd77 - (HEAD, initscripts-9.49.41-1.el7) WRS: spec-run-dhclient-as-daemon-for-ipv6.patch (2018-09-27 11:36:40 -0400) <Scott Little> * 69d0bb4 - WRS: spec-ifup-eth-stop-waiting-if-link-is-up.patch (2018-09-27 11:36:40 -0400) <Scott Little> * e2942ca - WRS: ipv6-static-route-support.patch (2018-09-27 11:36:40 -0400) <Scott Little> * 4e0368a - WRS: fix-build-failures-due-to-unwanted-sgid.patch (2018-09-27 11:36:40 -0400) <Scott Little> * bc43676 - WRS: spec-sysconfig-unsafe-usage-of-linkdelay-variable.patch (2018-09-27 11:36:40 -0400) <Scott Little> * 6a38a1d - WRS: spec-sysconfig-affirmative-check-for-link-carrier.patch (2018-09-27 11:36:40 -0400) <Scott Little> * 730fe3c - WRS: 0001-Update-package-versioning-for-TIS-format.patch (2018-09-27 11:36:39 -0400) <Scott Little> * a083a54 - WRS: run-ifdown-on-all-interfaces.patch (2018-09-27 11:36:39 -0400) <Scott Little> * 7f02bc2 - WRS: force-delay-check-link-down.patch (2018-09-27 11:36:39 -0400) <Scott Little> * dbdb232 - WRS: 0001-Fix-Interfaces-intermittenly-do-not-come-up-on-boot.patch (2018-09-27 11:36:39 -0400) <Scott Little> * c51c427 - WRS: spec-add-mountnfs-init-script.patch (2018-09-27 11:36:39 -0400) <Scott Little> * 0a5a833 - WRS: 0001-Support-PROMISC-for-interfaces-config.patch (2018-09-27 11:36:39 -0400) <Scott Little> * e0bbac3 - WRS: 0001-Disable-zeroconf-route.patch (2018-09-27 11:36:39 -0400) <Scott Little> * 6485d13 - WRS: stop-creating-shared-dirs.patch (2018-09-27 11:36:39 -0400) <Scott Little> * 7223930 - WRS: spec-include-TiS-changes.patch (2018-09-27 11:36:39 -0400) <Scott Little> * b63eb3d - (tag: wrs_post_copy_list_initscripts-9.49.41-1.el7) WRS: COPY_LIST content (2018-09-27 11:36:39 -0400) <Scott Little> * 8e1e97a - (tag: pre_wrs_initscripts-9.49.41-1.el7) ORIGINAL: srpm extract (2018-09-27 11:36:38 -0400) <Scott Little> * a6928fd - (master) ORIGINAL: initial commit (2018-09-27 11:36:38 -0400) <Scott Little> # Create new meta-patches without fuzz. Note: patch comment text is often discarded # git format-patch <SHA_prior_to_first_WRS_patch> git format-patch $(git tag | grep wrs_post_copy) # use meld or other diff/merge tool to transfer the revised line numbers. A bit of coding magic is required to translate patch names. cd $MY_REPO/stx/stx-integ/base/initscripts/centos/meta_patches/ for src in $(ls -1 /localdisk/loadbuild/slittle1/starlingx-4/std/srpm_work/initscripts/rpmbuild/00*patch); do dest=$(ls -1 $(basename $src | cut -d '-' -f 3- | sed 's#.patch$#*#')); meld $src $dest; done
3) Can we pre-build rpm, such that all packages build against the same rpm version? Yes, as mentioned above we should use a consistent version.
4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD
Yup more info is needed
Sau!
On 18-09-26 12:16 PM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake- 1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm- tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-disc uss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus s
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 18-09-26 08:05 PM, Cordoba Malibran, Erich wrote:
Seems we can pass _default_patch_fuzz 0 as a variable to rpmbuild.
What could be the course of action here? - Wire rpmbuild to not allow fuzzy patches. - Then iteratively try to build until all the index are correct
-Erich
I don't think that would work. Some of the CentOS/EL7 src.rpm's ship with embedded fuzzy patches. We can't fix those.
Hi Saul, It looks like the first step is for all the 7.5 patches to be re-generated and de-fuzzed. I consider this to be gating the October release. Do you agree? I suggest that this gets addressed in master before the release branch creation. I will add a new task to the CentOS 7.5 rebase story for Cindy's team to work on. I would also like to propose that we continue to place master under code freeze/limited merge and only allow the items we consider gating for October. This would allow us to limit the churn to the next few days. Is this ok with everyone? Thanks, Ghada -----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Wednesday, September 26, 2018 7:45 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build On 09/26/2018 02:08 PM, Scott Little wrote:
I have also been investigating another intermittent build error affecting initscripts.
https://bugs.launchpad.net/starlingx/+bug/1794611
So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely.
Why is this the case in the first place, can't we ensure we only build rpm-4.14.0?
In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files.
Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct.
Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches.
All the work to upgrade to 7.5 has created a lot of fuzzy patches.
Wow, sorry I was not aware that they were all fuzzy patches.
audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2
So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging.
So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems.
Agreed, when updating patches should be rebased and de-fuzzed.
2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files?
It would be good to investigate that.
3) Can we pre-build rpm, such that all packages build against the same rpm version?
Yes, as mentioned above we should use a consistent version.
4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD
Yup more info is needed Sau!
On 18-09-26 12:16 PM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
I agree. Brent -----Original Message----- From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, September 26, 2018 9:10 PM To: Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io; Xie, Cindy <cindy.xie@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build Hi Saul, It looks like the first step is for all the 7.5 patches to be re-generated and de-fuzzed. I consider this to be gating the October release. Do you agree? I suggest that this gets addressed in master before the release branch creation. I will add a new task to the CentOS 7.5 rebase story for Cindy's team to work on. I would also like to propose that we continue to place master under code freeze/limited merge and only allow the items we consider gating for October. This would allow us to limit the churn to the next few days. Is this ok with everyone? Thanks, Ghada -----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Wednesday, September 26, 2018 7:45 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build On 09/26/2018 02:08 PM, Scott Little wrote:
I have also been investigating another intermittent build error affecting initscripts.
https://bugs.launchpad.net/starlingx/+bug/1794611
So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely.
Why is this the case in the first place, can't we ensure we only build rpm-4.14.0?
In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files.
Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct.
Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches.
All the work to upgrade to 7.5 has created a lot of fuzzy patches.
Wow, sorry I was not aware that they were all fuzzy patches.
audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2
So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging.
So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems.
Agreed, when updating patches should be rebased and de-fuzzed.
2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files?
It would be good to investigate that.
3) Can we pre-build rpm, such that all packages build against the same rpm version?
Yes, as mentioned above we should use a consistent version.
4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD
Yup more info is needed Sau!
On 18-09-26 12:16 PM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Cindy, do you agree? How long will it take for the patches to be corrected? Team - assuming her answer is "more than a day or two", do we want to hold the release branch creation that long? We're moving the release into day-for-day slip mode... brucej -----Original Message----- From: Rowsell, Brent [mailto:Brent.Rowsell@windriver.com] Sent: Wednesday, September 26, 2018 6:12 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io; Xie, Cindy <cindy.xie@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build I agree. Brent -----Original Message----- From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, September 26, 2018 9:10 PM To: Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io; Xie, Cindy <cindy.xie@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build Hi Saul, It looks like the first step is for all the 7.5 patches to be re-generated and de-fuzzed. I consider this to be gating the October release. Do you agree? I suggest that this gets addressed in master before the release branch creation. I will add a new task to the CentOS 7.5 rebase story for Cindy's team to work on. I would also like to propose that we continue to place master under code freeze/limited merge and only allow the items we consider gating for October. This would allow us to limit the churn to the next few days. Is this ok with everyone? Thanks, Ghada -----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Wednesday, September 26, 2018 7:45 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build On 09/26/2018 02:08 PM, Scott Little wrote:
I have also been investigating another intermittent build error affecting initscripts.
https://bugs.launchpad.net/starlingx/+bug/1794611
So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely.
Why is this the case in the first place, can't we ensure we only build rpm-4.14.0?
In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files.
Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct.
Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches.
All the work to upgrade to 7.5 has created a lot of fuzzy patches.
Wow, sorry I was not aware that they were all fuzzy patches.
audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2
So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging.
So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems.
Agreed, when updating patches should be rebased and de-fuzzed.
2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files?
It would be good to investigate that.
3) Can we pre-build rpm, such that all packages build against the same rpm version?
Yes, as mentioned above we should use a consistent version.
4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD
Yup more info is needed Sau!
On 18-09-26 12:16 PM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi all, I will try to finish the fuzzy correction today. Best Regards Shuicheng -----Original Message----- From: Jones, Bruce E Sent: Thursday, September 27, 2018 9:20 AM To: Rowsell, Brent <Brent.Rowsell@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io; Xie, Cindy <cindy.xie@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] build-pkgs cannot complete std build Cindy, do you agree? How long will it take for the patches to be corrected? Team - assuming her answer is "more than a day or two", do we want to hold the release branch creation that long? We're moving the release into day-for-day slip mode... brucej -----Original Message----- From: Rowsell, Brent [mailto:Brent.Rowsell@windriver.com] Sent: Wednesday, September 26, 2018 6:12 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io; Xie, Cindy <cindy.xie@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build I agree. Brent -----Original Message----- From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, September 26, 2018 9:10 PM To: Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io; Xie, Cindy <cindy.xie@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build Hi Saul, It looks like the first step is for all the 7.5 patches to be re-generated and de-fuzzed. I consider this to be gating the October release. Do you agree? I suggest that this gets addressed in master before the release branch creation. I will add a new task to the CentOS 7.5 rebase story for Cindy's team to work on. I would also like to propose that we continue to place master under code freeze/limited merge and only allow the items we consider gating for October. This would allow us to limit the churn to the next few days. Is this ok with everyone? Thanks, Ghada -----Original Message----- From: Saul Wold [mailto:sgw@linux.intel.com] Sent: Wednesday, September 26, 2018 7:45 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build On 09/26/2018 02:08 PM, Scott Little wrote:
I have also been investigating another intermittent build error affecting initscripts.
https://bugs.launchpad.net/starlingx/+bug/1794611
So any given package might be built by rpm-4.11.3, or by rpm-4.14.0. It's a race with many inputs. Ideally it shouldn't matter, but it does. build-pkgs --serial might reduce the odds, but likely won't solve it entirely.
Why is this the case in the first place, can't we ensure we only build rpm-4.14.0?
In this case it is a new option that rpm passes to the patch command, --no-backup-if-mismatch, first introduced by rpm-4.13. This option has the effect of suppressing the creation of '.orig' files.
Creation of .orig files are a consequence of applying patches that are not clean. Unclean patches are ones that require 'fuzzing', i.e. treat the patch line numbers as approximate, rather than a strict requirement, just so long as the before/after context seems to be correct.
Prior to StarlingX, my policy for rebasing patches was that no fuzz is tolerated in our patches.
All the work to upgrade to 7.5 has created a lot of fuzzy patches.
Wow, sorry I was not aware that they were all fuzzy patches.
audit-2.8.1-3.el7.tis.2 bash-4.2.46-30.el7.tis.3 dhcp-4.2.5-68.el7.centos.1.tis.8 dnsmasq-2.76-5.el7.tis.6 drbd-8.4.3-0.tis.6 facter-2.4.4-4.el7.tis.4 haproxy-1.5.18-7.el7.tis.7 initscripts-9.49.41-1.el7.tis.16 iptables-1.4.21-24.1.el7_5.tis.3 kubernetes-1.10.0-1.tis.1 libevent-2.0.21-4.el7.tis.2 lighttpd-1.4.50-1.el7.tis.6 logrotate-3.8.6-15.el7.tis.3 netpbm-10.79.00-7.el7.tis.2 net-snmp-5.7.2-33.el7_5.2.tis.10 net-tools-2.0-0.22.20131004git.el7.tis.2 nfs-utils-1.3.0-0.54.el7.tis.4 nss-pam-ldapd-0.8.13-16.el7.tis.4 ntp-4.2.6p5-28.el7.centos.tis.3 openldap-2.4.44-15.el7_5.tis.8 openssh-7.4p1-16.el7_4.tis.9 pam-1.1.8-22.el7.tis.4 puppet-4.8.2-1.el7.tis.2 puppet-ceph-2.4.1-1.el7.tis.4 puppet-horizon-11.5.0-1.el7.tis.1 python-2.7.5-69.el7_5.tis.3 python-keyring-5.7.1-1.tis.2 python-wsme-0.9.2-1.el7.tis.3 resource-agents-3.9.5-124.el7.tis.12 rsync-3.1.2-4.el7.tis.2 shadow-utils-4.1.5.1-24.el7.tis.4 sudo-1.8.19p2-14.el7_5.tis.3 watchdog-5.13-11.el7.tis.2
So any of these packages might or might not produce unwanted .orig files. The .orig files might or might not be packaged, or break packaging.
So there are a number of threads to pull at here. 1) Are fuzzy patches tolerated. I vote no. Sooner or later a fuzzy patch will be mis-applied and cause us problems.
Agreed, when updating patches should be rebased and de-fuzzed.
2) Can we force rpmbuild within mock to use a consistent policy with respect to creation of orig files?
It would be good to investigate that.
3) Can we pre-build rpm, such that all packages build against the same rpm version?
Yes, as mentioned above we should use a consistent version.
4) Are we patching any other low level build tools that have similar issues? Possibly explaining ceph? TBD
Yup more info is needed Sau!
On 18-09-26 12:16 PM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Wed, Sep 26, 2018 at 8:19 PM, Jones, Bruce E <bruce.e.jones@intel.com> wrote:
Team - assuming her answer is "more than a day or two", do we want to hold the release branch creation that long? We're moving the release into day-for-day slip mode...
The tradeoff is how much work it would be to backport that after a branch, and how much testing is invalidated after such a change is backported... If this needs to be completed for testing to begin then it is easy, I suspect that isn't the case though. If this is a build issue it may be release blocking but not branch blocking, provided a build can be expected to be available to begin testing. dt -- Dean Troyer dtroyer@gmail.com
I think we agreed in yesterday's meeting the build failure is blocking the branch. Whichever issues that we are finding (including fuzzy patches), it should hold off the branching. I really would like to see a successful build from master before a branch can be created. But the impact is to hold off the follow-on feature patches from merging to master. Thx. - cindy -----Original Message----- From: Dean Troyer [mailto:dtroyer@gmail.com] Sent: Thursday, September 27, 2018 9:26 AM To: Jones, Bruce E <bruce.e.jones@intel.com> Cc: Rowsell, Brent <Brent.Rowsell@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io; Xie, Cindy <cindy.xie@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: Re: [Starlingx-discuss] build-pkgs cannot complete std build On Wed, Sep 26, 2018 at 8:19 PM, Jones, Bruce E <bruce.e.jones@intel.com> wrote:
Team - assuming her answer is "more than a day or two", do we want to hold the release branch creation that long? We're moving the release into day-for-day slip mode...
The tradeoff is how much work it would be to backport that after a branch, and how much testing is invalidated after such a change is backported... If this needs to be completed for testing to begin then it is easy, I suspect that isn't the case though. If this is a build issue it may be release blocking but not branch blocking, provided a build can be expected to be available to begin testing. dt -- Dean Troyer dtroyer@gmail.com
Trying to get a serial build avoiding the possible mock problems I encounter with this issue: BUILD_AVOIDANCE_URL= /localdisk/designer/jenkins-slave/starlingx/cgcs-root/build-tools/build-srpms-serial --std /localdisk/designer/jenkins-slave/starlingx/cgcs-root/build-tools/build-srpms-serial: line 33: /build-srpms-common.sh: No such file or directory CLEAN_FLAG=0 22:26:39 build-srpms-serial --std 22:26:39 make: all 22:26:40 ===== Build SRPM for 'remote-clients' ===== 22:26:40 PKG_BASE=/localdisk/designer/jenkins-slave/starlingx/cgcs-root/stx/stx-clients/remote-clients 22:26:40 WORK_BASE=/localdisk/loadbuild/jenkins-slave/starlingx/std/inputs/stx/stx-clients/remote-clients 22:26:40 RPMBUILD_BASE=/localdisk/loadbuild/jenkins-slave/starlingx/std/inputs/stx/stx-clients/remote-clients/rpmbuild 22:26:40 /localdisk/designer/jenkins-slave/starlingx/cgcs-root/build-tools/build-srpms-serial: line 941: md5sums_from_input_vars: command not found 22:26:40 ERROR: build_dir_spec (943): md5sums_from_input_vars '' '/localdisk/designer/jenkins-slave/starlingx/cgcs-root/stx/stx-clients/remote-clients/centos/remote-clients.spec' '/localdisk/loadbuild/jenkins-slave/starlingx/std/rpmbuild/SOURCES/remote-clients' Doing sed -i 's/PARALLEL/SERIAL/g' build-tools/build-srpms-serial fixes this issue. However, after the srpms are created the build-rpm-serial script is unable to find these srpms. See below ######## Wed Sep 26 22:36:34 UTC 2018: build-srpm-serial --std was successful ######## Wed Sep 26 22:36:34 UTC 2018: Launching build-rpms-serial --std /localdisk/designer/jenkins-slave/starlingx/cgcs-root/build-tools/build-rpms-serial --std FORMAL_BUILD=0 modify-build-cfg /localdisk/loadbuild/jenkins-slave/starlingx/std/jenkins-slave-starlingx-tis-r5-pike-std.cfg Nothing to build in '/localdisk/loadbuild/jenkins-slave/starlingx/std/rpmbuild/SRPMS' ######## Wed Sep 26 22:36:35 UTC 2018: build-rpm-serial --std was successful This is reported here: https://bugs.launchpad.net/starlingx/+bug/1790583 I tried this but it didn't work for me: https://review.openstack.org/#/c/603983/ So, for now seems that it's not possible to use serial building. On Wed, 2018-09-26 at 12:16 -0400, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact <bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake- 1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 09/26/2018 09:16 AM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
I am seeing this failure also, but it does not go away after a second rebuild. I have the lastest stx-root (build-tools) with the recent patches. Is this directly related to the fuzz issue or is there something else we need to address in CEPH itself. This is blocking my local build. Sau!
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2-3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake-1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Our latest build, based on code synced at 2018-09-27T15:28:00 UTC, built successfully. It took three attempts to get ceph built. The first two passes aborted quickly due to missing packages. The final pass did not exhibit the 'aclocal: too many loops'**issue. The only build I have that exhibited the too many loops error was a snapshot on 2018-09-20T15:50:40 UTC I do have a designer with an older snapshot that seems to hit it regularly, so I'll work with him and see if we can learn more. I think we need more data from the community - Who's build is failing on ceph with *aclocal: too many loops?* - Who is building successfully ? - Who can build only intermittently? Info to collect for failed builds: - build command used? - Was it a new workspace, a cleaned workspace, or a previously used workspace? - $MY_WORKSPACE/build-std.log - $MY_WORKSPACE/std/results/*/ceph-*/*.log For successful builds, same info. Rather than full build logs, I can settle for: - grep '\(Success building\|iteration\|building ceph\)' $MY_WORKSPACE/build-std.log - grep compute_resources: build-std.log On 18-09-27 02:21 PM, Saul Wold wrote:
On 09/26/2018 09:16 AM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
I am seeing this failure also, but it does not go away after a second rebuild. I have the lastest stx-root (build-tools) with the recent patches.
Is this directly related to the fuzz issue or is there something else we need to address in CEPH itself.
This is blocking my local build.
Sau!
On 18-09-27 03:53 PM, Scott Little wrote: > Our latest build, based on code synced at 2018-09-27T15:28:00 UTC, > built successfully. > > It took three attempts to get ceph built. The first two passes > aborted quickly due to missing packages. The final pass did not > exhibit the 'aclocal: too many loops'**issue. > > The only build I have that exhibited the too many loops error was a > snapshot on 2018-09-20T15:50:40 UTC > > I do have a designer with an older snapshot that seems to hit it > regularly, so I'll work with him and see if we can learn more. > > I think we need more data from the community > - Who's build is failing on ceph with *aclocal: too many loops?* > - Who is building successfully ? > - Who can build only intermittently? > > > > Info to collect for failed builds: - repo sync timestamp > - build command used? > - Was it a new workspace, a cleaned workspace, or a previously used > workspace? - $MY_WORKSPACE/CONTEXT > - $MY_WORKSPACE/build-std.log > - $MY_WORKSPACE/std/results/*/ceph-*/*.log > > For successful builds, same info. Rather than full build logs, I can > settle for: > - grep '\(Success building\|iteration\|building ceph\)' > $MY_WORKSPACE/build-std.log > - grep compute_resources: build-std.log > > > > > > On 18-09-27 02:21 PM, Saul Wold wrote: >> On 09/26/2018 09:16 AM, Scott Little wrote: >>> aclocal 'too many loops' has been popping up sporadically for a week >>> or two now. Possibly 7.5 related. >>> >>> I suspect that there is a build order and/or race condition element >>> to this. It often goes away if you just run build-pkgs a second time. >>> >> I am seeing this failure also, but it does not go away after a second >> rebuild. I have the lastest stx-root (build-tools) with the recent >> patches. >> >> Is this directly related to the fuzz issue or is there something else >> we need to address in CEPH itself. >> >> This is blocking my local build. >> >> Sau! > > > > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi, I did this experiment: I cherry pick all the defuzzed patches and try to build. The result was that the build failing packages now are: libvirt-python-4.7.0-1.tis.1.src.rpm vm-topology-1.0- 1.tis.src.rpm libvirt-4.7.0-1.tis.22.src.rpm which basically is just libvirt, the other ones are just depending on libvirt. Checking the build.log for libvirt: Making all in src make[2]: Entering directory `/builddir/build/BUILD/libvirt-4.7.0/src' BUILDSTDERR: make[2]: *** No rule to make target `keycodemapdb/data/keymaps.csv', needed by `util/virkeycodetable_linux.h'. Stop. make[2]: Leaving directory `/builddir/build/BUILD/libvirt-4.7.0/src' BUILDSTDERR: make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/builddir/build/BUILD/libvirt-4.7.0' BUILDSTDERR: make: *** [all] Error 2 RPM build errors: BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Chutrp (%build) BUILDSTDERR: Macro expanded in comment on line 2067: %{_datadir}/systemtap/tapset/libvirt_probes*.stp BUILDSTDERR: Macro expanded in comment on line 2068: %{_datadir}/systemtap/tapset/libvirt_qemu_probes*.stp BUILDSTDERR: Macro expanded in comment on line 2069: %{_datadir}/systemtap/tapset/libvirt_functions.stp BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Chutrp (%build) Child return code was: 1 So, this is a new failure, but the problem stuck in ceph wasn't present in this execution. Should be noticed that only one attempt of build- pkgs was needed to get here. -Erich On Thu, 2018-09-27 at 11:21 -0700, Saul Wold wrote:
On 09/26/2018 09:16 AM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
I am seeing this failure also, but it does not go away after a second rebuild. I have the lastest stx-root (build-tools) with the recent patches.
Is this directly related to the fuzz issue or is there something else we need to address in CEPH itself.
This is blocking my local build.
Sau!
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2- 3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake- 1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus s
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 2018-09-27 05:48 PM, Cordoba Malibran, Erich wrote:
Hi,
I did this experiment:
I cherry pick all the defuzzed patches and try to build. The result was that the build failing packages now are:
libvirt-python-4.7.0-1.tis.1.src.rpm vm-topology-1.0- 1.tis.src.rpm libvirt-4.7.0-1.tis.22.src.rpm
which basically is just libvirt, the other ones are just depending on libvirt. Checking the build.log for libvirt:
Making all in src make[2]: Entering directory `/builddir/build/BUILD/libvirt-4.7.0/src' BUILDSTDERR: make[2]: *** No rule to make target `keycodemapdb/data/keymaps.csv', needed by `util/virkeycodetable_linux.h'. Stop.
Hi Erich, Check your stx/downloads directory, what version of keycodemapdb do you have in there? You need keycodemapdb-16e5b07.tar.gz which you should have received via the change to stx-tools/centos-mirror-tools/tarball-dl.lst -Jim
make[2]: Leaving directory `/builddir/build/BUILD/libvirt-4.7.0/src' BUILDSTDERR: make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/builddir/build/BUILD/libvirt-4.7.0' BUILDSTDERR: make: *** [all] Error 2 RPM build errors: BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Chutrp (%build) BUILDSTDERR: Macro expanded in comment on line 2067: %{_datadir}/systemtap/tapset/libvirt_probes*.stp BUILDSTDERR: Macro expanded in comment on line 2068: %{_datadir}/systemtap/tapset/libvirt_qemu_probes*.stp BUILDSTDERR: Macro expanded in comment on line 2069: %{_datadir}/systemtap/tapset/libvirt_functions.stp BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Chutrp (%build) Child return code was: 1
So, this is a new failure, but the problem stuck in ceph wasn't present in this execution. Should be noticed that only one attempt of build- pkgs was needed to get here.
-Erich
On Thu, 2018-09-27 at 11:21 -0700, Saul Wold wrote:
On 09/26/2018 09:16 AM, Scott Little wrote:
aclocal 'too many loops' has been popping up sporadically for a week or two now. Possibly 7.5 related.
I suspect that there is a build order and/or race condition element to this. It often goes away if you just run build-pkgs a second time.
I am seeing this failure also, but it does not go away after a second rebuild. I have the lastest stx-root (build-tools) with the recent patches.
Is this directly related to the fuzz issue or is there something else we need to address in CEPH itself.
This is blocking my local build.
Sau!
The second possible element is that build-pkgs is using flags that preserve the mock environment between packages. The goal was to avoid reinstalling required packages that are often common across the packages we build. It was a build time speedup that has been in use for 2- 3 years now without incident.
Google shows that other folks have hit 'aclocal too many loops' as well, and there are suggestions that it might be fixed in a recent update to automake. Scanning the changelog for the latest and greatest from gnu.org doesn't show any obvious fixes addressing this issue.
Scott
On 18-09-26 11:32 AM, Cordoba Malibran, Erich wrote:
BUILDSTDERR: aclocal: error: too many loops BUILDSTDERR: aclocal: Please contact<bug-automake@gnu.org>. BUILDSTDERR: at /usr/share/automake-1.13/Automake/Channels.pm line 662. BUILDSTDERR: Automake::Channels::msg('automake', '', 'too many loops') called at /usr/share/automake- 1.13/Automake/ChannelDefs.pm line 212 BUILDSTDERR: Automake::ChannelDefs::prog_error('too many loops') called at /usr/bin/aclocal line 1187 BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build) BUILDSTDERR: Macro expanded in comment on line 214: %global _libexecdir %{_exec_prefix}/lib BUILDSTDERR: Bad exit status from /var/tmp/rpm-tmp.Sj0E7c (%build)
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus s
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (10)
-
Cordoba Malibran, Erich
-
Dean Troyer
-
Jim Somerville
-
Jones, Bruce E
-
Khalil, Ghada
-
Lin, Shuicheng
-
Rowsell, Brent
-
Saul Wold
-
Scott Little
-
Xie, Cindy