[Starlingx-discuss] download_mirror.sh using bash -e causes early exit
Hi folks, I was just running a fresh download_mirror.sh execution and ran into a problem. I run it as follows, to get timing info and record output: time ./download_mirror.sh 2>&1 | tee dpenney.out The script exited here: --> run: sudo -E yumdownloader -q -C --source python-voluptuous-0.8.9-1.el7 Looking for python-voluptuous-0.8.9-1.el7.src.rpm done!! finish 1st round of RPM downloading successfully! ERROR: -------RPMs missing 6 in yumdownloader with L1 match --------------- real 84m3.238s user 62m49.391s sys 9m46.608s It seemed like an odd place to finish "1st round", so I took a look around there, checking various timestamps to try to see where it quit, and found the last command executed was this line: ## verify all RPMs SRPMs we download for the GPG keys find ./output -type f -name "*.rpm" | xargs rpm -K | grep -i "MISSING KEYS" > ./rpm-gpg-key-missing.txt Because I had no missing keys, the last piped command returned a non-zero status. And since this script is launched with "bash -e" on the first line, and I ran the script directly instead of "bash download_mirror.sh", the script exited at this point. The other download scripts use "bash -e" here as well. Since these are called directly from download_mirror.sh, they could be exiting early silently. Cheers, Don. Don Penney, Developer, Wind River
find ./output -type f -name "*.rpm" | xargs rpm -K | { grep -i "MISSING KEYS" || true } > ./rpm-gpg-key-missing.txt Scott On 18-07-26 03:46 PM, Penney, Don wrote:
Hi folks,
I was just running a fresh download_mirror.sh execution and ran into a problem. I run it as follows, to get timing info and record output:
time ./download_mirror.sh 2>&1 | tee dpenney.out
The script exited here:
--> run: sudo -E yumdownloader -q -C --source python-voluptuous-0.8.9-1.el7
Looking for python-voluptuous-0.8.9-1.el7.src.rpm
done!!
finish 1st round of RPM downloading successfully!
ERROR: -------RPMs missing 6 in yumdownloader with L1 match ---------------
real 84m3.238s
user 62m49.391s
sys 9m46.608s
It seemed like an odd place to finish “1^st round”, so I took a look around there, checking various timestamps to try to see where it quit, and found the last command executed was this line:
## verify all RPMs SRPMs we download for the GPG keys
find ./output -type f -name "*.rpm" | xargs rpm -K | grep -i "MISSING KEYS" > ./rpm-gpg-key-missing.txt
Because I had no missing keys, the last piped command returned a non-zero status. And since this script is launched with “bash -e” on the first line, and I ran the script directly instead of “bash download_mirror.sh”, the script exited at this point.
The other download scripts use “bash -e” here as well. Since these are called directly from download_mirror.sh, they could be exiting early silently.
Cheers,
Don.
*Don Penney*, Developer, *Wind River*
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
We need to drop the -e from these scripts. Proper error checking would need to be added to commands that need it. The failure to access the blizoo repo we've been experiencing caused the 1st stage download to silently abort when the "yum makecache" failed. failure: repodata/repomd.xml from Starlingx-epel.blizoo.mk_epel_7Server_x86_64: [Errno 256] No more mirrors to try. http://epel.blizoo.mk/epel/7Server/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden step #2: start 1st round of downloading RPMs and SRPMs with L1 match criteria... using ./rpms_from_centos_repo.lst as the download name lists [root@fd7ddd238144 localdisk]# cat logs/log_download_rpms_from_3rd_party.txt using ./rpms_from_3rd_parties.lst as the download name lists 2018-07-26_2009 Loaded plugins: fastestmirror, ovl [root@fd7ddd238144 localdisk]# From: Scott Little [mailto:scott.little@windriver.com] Sent: Thursday, July 26, 2018 3:53 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] download_mirror.sh using bash -e causes early exit find ./output -type f -name "*.rpm" | xargs rpm -K | { grep -i "MISSING KEYS" || true } > ./rpm-gpg-key-missing.txt Scott On 18-07-26 03:46 PM, Penney, Don wrote: Hi folks, I was just running a fresh download_mirror.sh execution and ran into a problem. I run it as follows, to get timing info and record output: time ./download_mirror.sh 2>&1 | tee dpenney.out The script exited here: --> run: sudo -E yumdownloader -q -C --source python-voluptuous-0.8.9-1.el7 Looking for python-voluptuous-0.8.9-1.el7.src.rpm done!! finish 1st round of RPM downloading successfully! ERROR: -------RPMs missing 6 in yumdownloader with L1 match --------------- real 84m3.238s user 62m49.391s sys 9m46.608s It seemed like an odd place to finish "1st round", so I took a look around there, checking various timestamps to try to see where it quit, and found the last command executed was this line: ## verify all RPMs SRPMs we download for the GPG keys find ./output -type f -name "*.rpm" | xargs rpm -K | grep -i "MISSING KEYS" > ./rpm-gpg-key-missing.txt Because I had no missing keys, the last piped command returned a non-zero status. And since this script is launched with "bash -e" on the first line, and I ran the script directly instead of "bash download_mirror.sh", the script exited at this point. The other download scripts use "bash -e" here as well. Since these are called directly from download_mirror.sh, they could be exiting early silently. Cheers, Don. Don Penney, Developer, Wind River _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
There are 3 repos in StarlingX_3rd.repo that are unreachable currently from our site, due to firewall concerns. Disabling these repos allows me to complete the download_mirror.sh $ git diff StarlingX_3rd.repo diff --git a/centos-mirror-tools/StarlingX_3rd.repo b/centos-mirror-tools/StarlingX_3rd.repo index d9c5b37..d669784 100644 --- a/centos-mirror-tools/StarlingX_3rd.repo +++ b/centos-mirror-tools/StarlingX_3rd.repo @@ -11,17 +11,17 @@ enabled=1 [Starlingx-epel.blizoo.mk_epel_7Server_x86_64] name=Starlingx-epel.blizoo.mk_epel_7Server_x86_64 - epel.blizoo.mk_epel_7Server_x86_64 baseurl=http://epel.blizoo.mk/epel/7Server/x86_64 -enabled=1 +enabled=0 [Starlingx-epel.blizoo.mk_epel_7_SRPMS] name=Starlingx-epel.blizoo.mk_epel_7_SRPMS - epel.blizoo.mk_epel_7_SRPMS baseurl=http://epel.blizoo.mk/epel/7/SRPMS -enabled=1 +enabled=0 [Starlingx-ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64] name=Starlingx-ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64 - ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64 baseurl=http://ftp.iij.ad.jp/pub/linux/centos/7/rt/x86_64 -enabled=1 +enabled=0 [Starlingx-ftp.riken.jp_Linux_cern_centos_7_rt_Sources] name=Starlingx-ftp.riken.jp_Linux_cern_centos_7_rt_Sources - ftp.riken.jp_Linux_cern_centos_7_rt_Sources Without these disabled, the "yum makecache" aborts with an error. With them disabled, the only missing RPMs I have now are those covered by this pending update: https://review.openstack.org/#/c/586393/ From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Friday, July 27, 2018 8:56 AM To: Little, Scott; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] download_mirror.sh using bash -e causes early exit We need to drop the -e from these scripts. Proper error checking would need to be added to commands that need it. The failure to access the blizoo repo we've been experiencing caused the 1st stage download to silently abort when the "yum makecache" failed. failure: repodata/repomd.xml from Starlingx-epel.blizoo.mk_epel_7Server_x86_64: [Errno 256] No more mirrors to try. http://epel.blizoo.mk/epel/7Server/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden step #2: start 1st round of downloading RPMs and SRPMs with L1 match criteria... using ./rpms_from_centos_repo.lst as the download name lists [root@fd7ddd238144 localdisk]# cat logs/log_download_rpms_from_3rd_party.txt using ./rpms_from_3rd_parties.lst as the download name lists 2018-07-26_2009 Loaded plugins: fastestmirror, ovl [root@fd7ddd238144 localdisk]# From: Scott Little [mailto:scott.little@windriver.com] Sent: Thursday, July 26, 2018 3:53 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] download_mirror.sh using bash -e causes early exit find ./output -type f -name "*.rpm" | xargs rpm -K | { grep -i "MISSING KEYS" || true } > ./rpm-gpg-key-missing.txt Scott On 18-07-26 03:46 PM, Penney, Don wrote: Hi folks, I was just running a fresh download_mirror.sh execution and ran into a problem. I run it as follows, to get timing info and record output: time ./download_mirror.sh 2>&1 | tee dpenney.out The script exited here: --> run: sudo -E yumdownloader -q -C --source python-voluptuous-0.8.9-1.el7 Looking for python-voluptuous-0.8.9-1.el7.src.rpm done!! finish 1st round of RPM downloading successfully! ERROR: -------RPMs missing 6 in yumdownloader with L1 match --------------- real 84m3.238s user 62m49.391s sys 9m46.608s It seemed like an odd place to finish "1st round", so I took a look around there, checking various timestamps to try to see where it quit, and found the last command executed was this line: ## verify all RPMs SRPMs we download for the GPG keys find ./output -type f -name "*.rpm" | xargs rpm -K | grep -i "MISSING KEYS" > ./rpm-gpg-key-missing.txt Because I had no missing keys, the last piped command returned a non-zero status. And since this script is launched with "bash -e" on the first line, and I ran the script directly instead of "bash download_mirror.sh", the script exited at this point. The other download scripts use "bash -e" here as well. Since these are called directly from download_mirror.sh, they could be exiting early silently. Cheers, Don. Don Penney, Developer, Wind River _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
This is great - can somebody merge this patch (https://review.openstack.org/#/c/586393/)? I saw Intel's auto-build log also shows that these are the only files still missing: error: failed packages: python2-rpm-macros-3-20.el7.noarch.rpm python2-sphinxcontrib-httpdomain-1.5.0-3.el7.noarch.rpm python-rpm-macros-3-20.el7.noarch.rpm python-srpm-macros-3-20.el7.noarch.rpm thx. - cindy From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Friday, July 27, 2018 10:23 PM To: Penney, Don <Don.Penney@windriver.com>; Little, Scott <Scott.Little@windriver.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] download_mirror.sh using bash -e causes early exit There are 3 repos in StarlingX_3rd.repo that are unreachable currently from our site, due to firewall concerns. Disabling these repos allows me to complete the download_mirror.sh $ git diff StarlingX_3rd.repo diff --git a/centos-mirror-tools/StarlingX_3rd.repo b/centos-mirror-tools/StarlingX_3rd.repo index d9c5b37..d669784 100644 --- a/centos-mirror-tools/StarlingX_3rd.repo +++ b/centos-mirror-tools/StarlingX_3rd.repo @@ -11,17 +11,17 @@ enabled=1 [Starlingx-epel.blizoo.mk_epel_7Server_x86_64] name=Starlingx-epel.blizoo.mk_epel_7Server_x86_64 - epel.blizoo.mk_epel_7Server_x86_64 baseurl=http://epel.blizoo.mk/epel/7Server/x86_64 -enabled=1 +enabled=0 [Starlingx-epel.blizoo.mk_epel_7_SRPMS] name=Starlingx-epel.blizoo.mk_epel_7_SRPMS - epel.blizoo.mk_epel_7_SRPMS baseurl=http://epel.blizoo.mk/epel/7/SRPMS -enabled=1 +enabled=0 [Starlingx-ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64] name=Starlingx-ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64 - ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64<ftp://ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64> baseurl=http://ftp.iij.ad.jp/pub/linux/centos/7/rt/x86_64 -enabled=1 +enabled=0 [Starlingx-ftp.riken.jp_Linux_cern_centos_7_rt_Sources] name=Starlingx-ftp.riken.jp_Linux_cern_centos_7_rt_Sources - ftp.riken.jp_Linux_cern_centos_7_rt_Sources<ftp://ftp.riken.jp_Linux_cern_centos_7_rt_Sources> Without these disabled, the "yum makecache" aborts with an error. With them disabled, the only missing RPMs I have now are those covered by this pending update: https://review.openstack.org/#/c/586393/ From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Friday, July 27, 2018 8:56 AM To: Little, Scott; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] download_mirror.sh using bash -e causes early exit We need to drop the -e from these scripts. Proper error checking would need to be added to commands that need it. The failure to access the blizoo repo we've been experiencing caused the 1st stage download to silently abort when the "yum makecache" failed. failure: repodata/repomd.xml from Starlingx-epel.blizoo.mk_epel_7Server_x86_64: [Errno 256] No more mirrors to try. http://epel.blizoo.mk/epel/7Server/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden step #2: start 1st round of downloading RPMs and SRPMs with L1 match criteria... using ./rpms_from_centos_repo.lst as the download name lists [root@fd7ddd238144 localdisk]# cat logs/log_download_rpms_from_3rd_party.txt using ./rpms_from_3rd_parties.lst as the download name lists 2018-07-26_2009 Loaded plugins: fastestmirror, ovl [root@fd7ddd238144 localdisk]# From: Scott Little [mailto:scott.little@windriver.com] Sent: Thursday, July 26, 2018 3:53 PM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] download_mirror.sh using bash -e causes early exit find ./output -type f -name "*.rpm" | xargs rpm -K | { grep -i "MISSING KEYS" || true } > ./rpm-gpg-key-missing.txt Scott On 18-07-26 03:46 PM, Penney, Don wrote: Hi folks, I was just running a fresh download_mirror.sh execution and ran into a problem. I run it as follows, to get timing info and record output: time ./download_mirror.sh 2>&1 | tee dpenney.out The script exited here: --> run: sudo -E yumdownloader -q -C --source python-voluptuous-0.8.9-1.el7 Looking for python-voluptuous-0.8.9-1.el7.src.rpm done!! finish 1st round of RPM downloading successfully! ERROR: -------RPMs missing 6 in yumdownloader with L1 match --------------- real 84m3.238s user 62m49.391s sys 9m46.608s It seemed like an odd place to finish "1st round", so I took a look around there, checking various timestamps to try to see where it quit, and found the last command executed was this line: ## verify all RPMs SRPMs we download for the GPG keys find ./output -type f -name "*.rpm" | xargs rpm -K | grep -i "MISSING KEYS" > ./rpm-gpg-key-missing.txt Because I had no missing keys, the last piped command returned a non-zero status. And since this script is launched with "bash -e" on the first line, and I ran the script directly instead of "bash download_mirror.sh", the script exited at this point. The other download scripts use "bash -e" here as well. Since these are called directly from download_mirror.sh, they could be exiting early silently. Cheers, Don. Don Penney, Developer, Wind River _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Fri, Jul 27, 2018 at 9:45 AM, Xie, Cindy <cindy.xie@intel.com> wrote:
This is great – can somebody merge this patch (https://review.openstack.org/#/c/586393/)? I saw Intel’s auto-build log also shows that these are the only files still missing:
I just did... dt -- Dean Troyer dtroyer@gmail.com
participants (4)
-
Dean Troyer
-
Penney, Don
-
Scott Little
-
Xie, Cindy