[Starlingx-discuss] download_mirror.sh using bash -e causes early exit

[Xie, Cindy]

This is great - can somebody merge this patch (https://review.openstack.org/#/c/586393/)? I saw Intel's auto-build log also shows that these are the only files still missing:

error: failed packages: python2-rpm-macros-3-20.el7.noarch.rpm
python2-sphinxcontrib-httpdomain-1.5.0-3.el7.noarch.rpm
python-rpm-macros-3-20.el7.noarch.rpm
python-srpm-macros-3-20.el7.noarch.rpm

thx. - cindy

From: Penney, Don [mailto:Don.Penney at windriver.com]
Sent: Friday, July 27, 2018 10:23 PM
To: Penney, Don <Don.Penney at windriver.com>; Little, Scott <Scott.Little at windriver.com>; starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] download_mirror.sh using bash -e causes early exit

There are 3 repos in StarlingX_3rd.repo that are unreachable currently from our site, due to firewall concerns. Disabling these repos allows me to complete the download_mirror.sh

$ git diff StarlingX_3rd.repo
diff --git a/centos-mirror-tools/StarlingX_3rd.repo b/centos-mirror-tools/StarlingX_3rd.repo
index d9c5b37..d669784 100644
--- a/centos-mirror-tools/StarlingX_3rd.repo
+++ b/centos-mirror-tools/StarlingX_3rd.repo
@@ -11,17 +11,17 @@ enabled=1
[Starlingx-epel.blizoo.mk_epel_7Server_x86_64]
name=Starlingx-epel.blizoo.mk_epel_7Server_x86_64 - epel.blizoo.mk_epel_7Server_x86_64
baseurl=http://epel.blizoo.mk/epel/7Server/x86_64
-enabled=1
+enabled=0

[Starlingx-epel.blizoo.mk_epel_7_SRPMS]
name=Starlingx-epel.blizoo.mk_epel_7_SRPMS - epel.blizoo.mk_epel_7_SRPMS
baseurl=http://epel.blizoo.mk/epel/7/SRPMS
-enabled=1
+enabled=0

[Starlingx-ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64]
name=Starlingx-ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64 - ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64<ftp://ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64>
baseurl=http://ftp.iij.ad.jp/pub/linux/centos/7/rt/x86_64
-enabled=1
+enabled=0

[Starlingx-ftp.riken.jp_Linux_cern_centos_7_rt_Sources]
name=Starlingx-ftp.riken.jp_Linux_cern_centos_7_rt_Sources - ftp.riken.jp_Linux_cern_centos_7_rt_Sources<ftp://ftp.riken.jp_Linux_cern_centos_7_rt_Sources>


Without these disabled, the "yum makecache" aborts with an error. With them disabled, the only missing RPMs I have now are those covered by this pending update:

https://review.openstack.org/#/c/586393/




From: Penney, Don [mailto:Don.Penney at windriver.com]
Sent: Friday, July 27, 2018 8:56 AM
To: Little, Scott; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] download_mirror.sh using bash -e causes early exit

We need to drop the -e from these scripts. Proper error checking would need to be added to commands that need it.

The failure to access the blizoo repo we've been experiencing caused the 1st stage download to silently abort when the "yum makecache" failed.

failure: repodata/repomd.xml from Starlingx-epel.blizoo.mk_epel_7Server_x86_64: [Errno 256] No more mirrors to try.
http://epel.blizoo.mk/epel/7Server/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden
step #2: start 1st round of downloading RPMs and SRPMs with L1 match criteria...
using ./rpms_from_centos_repo.lst as the download name lists


[root at fd7ddd238144 localdisk]# cat logs/log_download_rpms_from_3rd_party.txt
using ./rpms_from_3rd_parties.lst as the download name lists
2018-07-26_2009
Loaded plugins: fastestmirror, ovl
[root at fd7ddd238144 localdisk]#



From: Scott Little [mailto:scott.little at windriver.com]
Sent: Thursday, July 26, 2018 3:53 PM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] download_mirror.sh using bash -e causes early exit

find ./output -type f -name "*.rpm" | xargs rpm -K | { grep -i "MISSING KEYS" || true } > ./rpm-gpg-key-missing.txt

Scott

On 18-07-26 03:46 PM, Penney, Don wrote:
Hi folks,

I was just running a fresh download_mirror.sh execution and ran into a problem. I run it as follows, to get timing info and record output:
time ./download_mirror.sh 2>&1 | tee dpenney.out

The script exited here:
--> run: sudo -E yumdownloader -q -C --source python-voluptuous-0.8.9-1.el7
Looking for python-voluptuous-0.8.9-1.el7.src.rpm
done!!
finish 1st round of RPM downloading successfully!
ERROR:  -------RPMs missing 6 in yumdownloader with L1 match ---------------

real    84m3.238s
user    62m49.391s
sys     9m46.608s

It seemed like an odd place to finish "1st round", so I took a look around there, checking various timestamps to try to see where it quit, and found the last command executed was this line:

## verify all RPMs SRPMs we download for the GPG keys
find ./output -type f -name "*.rpm" | xargs rpm -K | grep -i "MISSING KEYS" > ./rpm-gpg-key-missing.txt

Because I had no missing keys, the last piped command returned a non-zero status. And since this script is launched with "bash -e" on the first line, and I ran the script directly instead of "bash download_mirror.sh", the script exited at this point.

The other download scripts use "bash -e" here as well. Since these are called directly from download_mirror.sh, they could be exiting early silently.

Cheers,
Don.

Don Penney, Developer, Wind River



_______________________________________________

Starlingx-discuss mailing list

Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>

http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20180727/5f784d74/attachment-0001.html>