[Starlingx-discuss] Download packages issues

[Cordoba Malibran, Erich]

Hi Scott.

Indeed, the mirror downloading is very fragile, actually some of the packages can't be downloaded 
with yumdownloader and direct wget is used. Every time that a package has a new patch a new 
release version is generated and our mirror is broken.

Not sure about hosting our own mirror, although it will ease our process in a short term, the path
to be able to run with latest CentOS (or another distro) will be harder to reach. So, this could be a first step
going there. Probably, in the future we can remove from the build system the dependency on specific srpm
versions (1.2.3-10 <- moving from 10 to 11 shouldn't be an issue) and just update patches when is necessary.

Anyway, in this particular problem, kernel-3.10.0-862.3.2.el7.src.rpm was added here[0] but the proper
repository wasn't added. The fix for this was already merged[1]

The issue about not finding git was fixed last week, so maybe a git pull may help.

-Erich

[0] https://review.openstack.org/#/c/580689/
[1] https://review.openstack.org/#/c/581471/


From: Scott Little <scott.little at windriver.com>
Date: Wednesday, July 11, 2018 at 9:09 AM
To: "starlingx-discuss at lists.starlingx.io" <starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] Download packages issues

The 'bash download_mirror.sh' step seems to be very fragile.  e.g. this morning I tried to start fresh and it failed at several points.
   ------ using kernel-3.10.0-862.3.2.el7 to search kernel-3.10.0-862.3.2.el7.src.rpm ------
   --> run: sudo -E yumdownloader -q -C --source kernel-3.10.0-862.3.2.el7
   Looking for kernel-3.10.0-862.3.2.el7.src.rpm
   No source RPM found for kernel-3.10.0-862.3.2.el7.x86_64
   Nothing to download


    ------ using epel-rpm-macros-7-18 to search epel-rpm-macros-7-18.noarch.rpm ------
   --> run: sudo -E yumdownloader -q -C epel-rpm-macros-7-18 --archlist=noarch,x86_64
   Looking for epel-rpm-macros-7-18.noarch.rpm...
   No Match for argument epel-rpm-macros-7-18
   Nothing to download

and the tarball downloads failed on.
   --2018-07-11 13:13:28--  http://git.infradead.org/users/jjs/linux-tpmdd.git/snapshot/fd5c78694f3f1c875e293de7a641ba8a3d60d00d.tar.g
   Resolving git.infradead.org (git.infradead.org)... 198.137.202.133, 2607:7c80:54:e::133
   Connecting to git.infradead.org (git.infradead.org)|198.137.202.133|:80... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: unspecified [application/x-gzip]
   Saving to: 'fd5c78694f3f1c875e293de7a641ba8a3d60d00d.tar.gz'

   2018-07-11 13:14:34 (0.00 B/s) - 'fd5c78694f3f1c875e293de7a641ba8a3d60d00d.tar.gz' saved [0]

   tar: This does not look like a tar archive

   gzip: stdin: unexpected end of file
   tar: Child returned status 1
   tar: Error is not recoverable: exiting now
   mv: cannot stat 'linux-tpmdd-fd5c786/drivers/char/tpm': No such file or directory
   tar: tpm: Cannot stat: No such file or directory
   tar: Exiting with failure status due to previous errors
   /localdisk
   tss2-930.tar.gz
   /localdisk/output/stx-r1/CentOS/pike/downloads /localdisk
   ./tarball-dl.sh: line 156: git: command not found
   ./tarball-dl.sh: line 157: pushd: ibmtpm20tss-tss: No such file or directory
   ./tarball-dl.sh: line 158: git: command not found
   /localdisk
   mv: cannot stat 'ibmtpm20tss-tss': No such file or directory
   tar: tss2-930: Cannot stat: No such file or directory
   tar: Exiting with failure status due to previous errors
   ./tarball-dl.sh: line 165: popd: directory stack empty

Is this really going to be our default operating mode?   Never knowing, day by day, when the next package will be dropped from an upstream source.  Is it possible for strarlingX to host it's own mirrors... of at least the needed subset of packages?
Scott