[Starlingx-discuss] Download packages issues

[Rowsell, Brent]

Comments inline 

-----Original Message-----
From: Cordoba Malibran, Erich [mailto:erich.cordoba.malibran at intel.com] 
Sent: Wednesday, July 11, 2018 3:18 PM
To: Little, Scott <Scott.Little at windriver.com>; starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] Download packages issues

Hi Scott.

Indeed, the mirror downloading is very fragile, actually some of the packages can't be downloaded with yumdownloader and direct wget is used. Every time that a package has a new patch a new release version is generated and our mirror is broken.

Not sure about hosting our own mirror, although it will ease our process in a short term, the path to be able to run with latest CentOS (or another distro) will be harder to reach. So, this could be a first step going there. Probably, in the future we can remove from the build system the dependency on specific srpm versions (1.2.3-10 <- moving from 10 to 11 shouldn't be an issue) and just update patches when is necessary.

[BR] We are not going to be able to remove dependencies on specific rpm versions for packages with patches against them which is currently a significant number. We need predictable builds and finding out at build time that patches don’t apply is not really acceptable. 
Once this has been eliminated/significantly reduced then perhaps that becomes an option. 

Anyway, in this particular problem, kernel-3.10.0-862.3.2.el7.src.rpm was added here[0] but the proper repository wasn't added. The fix for this was already merged[1]

The issue about not finding git was fixed last week, so maybe a git pull may help.

-Erich

[0] https://review.openstack.org/#/c/580689/
[1] https://review.openstack.org/#/c/581471/


From: Scott Little <scott.little at windriver.com>
Date: Wednesday, July 11, 2018 at 9:09 AM
To: "starlingx-discuss at lists.starlingx.io" <starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] Download packages issues

The 'bash download_mirror.sh' step seems to be very fragile.  e.g. this morning I tried to start fresh and it failed at several points.
   ------ using kernel-3.10.0-862.3.2.el7 to search kernel-3.10.0-862.3.2.el7.src.rpm ------
   --> run: sudo -E yumdownloader -q -C --source kernel-3.10.0-862.3.2.el7
   Looking for kernel-3.10.0-862.3.2.el7.src.rpm
   No source RPM found for kernel-3.10.0-862.3.2.el7.x86_64
   Nothing to download


    ------ using epel-rpm-macros-7-18 to search epel-rpm-macros-7-18.noarch.rpm ------
   --> run: sudo -E yumdownloader -q -C epel-rpm-macros-7-18 --archlist=noarch,x86_64
   Looking for epel-rpm-macros-7-18.noarch.rpm...
   No Match for argument epel-rpm-macros-7-18
   Nothing to download

and the tarball downloads failed on.
   --2018-07-11 13:13:28--  http://git.infradead.org/users/jjs/linux-tpmdd.git/snapshot/fd5c78694f3f1c875e293de7a641ba8a3d60d00d.tar.g
   Resolving git.infradead.org (git.infradead.org)... 198.137.202.133, 2607:7c80:54:e::133
   Connecting to git.infradead.org (git.infradead.org)|198.137.202.133|:80... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: unspecified [application/x-gzip]
   Saving to: 'fd5c78694f3f1c875e293de7a641ba8a3d60d00d.tar.gz'

   2018-07-11 13:14:34 (0.00 B/s) - 'fd5c78694f3f1c875e293de7a641ba8a3d60d00d.tar.gz' saved [0]

   tar: This does not look like a tar archive

   gzip: stdin: unexpected end of file
   tar: Child returned status 1
   tar: Error is not recoverable: exiting now
   mv: cannot stat 'linux-tpmdd-fd5c786/drivers/char/tpm': No such file or directory
   tar: tpm: Cannot stat: No such file or directory
   tar: Exiting with failure status due to previous errors
   /localdisk
   tss2-930.tar.gz
   /localdisk/output/stx-r1/CentOS/pike/downloads /localdisk
   ./tarball-dl.sh: line 156: git: command not found
   ./tarball-dl.sh: line 157: pushd: ibmtpm20tss-tss: No such file or directory
   ./tarball-dl.sh: line 158: git: command not found
   /localdisk
   mv: cannot stat 'ibmtpm20tss-tss': No such file or directory
   tar: tss2-930: Cannot stat: No such file or directory
   tar: Exiting with failure status due to previous errors
   ./tarball-dl.sh: line 165: popd: directory stack empty

Is this really going to be our default operating mode?   Never knowing, day by day, when the next package will be dropped from an upstream source.  Is it possible for strarlingX to host it's own mirrors... of at least the needed subset of packages?
Scott


_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss