[Starlingx-discuss] StarlingX: Tar Compressed Files Repository Restructuring

Thu Jul 26 04:10:50 UTC 2018

On 07/24/2018 06:51 PM, Liu, ZhipengS wrote:
> Hi Abraham,
> 
> Just 2 proposals.
> 1) Every time, when I run download_mirror, it will always download mvn which will take around 1~2 hours.
> Could you add a check in the script that if mvn.repo.tgz exists, do not try to download it again. Same for other package
> 2) When download list changed, the script only downloads added packages and also removes the ones of old versions that we do not used anymore.
> 
I was thinking we could try to incorporate somekind of checksum 
information, in order to validate the downloads are correct and complete.

more below
> Zhipeng
> 
> -----Original Message-----
> From: Arce Moreno, Abraham [mailto:abraham.arce.moreno at intel.com]
> Sent: 2018年7月24日 7:36
> To: starlingx-discuss at lists.starlingx.io
> Subject: [Starlingx-discuss] StarlingX: Tar Compressed Files Repository Restructuring
> 
> The build of StarlingX relies, besides RPM Binaries and Sources, in a collection of packages in the form of Tar Compressed files. Initial review already merged can be found under Gerrit Review "StarlingX: Tar Compressed files repository" [0]
> 
> We have received several feedback about its content and it is time to gather all of it including security related discussion to restructure the way it is handled, if needed.
> Please take some time to read, review and send back comments.
> The complete analysis of these required packages is found at [1].
> 
> [ Call Sequence ]
> 
> The call sequence under CentOS Mirror Tools [2] is as follows ending in executing our tarball-dl.sh:
> 
> 1 download_mirror.sh
>     1.1 dl_rpms.sh	
>     1.2 dl_other_from_centos_repo.sh
>     1.3 tarball-dl.sh
> 
> Open Questions:
> - Should we keep this script being called from download_mirror.sh?
> - Should we keep this file being written as shell script?
> - Any comment based on midterm / long term changes?
> - Any other comment?
> 
So the script is doing 3 things, I think there should be more checking 
as to what exists and what's needed.  The RPM fetching can be done 
independant of the tarball, so that we can use a query of the repos 
instead of full download.

Another thought is to provide the option to point this at somekind of 
company based mirror, so that each larger company could keep an internal 
mirror, but smaller organizations will hit the external sources.

> [ List of Packages ]
> 
> The script tarball-dl.sh [3] takes a text file tarball-dl.lst [4] as argument which contains 3 columns:
> 
> - Column 1, name of package including extension as it is referenced by the build
>    system.
> - Column 2, name of the directory path after it is decompressed as it is referenced
>    in the build system.
> - Column 3, the URL for the package download.
> 
> Open Questions:
> - We use "#" to separate columns however in our review we have suggestions to use other
>    characters such as "|" including also change to CSV format, what do you think?
> - Any other comment?
> 
> [ List of Packages: Column 1 ]
> 
> Our build system have references to the name of the file, the file name might include:
>    - name of the package
>    - version
>    - commit id
> 
> Some examples?
>    - python-cephclient-v0.1.0.5.tar.gz
>    - gnulib-ffc927e.tar.gz
>    - puppet-boolean-22b726dd78b0a60a224cc7054aebbf28e9306f62.tar.gz
> 
> A special character "!" is used at the beginning of 6 package file names denoting they require special handling is required: additional steps besides the common process including but not limited to remove text from some files, use only one directory, etc:
> 
>      - integrity-kmod-668a8270.tar.gz
>      - mariadb-10.1.28.tar.gz
>      - mvn.repo.tgz
>      - qat1.7.upstream.l.1.0.3-42.tar.gz
>      - tpm-kmod-668a8270.tar.gz
>      - tss2-930.tar.gz
> 
> Open Questions:
> - Any specific method to handle these 6 packages?
> - Any other comment?
> 
This one is harder, since each of these is a different special case to 
deal with git or multiple downloads (mvn). Short of creating some sort 
of "plugin" mechanism that handles each special case.

> [ List of Packages: Column 2 ]
> 
> Our build system uses defined paths after the Tarball package filename is decompressed, see Columns H " 'Directory Name Cgcs-Dl" and Column I " 'Directory Name tarball-dl.sh" for a
> comparison:
> 
> - It could be the same after it is decompressed from the original file
>    - dpkg_1.18.24.tar.xz#dpkg-1.18.24
> - It could use a different name after it is decompressed from the original file
>    - !integrity-kmod-668a8270.tar.gz#integrity
>    - puppet-dnsmasq-cff07e90890662972c97684a2baee964f68ff3ed.tar.gz#packstack/puppet/modules/dnsmasq/
> 
> Open Questions:
> - Do we want to have predefined paths or should we targeted default paths as given by the original file names?
> - Any other comment?
> 
I have not looked deeper, but are the predefined paths required for the 
way build system is currently looking for files?

> [ List of Packages: Column 3 ]
> 
> Nothing specific here but hardcoded URLs.
> 
> Open Questions:
> - Use wget Vs curl?

I would lean towards curl as it has better protocol support and supports 
socks for proxy activity.

> - Any other method to get the sources of the packages? Is it possible to migrate to source RPMs?
> 
> [ Security ]
> 
> Open Discussion! We went through an initial package integrity verification, see column F "GPG Key" but only 6 packages got their PGP key available in their official repository source.
> 
> Open Questions:
> - What is the best way to approach?
If there is a GPG key, then I think we should verify it, we could also 
validate that what we currently have does not need to be updated (see above)

> - Specific question. Can we say Tarballs coming from Git sources are naturally verified?
I think that makes sense, the problem with those tarballs is that we 
can't have a checksum on them since they are created on the fly and the 
time stamps change.

> - Any other comment?
> 
> [0] https://review.openstack.org/#/c/577953/
> [1] https://ethercalc.openstack.org/615btsy7i8k6
> [2] https://git.openstack.org/cgit/openstack/stx-tools/tree/centos-mirror-tools
> [3] https://git.openstack.org/cgit/openstack/stx-tools/tree/centos-mirror-tools/tarball-dl.sh
> [4] https://git.openstack.org/cgit/openstack/stx-tools/tree/centos-mirror-tools/tarball-dl.lst
> 
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> 