[Starlingx-discuss] StarlingX: Tar Compressed Files Repository Restructuring

[Arce Moreno, Abraham]

The build of StarlingX relies, besides RPM Binaries and Sources, in a collection of packages 
in the form of Tar Compressed files. Initial review already merged can be found under 
Gerrit Review "StarlingX: Tar Compressed files repository" [0]

We have received several feedback about its content and it is time to gather all of it
including security related discussion to restructure the way it is handled, if needed. 
Please take some time to read, review and send back comments. 
The complete analysis of these required packages is found at [1].

[ Call Sequence ]

The call sequence under CentOS Mirror Tools [2] is as follows ending in executing
our tarball-dl.sh:

1 download_mirror.sh
   1.1 dl_rpms.sh	
   1.2 dl_other_from_centos_repo.sh
   1.3 tarball-dl.sh

Open Questions:
- Should we keep this script being called from download_mirror.sh?
- Should we keep this file being written as shell script?
- Any comment based on midterm / long term changes?
- Any other comment?

[ List of Packages ]

The script tarball-dl.sh [3] takes a text file tarball-dl.lst [4] as argument which
contains 3 columns:

- Column 1, name of package including extension as it is referenced by the build
  system.
- Column 2, name of the directory path after it is decompressed as it is referenced
  in the build system.
- Column 3, the URL for the package download.

Open Questions:
- We use "#" to separate columns however in our review we have suggestions to use other
  characters such as "|" including also change to CSV format, what do you think?
- Any other comment?

[ List of Packages: Column 1 ]

Our build system have references to the name of the file, the file name 
might include:
  - name of the package
  - version
  - commit id

Some examples?
  - python-cephclient-v0.1.0.5.tar.gz
  - gnulib-ffc927e.tar.gz
  - puppet-boolean-22b726dd78b0a60a224cc7054aebbf28e9306f62.tar.gz

A special character "!" is used at the beginning of 6 package file names denoting they 
require special handling is required: additional steps besides the common process
including but not limited to remove text from some files, use only one directory, etc:

    - integrity-kmod-668a8270.tar.gz
    - mariadb-10.1.28.tar.gz
    - mvn.repo.tgz
    - qat1.7.upstream.l.1.0.3-42.tar.gz
    - tpm-kmod-668a8270.tar.gz
    - tss2-930.tar.gz

Open Questions:
- Any specific method to handle these 6 packages?
- Any other comment? 

[ List of Packages: Column 2 ]

Our build system uses defined paths after the Tarball package filename is decompressed, see
Columns H " 'Directory Name Cgcs-Dl" and Column I " 'Directory Name tarball-dl.sh" for a 
comparison:

- It could be the same after it is decompressed from the original file
  - dpkg_1.18.24.tar.xz#dpkg-1.18.24
- It could use a different name after it is decompressed from the original file
  - !integrity-kmod-668a8270.tar.gz#integrity
  - puppet-dnsmasq-cff07e90890662972c97684a2baee964f68ff3ed.tar.gz#packstack/puppet/modules/dnsmasq/

Open Questions:
- Do we want to have predefined paths or should we targeted default paths as given by the original file names?
- Any other comment?

[ List of Packages: Column 3 ]

Nothing specific here but hardcoded URLs.

Open Questions:
- Use wget Vs curl?
- Any other method to get the sources of the packages? Is it possible to migrate to source RPMs?

[ Security ]

Open Discussion! We went through an initial package integrity verification, see column F "GPG Key" but only 6 packages
got their PGP key available in their official repository source.

Open Questions:
- What is the best way to approach?
- Specific question. Can we say Tarballs coming from Git sources are naturally verified?
- Any other comment?

[0] https://review.openstack.org/#/c/577953/
[1] https://ethercalc.openstack.org/615btsy7i8k6
[2] https://git.openstack.org/cgit/openstack/stx-tools/tree/centos-mirror-tools
[3] https://git.openstack.org/cgit/openstack/stx-tools/tree/centos-mirror-tools/tarball-dl.sh
[4] https://git.openstack.org/cgit/openstack/stx-tools/tree/centos-mirror-tools/tarball-dl.lst