[Starlingx-discuss] StarlingX: Tar Compressed Files Repository Restructuring

[Liu, ZhipengS]

Hi Abraham,

Just 2 proposals. 
1) Every time, when I run download_mirror, it will always download mvn which will take around 1~2 hours.
Could you add a check in the script that if mvn.repo.tgz exists, do not try to download it again. Same for other packages.
2) When download list changed, the script only downloads added packages and also removes the ones of old versions that we do not used anymore.

Zhipeng

-----Original Message-----
From: Arce Moreno, Abraham [mailto:abraham.arce.moreno at intel.com] 
Sent: 2018年7月24日 7:36
To: starlingx-discuss at lists.starlingx.io
Subject: [Starlingx-discuss] StarlingX: Tar Compressed Files Repository Restructuring

The build of StarlingX relies, besides RPM Binaries and Sources, in a collection of packages in the form of Tar Compressed files. Initial review already merged can be found under Gerrit Review "StarlingX: Tar Compressed files repository" [0]

We have received several feedback about its content and it is time to gather all of it including security related discussion to restructure the way it is handled, if needed. 
Please take some time to read, review and send back comments. 
The complete analysis of these required packages is found at [1].

[ Call Sequence ]

The call sequence under CentOS Mirror Tools [2] is as follows ending in executing our tarball-dl.sh:

1 download_mirror.sh
   1.1 dl_rpms.sh	
   1.2 dl_other_from_centos_repo.sh
   1.3 tarball-dl.sh

Open Questions:
- Should we keep this script being called from download_mirror.sh?
- Should we keep this file being written as shell script?
- Any comment based on midterm / long term changes?
- Any other comment?

[ List of Packages ]

The script tarball-dl.sh [3] takes a text file tarball-dl.lst [4] as argument which contains 3 columns:

- Column 1, name of package including extension as it is referenced by the build
  system.
- Column 2, name of the directory path after it is decompressed as it is referenced
  in the build system.
- Column 3, the URL for the package download.

Open Questions:
- We use "#" to separate columns however in our review we have suggestions to use other
  characters such as "|" including also change to CSV format, what do you think?
- Any other comment?

[ List of Packages: Column 1 ]

Our build system have references to the name of the file, the file name might include:
  - name of the package
  - version
  - commit id

Some examples?
  - python-cephclient-v0.1.0.5.tar.gz
  - gnulib-ffc927e.tar.gz
  - puppet-boolean-22b726dd78b0a60a224cc7054aebbf28e9306f62.tar.gz

A special character "!" is used at the beginning of 6 package file names denoting they require special handling is required: additional steps besides the common process including but not limited to remove text from some files, use only one directory, etc:

    - integrity-kmod-668a8270.tar.gz
    - mariadb-10.1.28.tar.gz
    - mvn.repo.tgz
    - qat1.7.upstream.l.1.0.3-42.tar.gz
    - tpm-kmod-668a8270.tar.gz
    - tss2-930.tar.gz

Open Questions:
- Any specific method to handle these 6 packages?
- Any other comment? 

[ List of Packages: Column 2 ]

Our build system uses defined paths after the Tarball package filename is decompressed, see Columns H " 'Directory Name Cgcs-Dl" and Column I " 'Directory Name tarball-dl.sh" for a
comparison:

- It could be the same after it is decompressed from the original file
  - dpkg_1.18.24.tar.xz#dpkg-1.18.24
- It could use a different name after it is decompressed from the original file
  - !integrity-kmod-668a8270.tar.gz#integrity
  - puppet-dnsmasq-cff07e90890662972c97684a2baee964f68ff3ed.tar.gz#packstack/puppet/modules/dnsmasq/

Open Questions:
- Do we want to have predefined paths or should we targeted default paths as given by the original file names?
- Any other comment?

[ List of Packages: Column 3 ]

Nothing specific here but hardcoded URLs.

Open Questions:
- Use wget Vs curl?
- Any other method to get the sources of the packages? Is it possible to migrate to source RPMs?

[ Security ]

Open Discussion! We went through an initial package integrity verification, see column F "GPG Key" but only 6 packages got their PGP key available in their official repository source.

Open Questions:
- What is the best way to approach?
- Specific question. Can we say Tarballs coming from Git sources are naturally verified?
- Any other comment?

[0] https://review.openstack.org/#/c/577953/
[1] https://ethercalc.openstack.org/615btsy7i8k6
[2] https://git.openstack.org/cgit/openstack/stx-tools/tree/centos-mirror-tools
[3] https://git.openstack.org/cgit/openstack/stx-tools/tree/centos-mirror-tools/tarball-dl.sh
[4] https://git.openstack.org/cgit/openstack/stx-tools/tree/centos-mirror-tools/tarball-dl.lst

_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss