[Starlingx-discuss] StarlingX: Tar Compressed Files Repository Restructuring

[Saul Wold]

On 07/26/2018 08:09 AM, Arce Moreno, Abraham wrote:
> Thanks Saul!
> 
>>> Just 2 proposals.
>>> 1) Every time, when I run download_mirror, it will always download mvn
>> which will take around 1~2 hours.
>>> Could you add a check in the script that if mvn.repo.tgz exists, do
>>> not try to download it again. Same for other package
>>> 2) When download list changed, the script only downloads added packages
>> and also removes the ones of old versions that we do not used anymore.
>>>
>> I was thinking we could try to incorporate somekind of checksum
>> information, in order to validate the downloads are correct and complete.
> 
> Is this harcoded checksum checking the same way Yocto does as part of the recipes?
> Can we have the checksum incorporated in our existing text file as another column?
> Do we need another file?
> 
>>> The call sequence under CentOS Mirror Tools [2] is as follows ending in
>> executing our tarball-dl.sh:
>>>
>>> 1 download_mirror.sh
>>>      1.1 dl_rpms.sh
>>>      1.2 dl_other_from_centos_repo.sh
>>>      1.3 tarball-dl.sh
>>>
>>> Open Questions:
>>> - Should we keep this script being called from download_mirror.sh?
>>> - Should we keep this file being written as shell script?
>>> - Any comment based on midterm / long term changes?
>>> - Any other comment?
>>>
>> So the script is doing 3 things, I think there should be more checking as to
>> what exists and what's needed.  The RPM fetching can be done independant
>> of the tarball, so that we can use a query of the repos instead of full
>> download.
> 
> How about starting by splitting our download of tarballs from the rpms script?
> this will allow us to track independently the 2 efforts and avoid having a failure
> in the download of one because of the other.
> 
> Taking your idea and Zhipeng's, how about something like the steps below?
> This will be also reflected in our Developer Guide:
> 
> 1 "Step Download RPMs" > download_mirror.sh
>     1.1 dl_rpms.sh
>     1.2 dl_other_from_centos_repo.sh
Can you do an existence check here also and use the repoquery or is that 
already part of dl_rpms?  Can these scripts be merged into one that just 
uses 2 different input information?

> 2 "Step Download Tarballs" > tarball-dl.sh
>      2.1 what exists
>      2.2 verify .. checksum?
>      2.3 continue / download
>      2.4 if download? verify .. checksum ?
> 
>> Another thought is to provide the option to point this at somekind of
>> company based mirror, so that each larger company could keep an internal
>> mirror, but smaller organizations will hit the external sources.
> 
> The "Step Download Tarball" can allow us to build that repo and same script to
> verify its integrity, complementing with Zhipeng's idea to download new versions
> or remove old ones.
> 
Sounds good, can we do the same for the RPM repo, not just tarballs?

Sau!

>>> A special character "!" is used at the beginning of 6 package file names
>> denoting they require special handling is required: additional steps besides
>> the common process including but not limited to remove text from some files,
>> use only one directory, etc:
>>>
>>>       - integrity-kmod-668a8270.tar.gz
>>>       - mariadb-10.1.28.tar.gz
>>>       - mvn.repo.tgz
>>>       - qat1.7.upstream.l.1.0.3-42.tar.gz
>>>       - tpm-kmod-668a8270.tar.gz
>>>       - tss2-930.tar.gz
>>>
>>> Open Questions:
>>> - Any specific method to handle these 6 packages?
>>> - Any other comment?
>>>
>> This one is harder, since each of these is a different special case to deal with
>> git or multiple downloads (mvn). Short of creating some sort of "plugin"
>> mechanism that handles each special case.
>>
> 
> I will ask for feedback to our experts about how this "plugin" can be achieved.
> This includes suggestion from Zhipeng related to mvn in previous email thread.
> 
>>> Our build system uses defined paths after the Tarball package filename
>>> is decompressed, see Columns H " 'Directory Name Cgcs-Dl" and Column I
>>> " 'Directory Name tarball-dl.sh" for a
>>> comparison:
>>>
>>> - It could be the same after it is decompressed from the original file
>>>     - dpkg_1.18.24.tar.xz#dpkg-1.18.24
>>> - It could use a different name after it is decompressed from the original file
>>>     - !integrity-kmod-668a8270.tar.gz#integrity
>>>     -
>>> puppet-dnsmasq-
>> cff07e90890662972c97684a2baee964f68ff3ed.tar.gz#packsta
>>> ck/puppet/modules/dnsmasq/
>>>
>>> Open Questions:
>>> - Do we want to have predefined paths or should we targeted default paths
>> as given by the original file names?
>>> - Any other comment?
>>>
>> I have not looked deeper, but are the predefined paths required for the way
>> build system is currently looking for files?
> 
> In my limited understanding of the Build System? Yes according to a first search of paths,
> but I might be wrong. I will leave to Wind River team to give the correct answer.
> 
>>> [ List of Packages: Column 3 ]
>>>
>>> Nothing specific here but hardcoded URLs.
>>>
>>> Open Questions:
>>> - Use wget Vs curl?
>>
>> I would lean towards curl as it has better protocol support and supports
>> socks for proxy activity.
> 
> Annotated
> 
>>
>>> - Any other method to get the sources of the packages? Is it possible to
>> migrate to source RPMs?
>>>
>>> [ Security ]
>>>
>>> Open Discussion! We went through an initial package integrity verification,
>> see column F "GPG Key" but only 6 packages got their PGP key available in
>> their official repository source.
>>>
>>> Open Questions:
>>> - What is the best way to approach?
>> If there is a GPG key, then I think we should verify it, we could also
>> validate that what we currently have does not need to be updated (see
>> above)
>>
>>> - Specific question. Can we say Tarballs coming from Git sources are
>> naturally verified?
>> I think that makes sense, the problem with those tarballs is that we
>> can't have a checksum on them since they are created on the fly and the
>> time stamps change.
> 
> Annotated + some tests in progress + some proposals
>