[Starlingx-discuss] StarlingX: Tar Compressed Files Repository Restructuring

[Arce Moreno, Abraham]

Thanks Saul!

> > Just 2 proposals.
> > 1) Every time, when I run download_mirror, it will always download mvn
> which will take around 1~2 hours.
> > Could you add a check in the script that if mvn.repo.tgz exists, do
> > not try to download it again. Same for other package
> > 2) When download list changed, the script only downloads added packages
> and also removes the ones of old versions that we do not used anymore.
> >
> I was thinking we could try to incorporate somekind of checksum
> information, in order to validate the downloads are correct and complete.

Is this harcoded checksum checking the same way Yocto does as part of the recipes? 
Can we have the checksum incorporated in our existing text file as another column? 
Do we need another file?

> > The call sequence under CentOS Mirror Tools [2] is as follows ending in
> executing our tarball-dl.sh:
> >
> > 1 download_mirror.sh
> >     1.1 dl_rpms.sh
> >     1.2 dl_other_from_centos_repo.sh
> >     1.3 tarball-dl.sh
> >
> > Open Questions:
> > - Should we keep this script being called from download_mirror.sh?
> > - Should we keep this file being written as shell script?
> > - Any comment based on midterm / long term changes?
> > - Any other comment?
> >
> So the script is doing 3 things, I think there should be more checking as to
> what exists and what's needed.  The RPM fetching can be done independant
> of the tarball, so that we can use a query of the repos instead of full
> download.

How about starting by splitting our download of tarballs from the rpms script?
this will allow us to track independently the 2 efforts and avoid having a failure
in the download of one because of the other.

Taking your idea and Zhipeng's, how about something like the steps below? 
This will be also reflected in our Developer Guide:

1 "Step Download RPMs" > download_mirror.sh
   1.1 dl_rpms.sh
   1.2 dl_other_from_centos_repo.sh
2 "Step Download Tarballs" > tarball-dl.sh
    2.1 what exists
    2.2 verify .. checksum?
    2.3 continue / download
    2.4 if download? verify .. checksum ?

> Another thought is to provide the option to point this at somekind of
> company based mirror, so that each larger company could keep an internal
> mirror, but smaller organizations will hit the external sources.

The "Step Download Tarball" can allow us to build that repo and same script to
verify its integrity, complementing with Zhipeng's idea to download new versions
or remove old ones.

> > A special character "!" is used at the beginning of 6 package file names
> denoting they require special handling is required: additional steps besides
> the common process including but not limited to remove text from some files,
> use only one directory, etc:
> >
> >      - integrity-kmod-668a8270.tar.gz
> >      - mariadb-10.1.28.tar.gz
> >      - mvn.repo.tgz
> >      - qat1.7.upstream.l.1.0.3-42.tar.gz
> >      - tpm-kmod-668a8270.tar.gz
> >      - tss2-930.tar.gz
> >
> > Open Questions:
> > - Any specific method to handle these 6 packages?
> > - Any other comment?
> >
> This one is harder, since each of these is a different special case to deal with
> git or multiple downloads (mvn). Short of creating some sort of "plugin"
> mechanism that handles each special case.
> 

I will ask for feedback to our experts about how this "plugin" can be achieved.
This includes suggestion from Zhipeng related to mvn in previous email thread.

> > Our build system uses defined paths after the Tarball package filename
> > is decompressed, see Columns H " 'Directory Name Cgcs-Dl" and Column I
> > " 'Directory Name tarball-dl.sh" for a
> > comparison:
> >
> > - It could be the same after it is decompressed from the original file
> >    - dpkg_1.18.24.tar.xz#dpkg-1.18.24
> > - It could use a different name after it is decompressed from the original file
> >    - !integrity-kmod-668a8270.tar.gz#integrity
> >    -
> > puppet-dnsmasq-
> cff07e90890662972c97684a2baee964f68ff3ed.tar.gz#packsta
> > ck/puppet/modules/dnsmasq/
> >
> > Open Questions:
> > - Do we want to have predefined paths or should we targeted default paths
> as given by the original file names?
> > - Any other comment?
> >
> I have not looked deeper, but are the predefined paths required for the way
> build system is currently looking for files?

In my limited understanding of the Build System? Yes according to a first search of paths,
but I might be wrong. I will leave to Wind River team to give the correct answer.

> > [ List of Packages: Column 3 ]
> >
> > Nothing specific here but hardcoded URLs.
> >
> > Open Questions:
> > - Use wget Vs curl?
> 
> I would lean towards curl as it has better protocol support and supports
> socks for proxy activity.

Annotated

> 
> > - Any other method to get the sources of the packages? Is it possible to
> migrate to source RPMs?
> >
> > [ Security ]
> >
> > Open Discussion! We went through an initial package integrity verification,
> see column F "GPG Key" but only 6 packages got their PGP key available in
> their official repository source.
> >
> > Open Questions:
> > - What is the best way to approach?
> If there is a GPG key, then I think we should verify it, we could also
> validate that what we currently have does not need to be updated (see
> above)
> 
> > - Specific question. Can we say Tarballs coming from Git sources are
> naturally verified?
> I think that makes sense, the problem with those tarballs is that we
> can't have a checksum on them since they are created on the fly and the
> time stamps change.

Annotated + some tests in progress + some proposals