[Starlingx-discuss] Adding packages to build .lst files

[Ponce Castaneda, Guillermo A]

Hi all,

Yes, this tool I made can come in handy to search for packages faster.
The tools utilizes a Google API to search for specific files, it can search for
any kind of file, althought we have to use it for rpms and srpms.

Google API returns a list of sites that match the search criteria, this script
goes to that list and search on the html for the <a href> tags looking for the
name of the package and tries to download them all.

In the end you get a directory tree like this: downloads/<package_name>/ with
none or many files.

Why is it written in node.js and not in any other language?
I know how to code on it (more or less) and the Google API was way easier to use
than python's, the one that I was able to make work on python was for python 2.

This script is still far from being perfect but I could move faster if more
people uses it and gives me feedback, what I have identified as improvement
opportunities are:
- It could receive a list of packages instead of a single one.
- It could look for rpm integrity or at least if the package is signed.

I will add it to the stx-tools repo shortly and see if it goes through.


On 7/17/18, 3:45 PM, "Arce Moreno, Abraham" <abraham.arce.moreno at intel.com> wrote:
    > From: Arce Moreno, Abraham [mailto:abraham.arce.moreno at intel.com]
    > Sent: Tuesday, July 17, 2018 3:29 PM
    > To: Dean Troyer <dtroyer at gmail.com>; Penney, Don
    > <Don.Penney at windriver.com>
    > Cc: starlingx-discuss at lists.starlingx.io
    > Subject: Re: [Starlingx-discuss] Adding packages to build .lst files
    > 
    > > As a side note: I started following the README in stx-tools to try out
    > > the download utility, to hopefully glean a little more info about the
    > > LST files on my own. I setup a fresh repo yesterday afternoon, then
    > > setup and launched the docker image this morning. The download fails:
    > >
    > > done!!
    > > finish 1st round of RPM downloading successfully!
    > > ERROR:  -------RPMs missing 4 in yumdownloader with L1 match
    > > ---------------
    > >
    > > The failed RPMs are recorded:
    > > $ cat centos_rpms_missing_L1.txt
    > > epel-rpm-macros-7-18.noarch.rpm
    > > python2-rpm-macros-3-18.el7.noarch.rpm
    > > python-rpm-macros-3-18.el7.noarch.rpm
    > > python-srpm-macros-3-18.el7.noarch.rpm
    > 
    > Memo has created a node script to download those dependencies and it
    > perfectly worked For me today to gather the specific versions. I am wondering
    > if we can take out to help with our missing Packages for now, here are the
    > steps using that tool:
    > 
    >     # cat output/3rd-centos_rpms_missing_L1.txt
    > output/3rd_rpms_missing_L1.txt \
    >               output/3rd_srpms_missing_L1.txt output/centos_rpms_missing_L1.txt
    > output/centos_srpms_missing_L1.txt
    >     epel-rpm-macros-7-18.noarch.rpm
    >     python2-rpm-macros-3-18.el7.noarch.rpm
    >     python-rpm-macros-3-18.el7.noarch.rpm
    >     python-srpm-macros-3-18.el7.noarch.rpm
    > 
    >     $ node index.js -p epel-rpm-macros-7-18.noarch.rpm
    >     $ node index.js -p python2-rpm-macros-3-18.el7.noarch.rpm
    >     $ node index.js -p python-rpm-macros-3-18.el7.noarch.rpm
    >     $ node index.js -p python-srpm-macros-3-18.el7.noarch.rpm
    > 
    > All Key ID for these packages ok
    > 
    >     $ rpm -vK /import/mirrors/CentOS/stx-r1/CentOS/pike/Binary/noarch/epel-
    > rpm-macros-7-18.noarch.rpm
    >     $ rpm -vK /import/mirrors/CentOS/stx-
    > r1/CentOS/pike/Binary/noarch/python2-rpm-macros-3-18.el7.noarch.rpm
    >     $ rpm -vK /import/mirrors/CentOS/stx-
    > r1/CentOS/pike/Binary/noarch/python-rpm-macros-3-18.el7.noarch.rpm
    >     $ rpm -vK /import/mirrors/CentOS/stx-
    > r1/CentOS/pike/Binary/noarch/python-srpm-macros-3-18.el7.noarch.rpm
    > 
    > _______________________________________________
    > Starlingx-discuss mailing list
    > Starlingx-discuss at lists.starlingx.io
    > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss