[Starlingx-discuss] Adding packages to build .lst files
Hi all, Yes, this tool I made can come in handy to search for packages faster. The tools utilizes a Google API to search for specific files, it can search for any kind of file, althought we have to use it for rpms and srpms. Google API returns a list of sites that match the search criteria, this script goes to that list and search on the html for the <a href> tags looking for the name of the package and tries to download them all. In the end you get a directory tree like this: downloads/<package_name>/ with none or many files. Why is it written in node.js and not in any other language? I know how to code on it (more or less) and the Google API was way easier to use than python's, the one that I was able to make work on python was for python 2. This script is still far from being perfect but I could move faster if more people uses it and gives me feedback, what I have identified as improvement opportunities are: - It could receive a list of packages instead of a single one. - It could look for rpm integrity or at least if the package is signed. I will add it to the stx-tools repo shortly and see if it goes through. On 7/17/18, 3:45 PM, "Arce Moreno, Abraham" <abraham.arce.moreno@intel.com> wrote: > From: Arce Moreno, Abraham [mailto:abraham.arce.moreno@intel.com] > Sent: Tuesday, July 17, 2018 3:29 PM > To: Dean Troyer <dtroyer@gmail.com>; Penney, Don > <Don.Penney@windriver.com> > Cc: starlingx-discuss@lists.starlingx.io > Subject: Re: [Starlingx-discuss] Adding packages to build .lst files > > > As a side note: I started following the README in stx-tools to try out > > the download utility, to hopefully glean a little more info about the > > LST files on my own. I setup a fresh repo yesterday afternoon, then > > setup and launched the docker image this morning. The download fails: > > > > done!! > > finish 1st round of RPM downloading successfully! > > ERROR: -------RPMs missing 4 in yumdownloader with L1 match > > --------------- > > > > The failed RPMs are recorded: > > $ cat centos_rpms_missing_L1.txt > > epel-rpm-macros-7-18.noarch.rpm > > python2-rpm-macros-3-18.el7.noarch.rpm > > python-rpm-macros-3-18.el7.noarch.rpm > > python-srpm-macros-3-18.el7.noarch.rpm > > Memo has created a node script to download those dependencies and it > perfectly worked For me today to gather the specific versions. I am wondering > if we can take out to help with our missing Packages for now, here are the > steps using that tool: > > # cat output/3rd-centos_rpms_missing_L1.txt > output/3rd_rpms_missing_L1.txt \ > output/3rd_srpms_missing_L1.txt output/centos_rpms_missing_L1.txt > output/centos_srpms_missing_L1.txt > epel-rpm-macros-7-18.noarch.rpm > python2-rpm-macros-3-18.el7.noarch.rpm > python-rpm-macros-3-18.el7.noarch.rpm > python-srpm-macros-3-18.el7.noarch.rpm > > $ node index.js -p epel-rpm-macros-7-18.noarch.rpm > $ node index.js -p python2-rpm-macros-3-18.el7.noarch.rpm > $ node index.js -p python-rpm-macros-3-18.el7.noarch.rpm > $ node index.js -p python-srpm-macros-3-18.el7.noarch.rpm > > All Key ID for these packages ok > > $ rpm -vK /import/mirrors/CentOS/stx-r1/CentOS/pike/Binary/noarch/epel- > rpm-macros-7-18.noarch.rpm > $ rpm -vK /import/mirrors/CentOS/stx- > r1/CentOS/pike/Binary/noarch/python2-rpm-macros-3-18.el7.noarch.rpm > $ rpm -vK /import/mirrors/CentOS/stx- > r1/CentOS/pike/Binary/noarch/python-rpm-macros-3-18.el7.noarch.rpm > $ rpm -vK /import/mirrors/CentOS/stx- > r1/CentOS/pike/Binary/noarch/python-srpm-macros-3-18.el7.noarch.rpm > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Tue, Jul 17, 2018 at 4:25 PM, Ponce Castaneda, Guillermo A <guillermo.a.ponce.castaneda@intel.com> wrote:
Why is it written in node.js and not in any other language? I know how to code on it (more or less) and the Google API was way easier to use than python's, the one that I was able to make work on python was for python 2.
Is there any other node in STX? I am not aware of any and would be reluctant to add yet another platform requirement to this build system. I know I'd be reluctant to install node on my laptop just to run this... it's bad enough the python bits I have to go through to install OpenStackClient and I wrote the dang thing.
This script is still far from being perfect but I could move faster if more people uses it and gives me feedback, what I have identified as improvement opportunities are: - It could receive a list of packages instead of a single one. - It could look for rpm integrity or at least if the package is signed.
Validating signed packages is an absolute requirement.
I will add it to the stx-tools repo shortly and see if it goes through.
Posting a review is the right way to share something to get feedback on it. Make it clear in the commit message that is your intention, optionally even set workflow -1 to prevent it from being merged although sometimes people don't even look at those... dt -- Dean Troyer dtroyer@gmail.com
>On 7/17/18, 7:18 PM, "Dean Troyer" <dtroyer@gmail.com> wrote: >On Tue, Jul 17, 2018 at 4:25 PM, Ponce Castaneda, Guillermo A ><guillermo.a.ponce.castaneda@intel.com> wrote: >> Why is it written in node.js and not in any other language? >> I know how to code on it (more or less) and the Google API was way easier to use >> than python's, the one that I was able to make work on python was for python 2. >Is there any other node in STX? I am not aware of any and would be >reluctant to add yet another platform requirement to this build >system. I know I'd be reluctant to install node on my laptop just to >run this... it's bad enough the python bits I have to go through to >install OpenStackClient and I wrote the dang thing. That is correct, I do not think there is another node.js code, and that is why I also created a Dockerfile so nobody has to install node on their machines and just run it inside a container. >> This script is still far from being perfect but I could move faster if more >> people uses it and gives me feedback, what I have identified as improvement >> opportunities are: >> - It could receive a list of packages instead of a single one. >> - It could look for rpm integrity or at least if the package is signed. >Validating signed packages is an absolute requirement. Totally agree, I haven't found a way to do it with node instructions, but I guess I can just add an 'rpm -vK' command inside a subprocess like instruction. >> I will add it to the stx-tools repo shortly and see if it goes through. >Posting a review is the right way to share something to get feedback >on it. Make it clear in the commit message that is your intention, >optionally even set workflow -1 to prevent it from being merged >although sometimes people don't even look at those... >dt -- Dean Troyer dtroyer@gmail.com
On Wed, Jul 18, 2018 at 8:26 AM, Ponce Castaneda, Guillermo A <guillermo.a.ponce.castaneda@intel.com> wrote:
That is correct, I do not think there is another node.js code, and that is why I also created a Dockerfile so nobody has to install node on their machines and just run it inside a container.
I'm sorry but I do not consider requiring Docker and some non-trivial OS image to run a simple script to be an improvement. It isn't just the installation of the runtime it is also the cognitive overhead of requiring another bit of language expertise to maintain the tool. If this were a node project I'd say the same thing about introducing a python script. dt -- Dean Troyer dtroyer@gmail.com
>On 7/18/18, 11:47 AM, "Dean Troyer" <dtroyer@gmail.com> wrote: >On Wed, Jul 18, 2018 at 8:26 AM, Ponce Castaneda, Guillermo A ><guillermo.a.ponce.castaneda@intel.com> wrote: >> That is correct, I do not think there is another node.js code, and that is why I >> also created a Dockerfile so nobody has to install node on their machines and >> just run it inside a container. >I'm sorry but I do not consider requiring Docker and some non-trivial >OS image to run a simple script to be an improvement. It isn't just >the installation of the runtime it is also the cognitive overhead of >requiring another bit of language expertise to maintain the tool. If >this were a node project I'd say the same thing about introducing a >python script. Thanks for the feedback Dean, I will the improve this tool on the points you are mentioning and will try going at it again. >dt Guillermo Ponce -- Dean Troyer dtroyer@gmail.com
Hello team, I am giving it another try to introduce and automatic package downloader to the mirror creation process, this time I managed to actually write it on python3. The goal of this tool is still the same, to avoid the hassle of manually go looking and downloading failing packages from Google and just run a single command instead. You can go and review the code here: https://review.openstack.org/#/c/585009/ I really appreciate any feedback, specially opportunities of improvement and any kind comments. Thank You - Guillermo Ponce On 7/18/18, 12:42 PM, "Ponce Castaneda, Guillermo A" <guillermo.a.ponce.castaneda@intel.com> wrote: >>On 7/18/18, 11:47 AM, "Dean Troyer" <dtroyer@gmail.com> wrote: >>On Wed, Jul 18, 2018 at 8:26 AM, Ponce Castaneda, Guillermo A >><guillermo.a.ponce.castaneda@intel.com> wrote: >>> That is correct, I do not think there is another node.js code, and that is why I >>> also created a Dockerfile so nobody has to install node on their machines and >>> just run it inside a container. >>I'm sorry but I do not consider requiring Docker and some non-trivial >>OS image to run a simple script to be an improvement. It isn't just >>the installation of the runtime it is also the cognitive overhead of >>requiring another bit of language expertise to maintain the tool. If >>this were a node project I'd say the same thing about introducing a >>python script. >Thanks for the feedback Dean, >I will the improve this tool on the points you are mentioning and will try going >at it again. >>dt >Guillermo Ponce >-- >>Dean Troyer >>dtroyer@gmail.com
Do we trust every link on google to point to a malware free version of the package ? I think it's ill advised to take the human out of the loop. At most, list the validated download url's found... along with a warning from all but a few trusted url's. Scott On 18-07-23 03:58 PM, Ponce Castaneda, Guillermo A wrote:
Hello team,
I am giving it another try to introduce and automatic package downloader to the mirror creation process, this time I managed to actually write it on python3.
The goal of this tool is still the same, to avoid the hassle of manually go looking and downloading failing packages from Google and just run a single command instead.
You can go and review the code here: https://review.openstack.org/#/c/585009/ I really appreciate any feedback, specially opportunities of improvement and any kind comments.
Thank You - Guillermo Ponce
On 7/18/18, 12:42 PM, "Ponce Castaneda, Guillermo A" <guillermo.a.ponce.castaneda@intel.com> wrote:
>>On 7/18/18, 11:47 AM, "Dean Troyer" <dtroyer@gmail.com> wrote:
>>On Wed, Jul 18, 2018 at 8:26 AM, Ponce Castaneda, Guillermo A >><guillermo.a.ponce.castaneda@intel.com> wrote: >>> That is correct, I do not think there is another node.js code, and that is why I >>> also created a Dockerfile so nobody has to install node on their machines and >>> just run it inside a container.
>>I'm sorry but I do not consider requiring Docker and some non-trivial >>OS image to run a simple script to be an improvement. It isn't just >>the installation of the runtime it is also the cognitive overhead of >>requiring another bit of language expertise to maintain the tool. If >>this were a node project I'd say the same thing about introducing a >>python script.
>Thanks for the feedback Dean, >I will the improve this tool on the points you are mentioning and will try going >at it again.
>>dt
>Guillermo Ponce
>--
>>Dean Troyer >>dtroyer@gmail.com
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>On 7/23/18, 3:08 PM, "Scott Little" <scott.little@windriver.com> wrote: >Do we trust every link on google to point to a malware free version of >the package ? >I think it's ill advised to take the human out of the loop. At most, >list the validated download url's found... along with a warning from all >but a few trusted url's. Totally agree, the script prints the URL where the package comes from on Line 64: 64 print("Downloading file %02d: %s" % (count, p)) Regarding the trusted URLs, I did not know there was a list of them, in fact I thought the problem was that since the package was not found on the repositories, which are trusted, we needed to look for them on the web, if there is a list, I will make use of it and display any warning message just as suggested. The workaround so far is to run a command rpm -Kv <package.rpm> on the packages to be sure the packages are signed and throw away the ones that are not. As for the automation of the rpm check process, it is still work in progress, I want to avoid using a subprocess command to do it and I want to do it using only python modules, so far I haven't had luck when looking for a module that does that, another reason not to use a subprocess command is that if you run the program from a system that does not have rpm command the script would fail anyway. >Scott Thank you - Guillermo Ponce >>On 18-07-23 03:58 PM, Ponce Castaneda, Guillermo A wrote: >> Hello team, >> I am giving it another try to introduce and automatic package downloader >> to the mirror creation process, this time I managed to actually write it >> on python3. >> The goal of this tool is still the same, to avoid the hassle of manually go >> looking and downloading failing packages from Google and just run a >> single command instead. >> You can go and review the code here: https://review.openstack.org/#/c/585009/ >> I really appreciate any feedback, specially opportunities of improvement and any >> kind comments. >> Thank You >> - Guillermo Ponce >> On 7/18/18, 12:42 PM, "Ponce Castaneda, Guillermo A" <guillermo.a.ponce.castaneda@intel.com> wrote: >> >>>On 7/18/18, 11:47 AM, "Dean Troyer" <dtroyer@gmail.com> wrote: >>>On Wed, Jul 18, 2018 at 8:26 AM, Ponce Castaneda, Guillermo A >>><guillermo.a.ponce.castaneda@intel.com> wrote: >>>> That is correct, I do not think there is another node.js code, and that is why I >>>> also created a Dockerfile so nobody has to install node on their machines and >>>> just run it inside a container. >>>I'm sorry but I do not consider requiring Docker and some non-trivial >>>OS image to run a simple script to be an improvement. It isn't just >>>the installation of the runtime it is also the cognitive overhead of >>>requiring another bit of language expertise to maintain the tool. If >>>this were a node project I'd say the same thing about introducing a >>>python script. >>Thanks for the feedback Dean, >>I will the improve this tool on the points you are mentioning and will try going >>at it again. >>>dt >>Guillermo Ponce >>-- >>>Dean Troyer >>>dtroyer@gmail.com >> _______________________________________________ >> Starlingx-discuss mailing list >> Starlingx-discuss@lists.starlingx.io >> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss >_______________________________________________ >Starlingx-discuss mailing list >Starlingx-discuss@lists.starlingx.io >http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (3)
-
Dean Troyer
-
Ponce Castaneda, Guillermo A
-
Scott Little