[Starlingx-discuss] Adding packages to build .lst files
Hi folks, We've got some packages to add to the build, and I'm wondering if there's some description of the various .lst files in stxtools/centos-mirror-tools. For example, how do you decide whether a package is listed in rpms_from_centos_repo.lst vs rpms_from_centos_3rd_parties.lst? I started writing a script to pull together a list of the dependencies based on Requires and BuildRequires. As I understand it, we need to add all such dependencies, recursively, to these .lst files. Using the StarlingX.repo config file as an arg to repoquery, I get some packages overlapping with rpms_from_centos_3rd_parties.lst, however, so I'm hoping there's some clear direction with these files. My script is finding hundreds of rpms to add to rpms_from_centos_repo.lst with its calls to repoquery to resolve the dependencies, so either I've got lots of overlap and many of these need to go to the 3rdparty file, my repoquery approach for automating this is completely wrong, or these new packages are using a completely different set of dependencies for the builds than anything else we've got. I've attached the first draft of my script for anyone that's curious. Thanks, Don. Don Penney, Developer, Wind River
Providing the script in a zip file for anyone who couldn't see the original shell script file attachment. As a side note: I started following the README in stx-tools to try out the download utility, to hopefully glean a little more info about the LST files on my own. I setup a fresh repo yesterday afternoon, then setup and launched the docker image this morning. The download fails: done!! finish 1st round of RPM downloading successfully! ERROR: -------RPMs missing 4 in yumdownloader with L1 match --------------- The failed RPMs are recorded: $ cat centos_rpms_missing_L1.txt epel-rpm-macros-7-18.noarch.rpm python2-rpm-macros-3-18.el7.noarch.rpm python-rpm-macros-3-18.el7.noarch.rpm python-srpm-macros-3-18.el7.noarch.rpm Seems odd to fail on these... so a repoquery to check: $ repoquery -c StarlingX.repo --quiet epel-rpm-macros python2-rpm-macros python-rpm-macros python-srpm-macros epel-rpm-macros-0:7-19.noarch python-rpm-macros-0:3-21.el7.noarch python-srpm-macros-0:3-21.el7.noarch python2-rpm-macros-0:3-21.el7.noarch Yep... versions don't match. So these pkgs have been upversioned in the repos, and the older versions are not maintained in the repos. So the LST files would seem to be quickly out of date. From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Monday, July 16, 2018 2:22 PM To: 'starlingx-discuss@lists.starlingx.io' Subject: [Starlingx-discuss] Adding packages to build .lst files Hi folks, We've got some packages to add to the build, and I'm wondering if there's some description of the various .lst files in stxtools/centos-mirror-tools. For example, how do you decide whether a package is listed in rpms_from_centos_repo.lst vs rpms_from_centos_3rd_parties.lst? I started writing a script to pull together a list of the dependencies based on Requires and BuildRequires. As I understand it, we need to add all such dependencies, recursively, to these .lst files. Using the StarlingX.repo config file as an arg to repoquery, I get some packages overlapping with rpms_from_centos_3rd_parties.lst, however, so I'm hoping there's some clear direction with these files. My script is finding hundreds of rpms to add to rpms_from_centos_repo.lst with its calls to repoquery to resolve the dependencies, so either I've got lots of overlap and many of these need to go to the 3rdparty file, my repoquery approach for automating this is completely wrong, or these new packages are using a completely different set of dependencies for the builds than anything else we've got. I've attached the first draft of my script for anyone that's curious. Thanks, Don. Don Penney, Developer, Wind River
On Mon, Jul 16, 2018 at 1:21 PM, Penney, Don <Don.Penney@windriver.com> wrote:
I started writing a script to pull together a list of the dependencies based on Requires and BuildRequires. As I understand it, we need to add all such dependencies, recursively, to these .lst files. Using the StarlingX.repo config file as an arg to repoquery, I get some packages overlapping with rpms_from_centos_3rd_parties.lst, however, so I’m hoping there’s some clear direction with these files. My script is finding hundreds of rpms to add to rpms_from_centos_repo.lst with its calls to repoquery to resolve the dependencies, so either I’ve got lots of overlap and many of these need to go to the 3rdparty file, my repoquery approach for automating this is completely wrong, or these new packages are using a completely different set of dependencies for the builds than anything else we’ve got.
Thank you! I haven't had time to look at that yet but have been hoping that we could generate even more of this stuff from the other parts of the repo. As for the volume of dependencies, it's been a while since I did something like that with RHEL, but it has always had copious amounts of dependencies even for "small" installations. (Does DBus still require X on a server? I don't think so, but it once did...)
I’ve attached the first draft of my script for anyone that’s curious.
The attachment was bounced by Intel's MTA because of the .sh attachment so @intel.com addresses missed this. I'd go ahead and propose a WIP review[0] to stx-tools and iterate on it there. dt [0] In case not everyone knows the common practices in Gerrit, WIP reviews usually begin the commit message with 'WIP' or the author/committer can set a Workflow -1 in the Reply dialog. Or both. Often this means it is not ready for detailed review or it could be a way to solicit feedback during development, it is convenient to mention the reason for WIP in the commit message. -- Dean Troyer dtroyer@gmail.com
As a side note: I started following the README in stx-tools to try out the download utility, to hopefully glean a little more info about the LST files on my own. I setup a fresh repo yesterday afternoon, then setup and launched the docker image this morning. The download fails:
done!! finish 1st round of RPM downloading successfully! ERROR: -------RPMs missing 4 in yumdownloader with L1 match ---------------
The failed RPMs are recorded: $ cat centos_rpms_missing_L1.txt epel-rpm-macros-7-18.noarch.rpm python2-rpm-macros-3-18.el7.noarch.rpm python-rpm-macros-3-18.el7.noarch.rpm python-srpm-macros-3-18.el7.noarch.rpm
Memo has created a node script to download those dependencies and it perfectly worked For me today to gather the specific versions. I am wondering if we can take out to help with our missing Packages for now, here are the steps using that tool: # cat output/3rd-centos_rpms_missing_L1.txt output/3rd_rpms_missing_L1.txt \ output/3rd_srpms_missing_L1.txt output/centos_rpms_missing_L1.txt output/centos_srpms_missing_L1.txt epel-rpm-macros-7-18.noarch.rpm python2-rpm-macros-3-18.el7.noarch.rpm python-rpm-macros-3-18.el7.noarch.rpm python-srpm-macros-3-18.el7.noarch.rpm $ node index.js -p epel-rpm-macros-7-18.noarch.rpm $ node index.js -p python2-rpm-macros-3-18.el7.noarch.rpm $ node index.js -p python-rpm-macros-3-18.el7.noarch.rpm $ node index.js -p python-srpm-macros-3-18.el7.noarch.rpm All Key ID for these packages ok $ rpm -vK /import/mirrors/CentOS/stx-r1/CentOS/pike/Binary/noarch/epel-rpm-macros-7-18.noarch.rpm $ rpm -vK /import/mirrors/CentOS/stx-r1/CentOS/pike/Binary/noarch/python2-rpm-macros-3-18.el7.noarch.rpm $ rpm -vK /import/mirrors/CentOS/stx-r1/CentOS/pike/Binary/noarch/python-rpm-macros-3-18.el7.noarch.rpm $ rpm -vK /import/mirrors/CentOS/stx-r1/CentOS/pike/Binary/noarch/python-srpm-macros-3-18.el7.noarch.rpm
participants (3)
-
Arce Moreno, Abraham
-
Dean Troyer
-
Penney, Don