[Starlingx-discuss] [Build][MultiOS] Use of current centOS spec files in Mock build system with repos
Hi team A few weeks ago I had the curiosity to investigate if there was an rpm repo in CENGN for starling X. Scott was very kind to point me about, thanks a lot Scott. Why this is related to multi-os? RPM repositories are warehouses of RPM package files that are easy to consume to build with upstream tools like mock. mock is a tool for building RPM packages. we can use mock to build packages for many different versions of CentOS/Red Hat and Fedora. After some experiments I was able to build basic packages from fault management using the following methodology: 1) set up mock in my Linux distro ( in my case centos 7.6 ) 2) use the following mock repositories config file to build: https://github.com/starlingx-staging/stx-packaging/blob/master/configs/docke... 3) build as: srpm : mock -r $(MOCK_CONFIG) --buildsrpm --spec=$(SPEC) --sources=$(SOURCES) --resultdir $(RES_DIR) --define="tis_patch_ver $(TIS_PATCH_VER)" rpm : mock -r $(MOCK_CONFIG) --rebuild $(SRPM) --resultdir $(RES_DIR) --define="tis_patch_ver $(TIS_PATCH_VER)" ( the variables point to the location of my spec file and source tar.gz files, the MOCK_CONFIG points to the path of my local-centos-7-x86_64.cfg file ) In the end, it works pretty straight forward, proving the following points from this experiment ( hopefully useful for someone else ) : a) If we use the CENGN repos ( plus the centos repo ) is possible to build an RPM w/o the need to download the mirror. Great job Scott and build team! b) Is possible to use alternative upstream tools to build a single RPM w/o the need to rebuild the build dependencies. Now a few questions came to me : a ) Are these CENGN repos in active use, I mean, are they updated with a new ISO or every time a developer merge a commit in master? b) After trying to build all the packages in fault management I found that: * the source files directories are not standard ( I know Erich is working on this ) * there are missing BuildRequires in the current centos spec files ( they build due to the specific architecture that builds system has to handle/install the build req in a parallel build ) In the aim of improving the project to some other members of the community use different tools to build different layers of the OS. I was wonder how valuable the community see the task of adding the proper BuildRequires in all the flock spec files that have missing ones. I create a launchpad and a patch for the missing build req I found in fm-rest-api: https://review.opendev.org/#/c/657869/ Any feedback from the community more than welcome Regards Victor Rodriguez
On 5/8/19 12:44 PM, Victor Rodriguez wrote:
Hi team
A few weeks ago I had the curiosity to investigate if there was an rpm repo in CENGN for starling X. Scott was very kind to point me about, thanks a lot Scott. Why this is related to multi-os? RPM repositories are warehouses of RPM package files that are easy to consume to build with upstream tools like mock. mock is a tool for building RPM packages. we can use mock to build packages for many different versions of CentOS/Red Hat and Fedora.
After some experiments I was able to build basic packages from fault management using the following methodology:
1) set up mock in my Linux distro ( in my case centos 7.6 ) 2) use the following mock repositories config file to build: https://github.com/starlingx-staging/stx-packaging/blob/master/configs/docke... 3) build as: srpm : mock -r $(MOCK_CONFIG) --buildsrpm --spec=$(SPEC) --sources=$(SOURCES) --resultdir $(RES_DIR) --define="tis_patch_ver $(TIS_PATCH_VER)" rpm : mock -r $(MOCK_CONFIG) --rebuild $(SRPM) --resultdir $(RES_DIR) --define="tis_patch_ver $(TIS_PATCH_VER)"
( the variables point to the location of my spec file and source tar.gz files, the MOCK_CONFIG points to the path of my local-centos-7-x86_64.cfg file )
In the end, it works pretty straight forward, proving the following points from this experiment ( hopefully useful for someone else ) :
a) If we use the CENGN repos ( plus the centos repo ) is possible to build an RPM w/o the need to download the mirror. Great job Scott and build team! b) Is possible to use alternative upstream tools to build a single RPM w/o the need to rebuild the build dependencies.
Now a few questions came to me :
a ) Are these CENGN repos in active use, I mean, are they updated with a new ISO or every time a developer merge a commit in master? I believe that we have a stable area and that the latest also contains actual RPM repos (with repodata) in both the inputs and outputs RPMS subdirectories, so it seems that they are always generated. So it's a
Great work to Erich, Marcela and Victor! process question about when to move them the a "green" master area to work with more stable RPMS.
b) After trying to build all the packages in fault management I found that: * the source files directories are not standard ( I know Erich is working on this )
If your referring to the source located in fm-common/source or in the case of fm-api it's fm-api, or fm-client: python-fmclient/fmclient, each are different. There is a way to normalize this in the spec file by setting %setup -c <src directory> fm-common: %prep %setup -n fm-common/sources python-fmclient: %prep %setup -n python-fmclient/fmclient I learned about this trying to work with Suse / OSB example.
* there are missing BuildRequires in the current centos spec files ( they build due to the specific architecture that builds system has to handle/install the build req in a parallel build )
This is important to fix and gain the info for not just RPMs, but DEBs aswell and any future build tools.
In the aim of improving the project to some other members of the community use different tools to build different layers of the OS. I was wonder how valuable the community see the task of adding the proper BuildRequires in all the flock spec files that have missing ones. I create a launchpad and a patch for the missing build req I found in fm-rest-api:
Looks good, other than the commit message is too wide ;-0! Sau!
Any feedback from the community more than welcome
Regards
Victor Rodriguez
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Victor, The CENGN repos are where the build results go. I believe these are built nightly. So, they are not updated on a per-commit basis but they are "eventually" up to date. If you encounter spec files with bad or missing BuildRequires (or Requires) then yes it is very valuable to raise a Launchpad. An audit tool would be helpful. Below are some notes I took for steps that used to work for me, but I have not tried them in a "very" long time, and the build tools may have changed. (the build tools make use of mock) Note that these steps remove the mock folders. I suspect the technique you are using could also help track down missing Requires. ================ How to verify a spec has correct BuildRequires: cd $MY_WORKSPACE rm -Rf mock* build-srpms --clean <pkg> build-srpms <pkg> build-rpms --clean <pkg> build-rpms <pkg> (If this builds, you are good) Usually "devel" or "staticdev" packages are missing ================ How to verify a spec has correct Requires: cd $MY_WORKSPACE rm -Rf mock* mock -r $MY_BUILD_CFG --install <full path to RPM> ie: mock -r $MY_BUILD_CFG --install rpmbuild/RPMS/python-keystoneclient-1.3.1-r3.noarch.rpm ====================== Al -----Original Message----- From: Victor Rodriguez [mailto:vm.rod25@gmail.com] Sent: Wednesday, May 08, 2019 3:45 PM To: Little, Scott; starlingx-discuss@lists.starlingx.io; Cordoba Malibran, Erich Subject: [Starlingx-discuss] [Build][MultiOS] Use of current centOS spec files in Mock build system with repos Hi team A few weeks ago I had the curiosity to investigate if there was an rpm repo in CENGN for starling X. Scott was very kind to point me about, thanks a lot Scott. Why this is related to multi-os? RPM repositories are warehouses of RPM package files that are easy to consume to build with upstream tools like mock. mock is a tool for building RPM packages. we can use mock to build packages for many different versions of CentOS/Red Hat and Fedora. After some experiments I was able to build basic packages from fault management using the following methodology: 1) set up mock in my Linux distro ( in my case centos 7.6 ) 2) use the following mock repositories config file to build: https://github.com/starlingx-staging/stx-packaging/blob/master/configs/docke... 3) build as: srpm : mock -r $(MOCK_CONFIG) --buildsrpm --spec=$(SPEC) --sources=$(SOURCES) --resultdir $(RES_DIR) --define="tis_patch_ver $(TIS_PATCH_VER)" rpm : mock -r $(MOCK_CONFIG) --rebuild $(SRPM) --resultdir $(RES_DIR) --define="tis_patch_ver $(TIS_PATCH_VER)" ( the variables point to the location of my spec file and source tar.gz files, the MOCK_CONFIG points to the path of my local-centos-7-x86_64.cfg file ) In the end, it works pretty straight forward, proving the following points from this experiment ( hopefully useful for someone else ) : a) If we use the CENGN repos ( plus the centos repo ) is possible to build an RPM w/o the need to download the mirror. Great job Scott and build team! b) Is possible to use alternative upstream tools to build a single RPM w/o the need to rebuild the build dependencies. Now a few questions came to me : a ) Are these CENGN repos in active use, I mean, are they updated with a new ISO or every time a developer merge a commit in master? b) After trying to build all the packages in fault management I found that: * the source files directories are not standard ( I know Erich is working on this ) * there are missing BuildRequires in the current centos spec files ( they build due to the specific architecture that builds system has to handle/install the build req in a parallel build ) In the aim of improving the project to some other members of the community use different tools to build different layers of the OS. I was wonder how valuable the community see the task of adding the proper BuildRequires in all the flock spec files that have missing ones. I create a launchpad and a patch for the missing build req I found in fm-rest-api: https://review.opendev.org/#/c/657869/ Any feedback from the community more than welcome Regards Victor Rodriguez _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Thu, May 9, 2019 at 10:38 AM Bailey, Henry Albert (Al) <Al.Bailey@windriver.com> wrote:
Hi Victor, The CENGN repos are where the build results go. I believe these are built nightly. So, they are not updated on a per-commit basis but they are "eventually" up to date.
Thanks for the info Abert, I think this is good enough to have nightly builds
If you encounter spec files with bad or missing BuildRequires (or Requires) then yes it is very valuable to raise a Launchpad.
An audit tool would be helpful.
Below are some notes I took for steps that used to work for me, but I have not tried them in a "very" long time, and the build tools may have changed. (the build tools make use of mock) Note that these steps remove the mock folders. I suspect the technique you are using could also help track down missing Requires.
Thanks so much for sharing this, I will re use part of it and adjust to my scripts to find all the missing build req. Once we finish we will have spec files that could be build in mock or any other spec build system.
================ How to verify a spec has correct BuildRequires:
cd $MY_WORKSPACE rm -Rf mock*
build-srpms --clean <pkg> build-srpms <pkg>
build-rpms --clean <pkg> build-rpms <pkg>
(If this builds, you are good) Usually "devel" or "staticdev" packages are missing
================ How to verify a spec has correct Requires:
cd $MY_WORKSPACE
rm -Rf mock*
mock -r $MY_BUILD_CFG --install <full path to RPM> ie: mock -r $MY_BUILD_CFG --install rpmbuild/RPMS/python-keystoneclient-1.3.1-r3.noarch.rpm
======================
Thanks a lot for sharing this, save me a lot of time Regards
Al
-----Original Message----- From: Victor Rodriguez [mailto:vm.rod25@gmail.com] Sent: Wednesday, May 08, 2019 3:45 PM To: Little, Scott; starlingx-discuss@lists.starlingx.io; Cordoba Malibran, Erich Subject: [Starlingx-discuss] [Build][MultiOS] Use of current centOS spec files in Mock build system with repos
Hi team
A few weeks ago I had the curiosity to investigate if there was an rpm repo in CENGN for starling X. Scott was very kind to point me about, thanks a lot Scott. Why this is related to multi-os? RPM repositories are warehouses of RPM package files that are easy to consume to build with upstream tools like mock. mock is a tool for building RPM packages. we can use mock to build packages for many different versions of CentOS/Red Hat and Fedora.
After some experiments I was able to build basic packages from fault management using the following methodology:
1) set up mock in my Linux distro ( in my case centos 7.6 ) 2) use the following mock repositories config file to build: https://github.com/starlingx-staging/stx-packaging/blob/master/configs/docke... 3) build as: srpm : mock -r $(MOCK_CONFIG) --buildsrpm --spec=$(SPEC) --sources=$(SOURCES) --resultdir $(RES_DIR) --define="tis_patch_ver $(TIS_PATCH_VER)" rpm : mock -r $(MOCK_CONFIG) --rebuild $(SRPM) --resultdir $(RES_DIR) --define="tis_patch_ver $(TIS_PATCH_VER)"
( the variables point to the location of my spec file and source tar.gz files, the MOCK_CONFIG points to the path of my local-centos-7-x86_64.cfg file )
In the end, it works pretty straight forward, proving the following points from this experiment ( hopefully useful for someone else ) :
a) If we use the CENGN repos ( plus the centos repo ) is possible to build an RPM w/o the need to download the mirror. Great job Scott and build team! b) Is possible to use alternative upstream tools to build a single RPM w/o the need to rebuild the build dependencies.
Now a few questions came to me :
a ) Are these CENGN repos in active use, I mean, are they updated with a new ISO or every time a developer merge a commit in master? b) After trying to build all the packages in fault management I found that: * the source files directories are not standard ( I know Erich is working on this ) * there are missing BuildRequires in the current centos spec files ( they build due to the specific architecture that builds system has to handle/install the build req in a parallel build )
In the aim of improving the project to some other members of the community use different tools to build different layers of the OS. I was wonder how valuable the community see the task of adding the proper BuildRequires in all the flock spec files that have missing ones. I create a launchpad and a patch for the missing build req I found in fm-rest-api:
https://review.opendev.org/#/c/657869/
Any feedback from the community more than welcome
Regards
Victor Rodriguez
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (3)
-
Bailey, Henry Albert (Al)
-
Saul Wold
-
Victor Rodriguez