[Starlingx-discuss] Understanding package building order
Hi all, I'm trying to understand how the build system decides what to build on specific scenarios. Let's say we have a complete build and then we run: $ build-pkgs bash This will rebuild the bash package, but also the build system decides to delete and rebuild a set of packages that I don't understand why are being rebuild, for example: iptables-1.4.21-18.0.1.el7.tis.3.x86_64.rpm kernel-3.10.0-862.6.3.el7.35.tis.x86_64.rpm qemu-kvm-common-ev-2.10.0-0.tis.0.x86_64.rpm sm-common-libs-1.0.0-19.tis.x86_64.rpm and a big list of packages. So, the main question here is why the build system decides to rebuild a kernel (taking an example) in order to rebuild bash. Thank you, -Erich
The default behaviour when building pkg A, is to also build anything in the BuildRequires of pkg A, and the requires of those packages, transitively until it has a full set. It then adds packages that have a direct BuildRequires on A. We do this to protect compiles in languages like C/C++ where a include file change in one package can have major impacts on the compile of other packages that uses the .h Paranoia by default e.g. A BuildRequires B and C, and C requires D, and E and F requires A E F \ / A / \ B C | D so the build set is: D B C A E F. This would also be the default build order for this simple case A, E and F as cleaned to force a rebuild. D, B and C are only built it not already built. The unfortunate part is that RPM BuildRequires doesn't distinguish a source code requirement from a tool requirement. So some packages could probably be excluded. An advisory in build_srpm.data to ignore some dependencies is a future enhancement I've been pondering. Another possibility is to only follow dependencies with -dev in the name.... not sure if that is 100% correct though. BuildRequires/Requires are also littered with dependency loops. This can play havoc with trying to find an optimal build order. Some times it's a case of try to build something, and if it fails, try something else and circle back with a second or third iteration if at least one package built ok. There were a dependency cache files in cgts-tis-repo that helped find good build orders, but StarlingX dropped that content last time I looked. Without it, I expect the iteration counts to get quite large, wasting a lot of time. One of nastiest dependency loops includes bash and the linux kernel. To exclude E and F from our hypothetical build ... build-pkgs --no-descendants A To exclude B C D from our hypothetical build ... build-pkgs --no-required A Finally we always add 'build-info' to the build list, and it may have descendants. To suppress this ... build-pkgs --no-build-info A Put it all together, for a fast build of A when dependencies are not a concern ... (e.g. resolving build issues internal to pkg A) ... build-pkgs --no-descendants --no-required --no-build-info A On 18-07-24 12:11 PM, Cordoba Malibran, Erich wrote:
Hi all,
I'm trying to understand how the build system decides what to build on specific scenarios. Let's say we have a complete build and then we run:
$ build-pkgs bash
This will rebuild the bash package, but also the build system decides to delete and rebuild a set of packages that I don't understand why are being rebuild, for example:
iptables-1.4.21-18.0.1.el7.tis.3.x86_64.rpm kernel-3.10.0-862.6.3.el7.35.tis.x86_64.rpm qemu-kvm-common-ev-2.10.0-0.tis.0.x86_64.rpm sm-common-libs-1.0.0-19.tis.x86_64.rpm
and a big list of packages. So, the main question here is why the build system decides to rebuild a kernel (taking an example) in order to rebuild bash.
Thank you,
-Erich _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Scott, thank you, this is super helpful. Is this covered in any of the StarlingX documents out on the wiki? If not, can someone add it please? brucej -----Original Message----- From: Scott Little [mailto:scott.little@windriver.com] Sent: Tuesday, July 24, 2018 9:57 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Understanding package building order The default behaviour when building pkg A, is to also build anything in the BuildRequires of pkg A, and the requires of those packages, transitively until it has a full set. It then adds packages that have a direct BuildRequires on A. We do this to protect compiles in languages like C/C++ where a include file change in one package can have major impacts on the compile of other packages that uses the .h Paranoia by default e.g. A BuildRequires B and C, and C requires D, and E and F requires A E F \ / A / \ B C | D so the build set is: D B C A E F. This would also be the default build order for this simple case A, E and F as cleaned to force a rebuild. D, B and C are only built it not already built. The unfortunate part is that RPM BuildRequires doesn't distinguish a source code requirement from a tool requirement. So some packages could probably be excluded. An advisory in build_srpm.data to ignore some dependencies is a future enhancement I've been pondering. Another possibility is to only follow dependencies with -dev in the name.... not sure if that is 100% correct though. BuildRequires/Requires are also littered with dependency loops. This can play havoc with trying to find an optimal build order. Some times it's a case of try to build something, and if it fails, try something else and circle back with a second or third iteration if at least one package built ok. There were a dependency cache files in cgts-tis-repo that helped find good build orders, but StarlingX dropped that content last time I looked. Without it, I expect the iteration counts to get quite large, wasting a lot of time. One of nastiest dependency loops includes bash and the linux kernel. To exclude E and F from our hypothetical build ... build-pkgs --no-descendants A To exclude B C D from our hypothetical build ... build-pkgs --no-required A Finally we always add 'build-info' to the build list, and it may have descendants. To suppress this ... build-pkgs --no-build-info A Put it all together, for a fast build of A when dependencies are not a concern ... (e.g. resolving build issues internal to pkg A) ... build-pkgs --no-descendants --no-required --no-build-info A On 18-07-24 12:11 PM, Cordoba Malibran, Erich wrote:
Hi all,
I'm trying to understand how the build system decides what to build on specific scenarios. Let's say we have a complete build and then we run:
$ build-pkgs bash
This will rebuild the bash package, but also the build system decides to delete and rebuild a set of packages that I don't understand why are being rebuild, for example:
iptables-1.4.21-18.0.1.el7.tis.3.x86_64.rpm kernel-3.10.0-862.6.3.el7.35.tis.x86_64.rpm qemu-kvm-common-ev-2.10.0-0.tis.0.x86_64.rpm sm-common-libs-1.0.0-19.tis.x86_64.rpm
and a big list of packages. So, the main question here is why the build system decides to rebuild a kernel (taking an example) in order to rebuild bash.
Thank you,
-Erich _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Scott, thank you, this is super helpful.
Pure gold!
Is this covered in any of the StarlingX documents out on the wiki? If not, can someone add it please?
I think we already have the right place for this information, some other shared in the past and in the future, all related to our Build System [0]. Please be patient so we can have a first draft by the end of next week. [0] https://wiki.openstack.org/wiki/StarlingX/Developer_Guide/Build_System
Thanks Scott! This was a really good explanation :)
The default behaviour when building pkg A, is to also build anything in the BuildRequires of pkg A, and the requires of those packages, transitively until it has a full set. It then adds packages that have a direct BuildRequires on A.
We do this to protect compiles in languages like C/C++ where a include file change in one package can have major impacts on the compile of other packages that uses the .h
Paranoia by default
e.g.
A BuildRequires B and C, and C requires D, and E and F requires A
E F \ / A / \ B C | D
so the build set is: D B C A E F. This would also be the default build order for this simple case A, E and F as cleaned to force a rebuild. D, B and C are only built it not already built.
The unfortunate part is that RPM BuildRequires doesn't distinguish a source code requirement from a tool requirement. So some packages could probably be excluded. An advisory in build_srpm.data to ignore some dependencies is a future enhancement I've been pondering. Another possibility is to only follow dependencies with -dev in the name.... not sure if that is 100% correct though.
BuildRequires/Requires are also littered with dependency loops. This can play havoc with trying to find an optimal build order. Some times it's a case of try to build something, and if it fails, try something else and circle back with a second or third iteration if at least one package built ok. There were a dependency cache files in cgts-tis-repo that helped find good build orders, but StarlingX dropped that content last time I looked. Without it, I expect the iteration counts to get quite large, wasting a lot of time.
One of nastiest dependency loops includes bash and the linux kernel.
To exclude E and F from our hypothetical build ... build-pkgs --no-descendants A
To exclude B C D from our hypothetical build ... build-pkgs --no-required A
Finally we always add 'build-info' to the build list, and it may have descendants. To suppress this ... build-pkgs --no-build-info A
Put it all together, for a fast build of A when dependencies are not a concern ... (e.g. resolving build issues internal to pkg A) ... build-pkgs --no-descendants --no-required --no-build-info A
On 18-07-24 12:11 PM, Cordoba Malibran, Erich wrote:
Hi all,
I'm trying to understand how the build system decides what to build on specific scenarios. Let's say we have a complete build and then we run:
$ build-pkgs bash
This will rebuild the bash package, but also the build system decides to delete and rebuild a set of packages that I don't understand why are being rebuild, for example:
iptables-1.4.21-18.0.1.el7.tis.3.x86_64.rpm kernel-3.10.0-862.6.3.el7.35.tis.x86_64.rpm qemu-kvm-common-ev-2.10.0-0.tis.0.x86_64.rpm sm-common-libs-1.0.0-19.tis.x86_64.rpm
and a big list of packages. So, the main question here is why the build system decides to rebuild a kernel (taking an example) in order to rebuild bash.
Thank you,
-Erich _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus s
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (4)
-
Arce Moreno, Abraham
-
Cordoba Malibran, Erich
-
Jones, Bruce E
-
Scott Little