[Starlingx-discuss] how to deal with loop dependent in building stage
Hi， We are trying to construct basic environment for porting starlingX on Debian. While when I dealing with the build order of user space packages, the loop dependent problem blocked me. The easiest example: source package A build depend on B while B is also build depend on A. We can just build A, B, A, B and only use the later result. A fairly complex example: A depends on B and C, B depends on D, C depends on F, D depends on A and F. In this case there will be three cycles as below: A->B->D->A, A->B->D->F->A, A->C->F->A . Even more, if in some cases we needn't B or D, then we have only one cycle: ACFA I tried to find a method to deal with it but failed. So I wonder how did we deal with such loop dependent before, on CentOS. Any advises about it? Thanks a lot Xiao
Hi Xiao, May I know the package names? and arch of them? Btw, there is a way to analyse circular dependent packages. [1] Debian wiki has leff info about circular build but [2] has something. In case of installation of fedora/centos/rhel, rpm -Uvh a.rpm b.rpm will solve the problem. [1] https://wiki.debian.org/CircularBuildDependencies [2] https://www.debian.org/doc/debian-policy/ch-relationships.html#binary-depend... Regards, Danishka On Sat, Sep 18, 2021 at 12:24 PM Zhang, Xiao <xiao.zhang@windriver.com> wrote:
Hi，
We are trying to construct basic environment for porting starlingX on Debian. While when I dealing with the build order of user space packages, the loop dependent problem blocked me.
The easiest example: source package A build depend on B while B is also build depend on A. We can just build A, B, A, B and only use the later result.
A fairly complex example: A depends on B and C, B depends on D, C depends on F, D depends on A and F. In this case there will be three cycles as below:
A->B->D->A, A->B->D->F->A, A->C->F->A . Even more, if in some cases we needn't B or D, then we have only one cycle: ACFA
I tried to find a method to deal with it but failed.
So I wonder how did we deal with such loop dependent before, on CentOS. Any advises about it?
Thanks a lot
Xiao
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Xiao, I have checked the possibilities with Debian project. Here is the response from Debian dev to this issue. [1] [1] https://lists.debian.org/debian-devel/2021/09/msg00267.html Regards, Danishka On Sat, Sep 18, 2021 at 2:35 PM open infra <openinfradn@gmail.com> wrote:
Hi Xiao,
May I know the package names? and arch of them? Btw, there is a way to analyse circular dependent packages. [1] Debian wiki has leff info about circular build but [2] has something.
In case of installation of fedora/centos/rhel, rpm -Uvh a.rpm b.rpm will solve the problem.
[1] https://wiki.debian.org/CircularBuildDependencies [2] https://www.debian.org/doc/debian-policy/ch-relationships.html#binary-depend...
Regards, Danishka
On Sat, Sep 18, 2021 at 12:24 PM Zhang, Xiao <xiao.zhang@windriver.com> wrote:
Hi，
We are trying to construct basic environment for porting starlingX on Debian. While when I dealing with the build order of user space packages, the loop dependent problem blocked me.
The easiest example: source package A build depend on B while B is also build depend on A. We can just build A, B, A, B and only use the later result.
A fairly complex example: A depends on B and C, B depends on D, C depends on F, D depends on A and F. In this case there will be three cycles as below:
A->B->D->A, A->B->D->F->A, A->C->F->A . Even more, if in some cases we needn't B or D, then we have only one cycle: ACFA
I tried to find a method to deal with it but failed.
So I wonder how did we deal with such loop dependent before, on CentOS. Any advises about it?
Thanks a lot
Xiao
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 9/19/2021 1:48 PM, open infra wrote:
Hi Xiao,
I have checked the possibilities with Debian project. Here is the response from Debian dev to this issue. [1]
Great thanks for your kindly help! These resolutions are too heavy and complex for us. Developers have to find method shrink the build depend package list to break the loop. On my understanding, suppose source package A build depend on binary b, source package B build depend on binary package a. While both binary package a and b are released by Debian repository. Here we just add some packages into both A and B. I suppose we should 1) Build patched A based on original b and get a a-v1; 2) Use a-v1 build patched B get b-v1; 3) Use b-v1 build patched A again get a-v2; 4) Use a-v2 build patched B again get b-v2; Then we just abandon v1 packages and use the v2 ones. While the above is the easiest loop and seems not too hard to make it, but for some complex loop like I described in the first mail, it is really hard to make it. And, Of course if we have other packages have no original binary packages can be used, we have to break the loop dependency manually as bootstrap does. That will be painful. :-( Currently, we have no whole package list yet, thus we have no idea if we really have loop dependency in the future. So I wonder if our original StarlingX(based on CentOS) do have such loop dependency issue? And if it has, what they are how they resolved it? Thanks a lot! Xiao
[1] https://lists.debian.org/debian-devel/2021/09/msg00267.html <https://urldefense.com/v3/__https://lists.debian.org/debian-devel/2021/09/msg00267.html__;!!AjveYdw8EvQ!KaJIz1wDBOtemtxYBvDyNBy5e-CyetG_0MowjUmBQdDE1XaPDGiCqnMVHsnfNdnQwyB9$>
Regards, Danishka
On Sat, Sep 18, 2021 at 2:35 PM open infra <openinfradn@gmail.com> wrote:
Hi Xiao,
May I know the package names? and arch of them? Btw, there is a way to analyse circular dependent packages. [1] Debian wiki has leff info about circular build but [2] has something.
In case of installation of fedora/centos/rhel, rpm -Uvh a.rpm b.rpm will solve the problem.
[1] https://wiki.debian.org/CircularBuildDependencies <https://urldefense.com/v3/__https://wiki.debian.org/CircularBuildDependencies__;!!AjveYdw8EvQ!KaJIz1wDBOtemtxYBvDyNBy5e-CyetG_0MowjUmBQdDE1XaPDGiCqnMVHsnfNXNTqspa$> [2] https://www.debian.org/doc/debian-policy/ch-relationships.html#binary-depend... <https://urldefense.com/v3/__https://www.debian.org/doc/debian-policy/ch-relationships.html*binary-dependencies-depends-recommends-suggests-enhances-pre-depends__;Iw!!AjveYdw8EvQ!KaJIz1wDBOtemtxYBvDyNBy5e-CyetG_0MowjUmBQdDE1XaPDGiCqnMVHsnfNWiz_Q1Y$>
Regards, Danishka
On Sat, Sep 18, 2021 at 12:24 PM Zhang, Xiao <xiao.zhang@windriver.com> wrote:
Hi，
We are trying to construct basic environment for porting starlingX on Debian. While when I dealing with the build order of user space packages, the loop dependent problem blocked me.
The easiest example: source package A build depend on B while B is also build depend on A. We can just build A, B, A, B and only use the later result.
A fairly complex example: A depends on B and C, B depends on D, C depends on F, D depends on A and F. In this case there will be three cycles as below:
A->B->D->A, A->B->D->F->A, A->C->F->A . Even more, if in some cases we needn't B or D, then we have only one cycle: ACFA
I tried to find a method to deal with it but failed.
So I wonder how did we deal with such loop dependent before, on CentOS. Any advises about it?
Thanks a lot
Xiao
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss <https://urldefense.com/v3/__http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss__;!!AjveYdw8EvQ!KaJIz1wDBOtemtxYBvDyNBy5e-CyetG_0MowjUmBQdDE1XaPDGiCqnMVHsnfNRZQicM9$>
For centos, we found that loops almost always depend on one of a handful of low level packages, e.g. bash, python, gcc, rpm We had two ways to try and deal with this. 1) Allow the use of a pre-compiled binary from upstream to satisfy the dependency when the StarlingX modification are unlikely to affect how dependent packages compile. Packages in the 'mock' lst files could satisfy this type of requirement. e.g. A=bash So in your example, compile order would be: F (vs upstream A), D (vs upstream A), C, B, A 2) Use a different 'build-type'... other than 'std' or 'rt' ... when the modified StarlingX package was likely to affect the output of dependent packages. Compile packages in that build type first. e.g. A=rpm = build-type 'installer' compile order would be: A (vs upstream binaries) B and C ... call this A-intermediate = build-type 'std' compile order would be: F (vs A-intermediate), D (vs A-intermediate), C, B, A Scott On 2021-09-18 2:50 a.m., Zhang, Xiao wrote:
Hi，
We are trying to construct basic environment for porting starlingX on Debian. While when I dealing with the build order of user space packages, the loop dependent problem blocked me.
The easiest example: source package A build depend on B while B is also build depend on A. We can just build A, B, A, B and only use the later result.
A fairly complex example: A depends on B and C, B depends on D, C depends on F, D depends on A and F. In this case there will be three cycles as below:
A->B->D->A, A->B->D->F->A, A->C->F->A . Even more, if in some cases we needn't B or D, then we have only one cycle: ACFA
I tried to find a method to deal with it but failed.
So I wonder how did we deal with such loop dependent before, on CentOS. Any advises about it?
Thanks a lot
Xiao
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
For centos, we found that loops almost always depend on one of a handful of low level packages, e.g. bash, python, gcc, rpm
We had two ways to try and deal with this.
1) Allow the use of a pre-compiled binary from upstream to satisfy the dependency when the StarlingX modification are unlikely to affect how dependent packages compile. Packages in the 'mock' lst files could satisfy this type of requirement. e.g. A=bash So in your example, compile order would be: F (vs upstream A), D (vs upstream A), C, B, A
2) Use a different 'build-type'... other than 'std' or 'rt' ... when the modified StarlingX package was likely to affect the output of dependent packages. Compile packages in that build type first. e.g. A=rpm = build-type 'installer' compile order would be: A (vs upstream binaries) B and C ... call this A-intermediate ? So, the 'installer' type of A doesn't depend on B and C, right? Just
On 9/20/2021 9:39 PM, Scott Little wrote: like the official method to break the loop manually?
= build-type 'std' compile order would be: F (vs A-intermediate), D (vs A-intermediate), C, B, A
Seems the first one is more fit for automatic build. Then, any special cases thus we have to use the second method? Or we can always use the first one? Thanks Xiao
Scott
On 2021-09-18 2:50 a.m., Zhang, Xiao wrote:
Hi，
We are trying to construct basic environment for porting starlingX on Debian. While when I dealing with the build order of user space packages, the loop dependent problem blocked me.
The easiest example: source package A build depend on B while B is also build depend on A. We can just build A, B, A, B and only use the later result.
A fairly complex example: A depends on B and C, B depends on D, C depends on F, D depends on A and F. In this case there will be three cycles as below:
A->B->D->A, A->B->D->F->A, A->C->F->A . Even more, if in some cases we needn't B or D, then we have only one cycle: ACFA
I tried to find a method to deal with it but failed.
So I wonder how did we deal with such loop dependent before, on CentOS. Any advises about it?
Thanks a lot
Xiao
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Inline comments in blue Scott On 2021-09-21 9:07 a.m., Zhang, Xiao wrote:
For centos, we found that loops almost always depend on one of a handful of low level packages, e.g. bash, python, gcc, rpm
We had two ways to try and deal with this.
1) Allow the use of a pre-compiled binary from upstream to satisfy the dependency when the StarlingX modification are unlikely to affect how dependent packages compile. Packages in the 'mock' lst files could satisfy this type of requirement. e.g. A=bash So in your example, compile order would be: F (vs upstream A), D (vs upstream A), C, B, A
2) Use a different 'build-type'... other than 'std' or 'rt' ... when the modified StarlingX package was likely to affect the output of dependent packages. Compile packages in that build type first. e.g. A=rpm = build-type 'installer' compile order would be: A (vs upstream binaries) B and C ... call this A-intermediate ? So, the 'installer' type of A doesn't depend on B and C, right? Just
On 9/20/2021 9:39 PM, Scott Little wrote: like the official method to break the loop manually?
[SL] I wouldn't say that A doesn't depend on B or C. Rather the 'installer' package set does not build B or C, so they do not factor into build order calculations. This does require that upstream B and C be available to satisfy build dependencies when compiling within the 'installer' package set. When compiling the 'std' set, but the packages from the 'installer' package must be available to satisfy dependencies. The 'installer' compiled versions of B and C must be of higher version that those from upstream... and thus installed instead of the upstream ones.
= build-type 'std' compile order would be: F (vs A-intermediate), D (vs A-intermediate), C, B, A
Seems the first one is more fit for automatic build. Then, any special cases thus we have to use the second method? Or we can always use the first one?
[SL] You'll need to support both methods of resolving dependency cycles. Note that the layered build concept of the CentOS build was another way to address partitioning of packages into sets that might help with breaking dependency loops by not having A,B,C in the same compile set. Look to the 'compiler' layer. I think you need to support at least one of the 'build-type' or 'build layer' concepts to begin with. Bonus points if you support both. Supporting both was needed for the 'rpm' package which we wanted to patch, and the patch does alter the format of the rpms generated, and the rpm package itself had to be in that format. So rpm was compiled for build-type 'installer' of layer 'compiler' (new rpm code in old rpm package), then recompiled in build-type 'std' of layer 'compiler' (new rpm code in new rpm package). Finally the std build of rpm was published for use by higher layer builds. Only designers working on packages in the 'installer' set, or the 'compiler' layer, need to worry about the multi pass build. For most packages, a single pass 'flock' layer, 'std' build-type, build is fine. Scott
Thanks
Xiao
Scott
On 2021-09-18 2:50 a.m., Zhang, Xiao wrote:
Hi，
We are trying to construct basic environment for porting starlingX on Debian. While when I dealing with the build order of user space packages, the loop dependent problem blocked me.
The easiest example: source package A build depend on B while B is also build depend on A. We can just build A, B, A, B and only use the later result.
A fairly complex example: A depends on B and C, B depends on D, C depends on F, D depends on A and F. In this case there will be three cycles as below:
A->B->D->A, A->B->D->F->A, A->C->F->A . Even more, if in some cases we needn't B or D, then we have only one cycle: ACFA
I tried to find a method to deal with it but failed.
So I wonder how did we deal with such loop dependent before, on CentOS. Any advises about it?
Thanks a lot
Xiao
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Comments inline: ...
For centos, we found that loops almost always depend on one of a handful of low level packages, e.g. bash, python, gcc, rpm
We had two ways to try and deal with this.
1) Allow the use of a pre-compiled binary from upstream to satisfy the dependency when the StarlingX modification are unlikely to affect how dependent packages compile. Packages in the 'mock' lst files could satisfy this type of requirement. e.g. A=bash So in your example, compile order would be: F (vs upstream A), D (vs upstream A), C, B, A
2) Use a different 'build-type'... other than 'std' or 'rt' ... when the modified StarlingX package was likely to affect the output of dependent packages. Compile packages in that build type first. e.g. A=rpm = build-type 'installer' compile order would be: A (vs upstream binaries) B and C ... call this A-intermediate
? So, the 'installer' type of A doesn't depend on B and C, right? Just like the official method to break the loop manually?
[SL] I wouldn't say that A doesn't depend on B or C. Rather the 'installer' package set does not build B or C, so they do not factor into build order calculations. This does require that upstream B and C be available to satisfy build dependencies when compiling within the 'installer' package set. When compiling the 'std' set, but the packages from the 'installer' package must be available to satisfy dependencies. The 'installer' compiled versions of B and C must be of higher version that those from upstream... and thus installed instead of the upstream ones.
So? In this kind of case, we also needn't break the loop manually but just separate the into different layers/sets and compile them in different stage, right? In the bottom logic(or, with the compiler's point of view), it is the same with the first one. The key difference comes from the special package A. It's so basic that it has to be compiled tice
= build-type 'std' compile order would be: F (vs A-intermediate), D (vs A-intermediate), C, B, A
Seems the first one is more fit for automatic build. Then, any special cases thus we have to use the second method? Or we can always use the first one?
[SL] You'll need to support both methods of resolving dependency cycles.
Note that the layered build concept of the CentOS build was another way to address partitioning of packages into sets that might help with breaking dependency loops by not having A,B,C in the same compile set. Look to the 'compiler' layer.
I think you need to support at least one of the 'build-type' or 'build layer' concepts to begin with. Bonus points if you support both. Supporting both was needed for the 'rpm' package which we wanted to patch, and the patch does alter the format of the rpms generated, and the rpm package itself had to be in that format. So rpm was compiled for build-type 'installer' of layer 'compiler' (new rpm code in old rpm package), then recompiled in build-type 'std' of layer 'compiler' (new rpm code in new rpm package). Finally the std build of rpm was published for use by higher layer builds.
Only designers working on packages in the 'installer' set, or the 'compiler' layer, need to worry about the multi pass build. For most packages, a single pass 'flock' layer, 'std' build-type, build is fine.
So, in build process, we need to build "compiler" layer firstly, then use the new build "compilers" build other layers. Some special packages like "rpm" may exist in more than one layers/sets. BTW: I suppose the upstream rpm is already powerful enough, why we have to modify it? Just bug fix or new features? If the later one, StarlingX need some special features it hasn't? It maybe very useful/important for us in porting StarlingX onto other Linux releases. Thanks Xiao
Scott
Thanks
Xiao
Scott
On 2021-09-18 2:50 a.m., Zhang, Xiao wrote:
Hi，
We are trying to construct basic environment for porting starlingX on Debian. While when I dealing with the build order of user space packages, the loop dependent problem blocked me.
The easiest example: source package A build depend on B while B is also build depend on A. We can just build A, B, A, B and only use the later result.
A fairly complex example: A depends on B and C, B depends on D, C depends on F, D depends on A and F. In this case there will be three cycles as below:
A->B->D->A, A->B->D->F->A, A->C->F->A . Even more, if in some cases we needn't B or D, then we have only one cycle: ACFA
I tried to find a method to deal with it but failed.
So I wonder how did we deal with such loop dependent before, on CentOS. Any advises about it?
Thanks a lot
Xiao
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
There was a security feature (not sure it's name or if it's still supported) that required a new rpm feature not available in the standard CentOS rpm (file signatures). The behavior of file signing was further modified by 0001-sign-files-only.patch Greg, can you comment on this? Do the Debian packaging tools need to support file signatures? Scott On 2021-09-23 4:56 a.m., Zhang, Xiao wrote:
Comments inline:
...
For centos, we found that loops almost always depend on one of a handful of low level packages, e.g. bash, python, gcc, rpm
We had two ways to try and deal with this.
1) Allow the use of a pre-compiled binary from upstream to satisfy the dependency when the StarlingX modification are unlikely to affect how dependent packages compile. Packages in the 'mock' lst files could satisfy this type of requirement. e.g. A=bash So in your example, compile order would be: F (vs upstream A), D (vs upstream A), C, B, A
2) Use a different 'build-type'... other than 'std' or 'rt' ... when the modified StarlingX package was likely to affect the output of dependent packages. Compile packages in that build type first. e.g. A=rpm = build-type 'installer' compile order would be: A (vs upstream binaries) B and C ... call this A-intermediate
? So, the 'installer' type of A doesn't depend on B and C, right? Just like the official method to break the loop manually?
[SL] I wouldn't say that A doesn't depend on B or C. Rather the 'installer' package set does not build B or C, so they do not factor into build order calculations. This does require that upstream B and C be available to satisfy build dependencies when compiling within the 'installer' package set. When compiling the 'std' set, but the packages from the 'installer' package must be available to satisfy dependencies. The 'installer' compiled versions of B and C must be of higher version that those from upstream... and thus installed instead of the upstream ones.
So? In this kind of case, we also needn't break the loop manually but just separate the into different layers/sets and compile them in different stage, right?
In the bottom logic(or, with the compiler's point of view), it is the same with the first one. The key difference comes from the special package A. It's so basic that it has to be compiled tice
= build-type 'std' compile order would be: F (vs A-intermediate), D (vs A-intermediate), C, B, A
Seems the first one is more fit for automatic build. Then, any special cases thus we have to use the second method? Or we can always use the first one?
[SL] You'll need to support both methods of resolving dependency cycles.
Note that the layered build concept of the CentOS build was another way to address partitioning of packages into sets that might help with breaking dependency loops by not having A,B,C in the same compile set. Look to the 'compiler' layer.
I think you need to support at least one of the 'build-type' or 'build layer' concepts to begin with. Bonus points if you support both. Supporting both was needed for the 'rpm' package which we wanted to patch, and the patch does alter the format of the rpms generated, and the rpm package itself had to be in that format. So rpm was compiled for build-type 'installer' of layer 'compiler' (new rpm code in old rpm package), then recompiled in build-type 'std' of layer 'compiler' (new rpm code in new rpm package). Finally the std build of rpm was published for use by higher layer builds.
Only designers working on packages in the 'installer' set, or the 'compiler' layer, need to worry about the multi pass build. For most packages, a single pass 'flock' layer, 'std' build-type, build is fine.
So, in build process, we need to build "compiler" layer firstly, then use the new build "compilers" build other layers. Some special packages like "rpm" may exist in more than one layers/sets.
BTW: I suppose the upstream rpm is already powerful enough, why we have to modify it? Just bug fix or new features? If the later one, StarlingX need some special features it hasn't? It maybe very useful/important for us in porting StarlingX onto other Linux releases.
Thanks
Xiao
Scott
Thanks
Xiao
Scott
On 2021-09-18 2:50 a.m., Zhang, Xiao wrote:
Hi，
We are trying to construct basic environment for porting starlingX on Debian. While when I dealing with the build order of user space packages, the loop dependent problem blocked me.
The easiest example: source package A build depend on B while B is also build depend on A. We can just build A, B, A, B and only use the later result.
A fairly complex example: A depends on B and C, B depends on D, C depends on F, D depends on A and F. In this case there will be three cycles as below:
A->B->D->A, A->B->D->F->A, A->C->F->A . Even more, if in some cases we needn't B or D, then we have only one cycle: ACFA
I tried to find a method to deal with it but failed.
So I wonder how did we deal with such loop dependent before, on CentOS. Any advises about it?
Thanks a lot
Xiao
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Linux IMA support was the name of the feature I believe. This link would seem to suggest that deb does not support file signatures as a standard feature ... https://wiki.debian.org/Teams/Dpkg/Spec/DebSignatures ... although some folks might be working on it. Scott On 2021-09-23 10:08 a.m., Scott Little wrote:
There was a security feature (not sure it's name or if it's still supported) that required a new rpm feature not available in the standard CentOS rpm (file signatures). The behavior of file signing was further modified by 0001-sign-files-only.patch
Greg, can you comment on this? Do the Debian packaging tools need to support file signatures?
Scott
On 2021-09-23 4:56 a.m., Zhang, Xiao wrote:
Comments inline:
...
For centos, we found that loops almost always depend on one of a handful of low level packages, e.g. bash, python, gcc, rpm
We had two ways to try and deal with this.
1) Allow the use of a pre-compiled binary from upstream to satisfy the dependency when the StarlingX modification are unlikely to affect how dependent packages compile. Packages in the 'mock' lst files could satisfy this type of requirement. e.g. A=bash So in your example, compile order would be: F (vs upstream A), D (vs upstream A), C, B, A
2) Use a different 'build-type'... other than 'std' or 'rt' ... when the modified StarlingX package was likely to affect the output of dependent packages. Compile packages in that build type first. e.g. A=rpm = build-type 'installer' compile order would be: A (vs upstream binaries) B and C ... call this A-intermediate
? So, the 'installer' type of A doesn't depend on B and C, right? Just like the official method to break the loop manually?
[SL] I wouldn't say that A doesn't depend on B or C. Rather the 'installer' package set does not build B or C, so they do not factor into build order calculations. This does require that upstream B and C be available to satisfy build dependencies when compiling within the 'installer' package set. When compiling the 'std' set, but the packages from the 'installer' package must be available to satisfy dependencies. The 'installer' compiled versions of B and C must be of higher version that those from upstream... and thus installed instead of the upstream ones.
So? In this kind of case, we also needn't break the loop manually but just separate the into different layers/sets and compile them in different stage, right?
In the bottom logic(or, with the compiler's point of view), it is the same with the first one. The key difference comes from the special package A. It's so basic that it has to be compiled tice
= build-type 'std' compile order would be: F (vs A-intermediate), D (vs A-intermediate), C, B, A
Seems the first one is more fit for automatic build. Then, any special cases thus we have to use the second method? Or we can always use the first one?
[SL] You'll need to support both methods of resolving dependency cycles.
Note that the layered build concept of the CentOS build was another way to address partitioning of packages into sets that might help with breaking dependency loops by not having A,B,C in the same compile set. Look to the 'compiler' layer.
I think you need to support at least one of the 'build-type' or 'build layer' concepts to begin with. Bonus points if you support both. Supporting both was needed for the 'rpm' package which we wanted to patch, and the patch does alter the format of the rpms generated, and the rpm package itself had to be in that format. So rpm was compiled for build-type 'installer' of layer 'compiler' (new rpm code in old rpm package), then recompiled in build-type 'std' of layer 'compiler' (new rpm code in new rpm package). Finally the std build of rpm was published for use by higher layer builds.
Only designers working on packages in the 'installer' set, or the 'compiler' layer, need to worry about the multi pass build. For most packages, a single pass 'flock' layer, 'std' build-type, build is fine.
So, in build process, we need to build "compiler" layer firstly, then use the new build "compilers" build other layers. Some special packages like "rpm" may exist in more than one layers/sets.
BTW: I suppose the upstream rpm is already powerful enough, why we have to modify it? Just bug fix or new features? If the later one, StarlingX need some special features it hasn't? It maybe very useful/important for us in porting StarlingX onto other Linux releases.
Thanks
Xiao
Scott
Thanks
Xiao
Scott
On 2021-09-18 2:50 a.m., Zhang, Xiao wrote:
Hi，
We are trying to construct basic environment for porting starlingX on Debian. While when I dealing with the build order of user space packages, the loop dependent problem blocked me.
The easiest example: source package A build depend on B while B is also build depend on A. We can just build A, B, A, B and only use the later result.
A fairly complex example: A depends on B and C, B depends on D, C depends on F, D depends on A and F. In this case there will be three cycles as below:
A->B->D->A, A->B->D->F->A, A->C->F->A . Even more, if in some cases we needn't B or D, then we have only one cycle: ACFA
I tried to find a method to deal with it but failed.
So I wonder how did we deal with such loop dependent before, on CentOS. Any advises about it?
Thanks a lot
Xiao
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 2021-09-23 10:08:41 -0400 (-0400), Scott Little wrote: [...]
Do the Debian packaging tools need to support file signatures? [...]
In Debian more generally, packages (these days mainly just source packages) are signed by their maintainers or uploaders as a sort of authentication to the build and distribution infrastructure the project maintains, but those signatures aren't really distributed to package consumers and may be irrelevant to your effort. If you do want to sign source packages and changes files at build time, you can supply signing keys to any of the usual package build tools (debuild, gbp, et cetera), but I wouldn't bother unless your solution assumes that the rebuilders and indexers can't trust the mechanisms by which those packages are moved between those systems. It's important for Debian proper because they have thousands of people uploading packages and they want to be sure the packages are really from authorized individuals, but that's a lot different from how StarlingX's packages are likely to be supplied. The apt-secure(8) manpage on any modern Debian system explains the consumer-facing archive signatures, which are over the package indices instead of individual packages, as those include lists of checksums for each package being distributed. Any time the set of packages/versions within the archive changes, the index is regenerated and signed again with the published archive key(s). This is what users installing software from the distribution rely on to be sure the packages they're getting are really the ones the distribution is serving and that they haven't been tampered with in transit. The package management tooling verifies the signatures on package indices any time it updates them, and then relies on the checksums included in each signed index to confirm a particular package is authentic. I have no idea how much of this is useful for you, but hopefully it helps. -- Jeremy Stanley
Thanks for the reply Jeremy, but I think your discussing the signing of packages as a whole, not the signing of files within packages. I'm trying to learn a bit about how Linux IMA works... The point of the feature is to have the executable files within the package (deb or rpm) carry signatures that the Linux kernel will validate before allowing the executable to run. I'm still a little fuzzy on why the signatures on the executables need to be injected as part of packaging rather than building. Is that a requirement for the chain of trust, or a conveniently centralized place to apply the policy to all packages? And what does that mean for packages not directly built by StarlingX? Perhaps it's moot. I'm pretty sure the IMA feature is not currently supported within StarlingX. I'm hoping Greg can step in and confirm. Scott On 2021-09-23 10:43 a.m., Jeremy Stanley wrote:
[Please note: This e-mail is from an EXTERNAL e-mail address]
On 2021-09-23 10:08:41 -0400 (-0400), Scott Little wrote: [...]
Do the Debian packaging tools need to support file signatures? [...]
In Debian more generally, packages (these days mainly just source packages) are signed by their maintainers or uploaders as a sort of authentication to the build and distribution infrastructure the project maintains, but those signatures aren't really distributed to package consumers and may be irrelevant to your effort. If you do want to sign source packages and changes files at build time, you can supply signing keys to any of the usual package build tools (debuild, gbp, et cetera), but I wouldn't bother unless your solution assumes that the rebuilders and indexers can't trust the mechanisms by which those packages are moved between those systems. It's important for Debian proper because they have thousands of people uploading packages and they want to be sure the packages are really from authorized individuals, but that's a lot different from how StarlingX's packages are likely to be supplied.
The apt-secure(8) manpage on any modern Debian system explains the consumer-facing archive signatures, which are over the package indices instead of individual packages, as those include lists of checksums for each package being distributed. Any time the set of packages/versions within the archive changes, the index is regenerated and signed again with the published archive key(s). This is what users installing software from the distribution rely on to be sure the packages they're getting are really the ones the distribution is serving and that they haven't been tampered with in transit. The package management tooling verifies the signatures on package indices any time it updates them, and then relies on the checksums included in each signed index to confirm a particular package is authentic.
I have no idea how much of this is useful for you, but hopefully it helps. -- Jeremy Stanley
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 2021-09-23 11:13:32 -0400 (-0400), Scott Little wrote:
Thanks for the reply Jeremy, but I think your discussing the signing of packages as a whole, not the signing of files within packages.
I'm trying to learn a bit about how Linux IMA works...
Oh, thanks, I did indeed misread and thought you were asking about an equivalent for the signing of .rpm files (packages), rather than files within the packages for consumption by the kernel's integrity measurement. Yes very different topics.
The point of the feature is to have the executable files within the package (deb or rpm) carry signatures that the Linux kernel will validate before allowing the executable to run.
I'm still a little fuzzy on why the signatures on the executables need to be injected as part of packaging rather than building. Is that a requirement for the chain of trust, or a conveniently centralized place to apply the policy to all packages? And what does that mean for packages not directly built by StarlingX? [...]
I'm a little fuzzy on the distinction between packaging and building, since package building (in the Debian context at least) drives executable compilation via policy included within the source package. Anyway, this looks like a relevant feature request for it along with a proof of concept implementation (albeit with 7 years of cobwebs): https://bugs.debian.org/766267 https://lists.debian.org/debian-dpkg/2014/08/threads.html#00006 The implementation looks fairly straightforward, the file signatures would be generated when the packages are assembled and stored with the file checksums normally tracked, then at installation those signatures would be copied into extended filesystem attributes for the relevant files, to be consumed by the kernel. If this is of interest to the StarlingX community, it might be an activity worth reigniting in Debian. It looks like the developers at IBM who originally proposed support for it did not pursue it further, at least that I can find any record of. -- Jeremy Stanley
Thanks for the pointers Scott On 2021-09-23 12:11 p.m., Jeremy Stanley wrote:
[Please note: This e-mail is from an EXTERNAL e-mail address]
On 2021-09-23 11:13:32 -0400 (-0400), Scott Little wrote:
Thanks for the reply Jeremy, but I think your discussing the signing of packages as a whole, not the signing of files within packages.
I'm trying to learn a bit about how Linux IMA works... Oh, thanks, I did indeed misread and thought you were asking about an equivalent for the signing of .rpm files (packages), rather than files within the packages for consumption by the kernel's integrity measurement. Yes very different topics.
The point of the feature is to have the executable files within the package (deb or rpm) carry signatures that the Linux kernel will validate before allowing the executable to run.
I'm still a little fuzzy on why the signatures on the executables need to be injected as part of packaging rather than building. Is that a requirement for the chain of trust, or a conveniently centralized place to apply the policy to all packages? And what does that mean for packages not directly built by StarlingX? [...]
I'm a little fuzzy on the distinction between packaging and building, since package building (in the Debian context at least) drives executable compilation via policy included within the source package.
Anyway, this looks like a relevant feature request for it along with a proof of concept implementation (albeit with 7 years of cobwebs):
https://bugs.debian.org/766267 https://lists.debian.org/debian-dpkg/2014/08/threads.html#00006
The implementation looks fairly straightforward, the file signatures would be generated when the packages are assembled and stored with the file checksums normally tracked, then at installation those signatures would be copied into extended filesystem attributes for the relevant files, to be consumed by the kernel.
If this is of interest to the StarlingX community, it might be an activity worth reigniting in Debian. It looks like the developers at IBM who originally proposed support for it did not pursue it further, at least that I can find any record of. -- Jeremy Stanley
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 2021-09-23 16:11:25 +0000 (+0000), Jeremy Stanley wrote: [...]
Anyway, this looks like a relevant feature request for it along with a proof of concept implementation (albeit with 7 years of cobwebs):
https://bugs.debian.org/766267 https://lists.debian.org/debian-dpkg/2014/08/threads.html#00006
The implementation looks fairly straightforward, the file signatures would be generated when the packages are assembled and stored with the file checksums normally tracked, then at installation those signatures would be copied into extended filesystem attributes for the relevant files, to be consumed by the kernel.
If this is of interest to the StarlingX community, it might be an activity worth reigniting in Debian. It looks like the developers at IBM who originally proposed support for it did not pursue it further, at least that I can find any record of.
A thread on the oss-security mailing list this week reminded me of the earlier discussion here. Interestingly, it's a developer at Red Hat questioning the (f)utility of IMA signatures on files: https://www.openwall.com/lists/oss-security/2021/11/30/1 -- Jeremy Stanley
Thanks for the link. Good read. Scott On 2021-12-01 2:53 p.m., Jeremy Stanley wrote:
[Please note: This e-mail is from an EXTERNAL e-mail address]
Anyway, this looks like a relevant feature request for it along with a proof of concept implementation (albeit with 7 years of cobwebs):
https://bugs.debian.org/766267 https://lists.debian.org/debian-dpkg/2014/08/threads.html#00006
The implementation looks fairly straightforward, the file signatures would be generated when the packages are assembled and stored with the file checksums normally tracked, then at installation those signatures would be copied into extended filesystem attributes for the relevant files, to be consumed by the kernel.
If this is of interest to the StarlingX community, it might be an activity worth reigniting in Debian. It looks like the developers at IBM who originally proposed support for it did not pursue it further, at least that I can find any record of. A thread on the oss-security mailing list this week reminded me of
On 2021-09-23 16:11:25 +0000 (+0000), Jeremy Stanley wrote: [...] the earlier discussion here. Interestingly, it's a developer at Red Hat questioning the (f)utility of IMA signatures on files:
https://www.openwall.com/lists/oss-security/2021/11/30/1
-- Jeremy Stanley
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Correct, IMA is not supported in StarlingX. Greg. From: Scott Little <scott.little@windriver.com> Sent: Thursday, September 23, 2021 11:14 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] how to deal with loop dependent in building stage Thanks for the reply Jeremy, but I think your discussing the signing of packages as a whole, not the signing of files within packages. I'm trying to learn a bit about how Linux IMA works... The point of the feature is to have the executable files within the package (deb or rpm) carry signatures that the Linux kernel will validate before allowing the executable to run. I'm still a little fuzzy on why the signatures on the executables need to be injected as part of packaging rather than building. Is that a requirement for the chain of trust, or a conveniently centralized place to apply the policy to all packages? And what does that mean for packages not directly built by StarlingX? Perhaps it's moot. I'm pretty sure the IMA feature is not currently supported within StarlingX. I'm hoping Greg can step in and confirm. Scott On 2021-09-23 10:43 a.m., Jeremy Stanley wrote: [Please note: This e-mail is from an EXTERNAL e-mail address] On 2021-09-23 10:08:41 -0400 (-0400), Scott Little wrote: [...] Do the Debian packaging tools need to support file signatures? [...] In Debian more generally, packages (these days mainly just source packages) are signed by their maintainers or uploaders as a sort of authentication to the build and distribution infrastructure the project maintains, but those signatures aren't really distributed to package consumers and may be irrelevant to your effort. If you do want to sign source packages and changes files at build time, you can supply signing keys to any of the usual package build tools (debuild, gbp, et cetera), but I wouldn't bother unless your solution assumes that the rebuilders and indexers can't trust the mechanisms by which those packages are moved between those systems. It's important for Debian proper because they have thousands of people uploading packages and they want to be sure the packages are really from authorized individuals, but that's a lot different from how StarlingX's packages are likely to be supplied. The apt-secure(8) manpage on any modern Debian system explains the consumer-facing archive signatures, which are over the package indices instead of individual packages, as those include lists of checksums for each package being distributed. Any time the set of packages/versions within the archive changes, the index is regenerated and signed again with the published archive key(s). This is what users installing software from the distribution rely on to be sure the packages they're getting are really the ones the distribution is serving and that they haven't been tampered with in transit. The package management tooling verifies the signatures on package indices any time it updates them, and then relies on the checksums included in each signed index to confirm a particular package is authentic. I have no idea how much of this is useful for you, but hopefully it helps. -- Jeremy Stanley _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss<https://urldefense.com/v3/__http:/lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss__;!!AjveYdw8EvQ!PLgEuvehtMaSlbjG9M_B5XUJRDFWNa6G-BTL2umbfcHz5qTq27RNcsTY4SIWdWTbSPs$>
participants (5)
-
Jeremy Stanley
-
open infra
-
Scott Little
-
Waines, Greg
-
Zhang, Xiao