[Starlingx-discuss] Recommended C/C++ compiler flag for security
Hi StarlingX community We can all agree that security is an important feature to be taken into consideration in any SW project. In the aim of improving the security of the StarlingX project, we have been taking the task to propose the use of some compiler flags that prevent and detect some security holes, especially by buffer overflow that could lead into ROP attacks. The list of flags that we are proposing are : Stack-based Buffer Overrun Detection: CFLAGS=”-fstack-protector-strong” Fortify source: CFLAGS="-O2 -D_FORTIFY_SOURCE=2" Format string vulnerabilities: CFLAGS="-Wformat -Wformat-security" Stack execution protection: LDFLAGS="-z noexecstack" Data relocation and protection (RELRO): LDLFAGS="-z relro -z now" These are being analyzed in the following Gerrit reviews (thanks a lot for all the good feedback) https://review.openstack.org/#/c/623608/ https://review.openstack.org/#/c/623603/ https://review.openstack.org/#/c/623601/ https://review.openstack.org/#/c/623599/ As requested in the Gerrit reviews, there is a proper need to first understand what these compiler flags do and what is the impact they have at the functional and performance area of the project. This is a preliminary report, we will be following up with a test plan for functional & performance test plans for the services as a next step. This report includes: * Detailed description of what the compiler flag does * Code example that shows how does it work to prevent attacks * If there is a change in the binary, we create a microbenchmark that shows us how the flag impact the performance https://github.com/VictorRodriguez/hobbies/tree/master/c_programing_exercise... As a result of the microbenchmark, the performance impact is not relevant ( less than 1% ) using an Ubuntu x86 system ( GCC 5 ) (more details on the HW and SW specification upon requests) The areas of the code we are suggesting on the patches are: * stx-ha * stx-metal * stx-nfv * stx-fault We do take care that these flags are not breaking the following areas after being applied. * Build process of the image * Sanity test cases after the image is created (Ada can give more details on the sanity report of the image generated with these flags) If running the sanity tests are not enough to prove that a change in compiler flags do not affect functionality, please gave us the right path to follow. As mentioned before, this is a preliminary report, and that we will be following up with a test plan for functional & performance test plans for the services as a next step. Hope this email helps to clarify some questions related to the flags and start the follow-up discussion. Regards Victor Rodriguez
On Thu, Dec 20, 2018 at 3:47 PM Victor Rodriguez <vm.rod25@gmail.com> wrote:
Hi StarlingX community
We can all agree that security is an important feature to be taken into consideration in any SW project. In the aim of improving the security of the StarlingX project, we have been taking the task to propose the use of some compiler flags that prevent and detect some security holes, especially by buffer overflow that could lead into ROP attacks.
The list of flags that we are proposing are :
Stack-based Buffer Overrun Detection: CFLAGS=”-fstack-protector-strong”
Fortify source: CFLAGS="-O2 -D_FORTIFY_SOURCE=2" Format string vulnerabilities: CFLAGS="-Wformat -Wformat-security" Stack execution protection: LDFLAGS="-z noexecstack" Data relocation and protection (RELRO): LDLFAGS="-z relro -z now"
These are being analyzed in the following Gerrit reviews (thanks a lot for all the good feedback)
https://review.openstack.org/#/c/623608/ https://review.openstack.org/#/c/623603/ https://review.openstack.org/#/c/623601/ https://review.openstack.org/#/c/623599/
As requested in the Gerrit reviews, there is a proper need to first understand what these compiler flags do and what is the impact they have at the functional and performance area of the project. This is a preliminary report, we will be following up with a test plan for functional & performance test plans for the services as a next step. This report includes:
* Detailed description of what the compiler flag does * Code example that shows how does it work to prevent attacks * If there is a change in the binary, we create a microbenchmark that shows us how the flag impact the performance
https://github.com/VictorRodriguez/hobbies/tree/master/c_programing_exercise...
As a result of the microbenchmark, the performance impact is not relevant ( less than 1% ) using an Ubuntu x86 system ( GCC 5 ) (more details on the HW and SW specification upon requests)
The areas of the code we are suggesting on the patches are:
* stx-ha * stx-metal * stx-nfv * stx-fault
We do take care that these flags are not breaking the following areas after being applied.
* Build process of the image * Sanity test cases after the image is created (Ada can give more details on the sanity report of the image generated with these flags)
If running the sanity tests are not enough to prove that a change in compiler flags do not affect functionality, please gave us the right path to follow.
As mentioned before, this is a preliminary report, and that we will be following up with a test plan for functional & performance test plans for the services as a next step.
Hope this email helps to clarify some questions related to the flags and start the follow-up discussion.
Thanks for the context Victor, it's very helpful to me. One thing I want to mention is something the Kata Containers team was talking about at the Berlin OpenStack summit, which is when many small performance hits start to add up. They have to be careful to ensure they don't have a bunch of smallish looking changes that add up to a large performance hit over a longer period of time. Overall I'm sure the StarlingX project would like to have some performance testing, if we don't already, though that can be challenging for an open source project. I had mentioned OPNFV's Functest and related projects on the TSC call, but now seeing which components are affected I'm not sure that would be directly helpful. I look forward to further discussions around this area. Thanks, Curtis
Regards
Victor Rodriguez
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
-- Blog: serverascode.com
On Fri, Dec 21, 2018, 07:08 Curtis <serverascode@gmail.com wrote:
On Thu, Dec 20, 2018 at 3:47 PM Victor Rodriguez <vm.rod25@gmail.com> wrote:
Hi StarlingX community
We can all agree that security is an important feature to be taken into consideration in any SW project. In the aim of improving the security of the StarlingX project, we have been taking the task to propose the use of some compiler flags that prevent and detect some security holes, especially by buffer overflow that could lead into ROP attacks.
The list of flags that we are proposing are :
Stack-based Buffer Overrun Detection: CFLAGS=”-fstack-protector-strong”
Fortify source: CFLAGS="-O2 -D_FORTIFY_SOURCE=2" Format string vulnerabilities: CFLAGS="-Wformat -Wformat-security" Stack execution protection: LDFLAGS="-z noexecstack" Data relocation and protection (RELRO): LDLFAGS="-z relro -z now"
These are being analyzed in the following Gerrit reviews (thanks a lot for all the good feedback)
https://review.openstack.org/#/c/623608/ https://review.openstack.org/#/c/623603/ https://review.openstack.org/#/c/623601/ https://review.openstack.org/#/c/623599/
As requested in the Gerrit reviews, there is a proper need to first understand what these compiler flags do and what is the impact they have at the functional and performance area of the project. This is a preliminary report, we will be following up with a test plan for functional & performance test plans for the services as a next step. This report includes:
* Detailed description of what the compiler flag does * Code example that shows how does it work to prevent attacks * If there is a change in the binary, we create a microbenchmark that shows us how the flag impact the performance
https://github.com/VictorRodriguez/hobbies/tree/master/c_programing_exercise...
As a result of the microbenchmark, the performance impact is not relevant ( less than 1% ) using an Ubuntu x86 system ( GCC 5 ) (more details on the HW and SW specification upon requests)
The areas of the code we are suggesting on the patches are:
* stx-ha * stx-metal * stx-nfv * stx-fault
We do take care that these flags are not breaking the following areas after being applied.
* Build process of the image * Sanity test cases after the image is created (Ada can give more details on the sanity report of the image generated with these flags)
If running the sanity tests are not enough to prove that a change in compiler flags do not affect functionality, please gave us the right path to follow.
As mentioned before, this is a preliminary report, and that we will be following up with a test plan for functional & performance test plans for the services as a next step.
Hope this email helps to clarify some questions related to the flags and start the follow-up discussion.
Thanks for the context Victor, it's very helpful to me.
Hi Curtis, glad it helps, it was fun to do the research
One thing I want to mention is something the Kata Containers team was talking about at the Berlin OpenStack summit, which is when many small performance hits start to add up. They have to be careful to ensure they don't have a bunch of smallish looking changes that add up to a large performance hit over a longer period of time.
You are right, it's a valid point that we need to take care too
Overall I'm sure the StarlingX project would like to have some performance testing, if we don't already, though that can be challenging for an open source project. I had mentioned OPNFV's Functest and related projects on the TSC call, but now seeing which components are affected I'm not sure that would be directly helpful. I look forward to further discussions around this area.
Thanks for let me know that, I will take a look at OPNFV's functest and other projects before the next TSC of 2019 I will do my best to came up with a proposal for a better performance testing. Thanks Victor Rodriguez
Thanks, Curtis
Regards
Victor Rodriguez
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
-- Blog: serverascode.com
Victor, Security work is never completed. There is always a long list of inventive new vulnerabilities and a laundry list of hardening work to be completed. The vulnerability work, considering the severity, is generally urgent. Hardening work is not urgent but important. In this case, we are dealing with a hardening initiative that focuses on a small area of the code. The challenge is that these small change proposed have larger implications. As was pointed out on the gerrit reviews, performance and / or functional testing is required. My concern is that we affect the timing / behaviour of stx-ha and stx-metal such that they do not work together in some scenarios. This will need to be tested and is certainly larger than a sanity. Also, I am wondering if there is a way to phase the effort. For example, is there a way to break up the flag changes such that the warnings are separated from the flags which change the compiled code? That way, we are not trying to jam everything through at once. Hope this helps. Happy to discuss when you return from Holliday. Regards, Ken Y From: Victor Rodriguez <vm.rod25@gmail.com> Date: Friday, December 28, 2018 at 7:34 PM To: Curtis <serverascode@gmail.com> Cc: "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security On Fri, Dec 21, 2018, 07:08 Curtis <serverascode@gmail.com<mailto:serverascode@gmail.com> wrote: On Thu, Dec 20, 2018 at 3:47 PM Victor Rodriguez <vm.rod25@gmail.com<mailto:vm.rod25@gmail.com>> wrote: Hi StarlingX community We can all agree that security is an important feature to be taken into consideration in any SW project. In the aim of improving the security of the StarlingX project, we have been taking the task to propose the use of some compiler flags that prevent and detect some security holes, especially by buffer overflow that could lead into ROP attacks. The list of flags that we are proposing are : Stack-based Buffer Overrun Detection: CFLAGS=”-fstack-protector-strong” Fortify source: CFLAGS="-O2 -D_FORTIFY_SOURCE=2" Format string vulnerabilities: CFLAGS="-Wformat -Wformat-security" Stack execution protection: LDFLAGS="-z noexecstack" Data relocation and protection (RELRO): LDLFAGS="-z relro -z now" These are being analyzed in the following Gerrit reviews (thanks a lot for all the good feedback) https://review.openstack.org/#/c/623608/ https://review.openstack.org/#/c/623603/ https://review.openstack.org/#/c/623601/ https://review.openstack.org/#/c/623599/ As requested in the Gerrit reviews, there is a proper need to first understand what these compiler flags do and what is the impact they have at the functional and performance area of the project. This is a preliminary report, we will be following up with a test plan for functional & performance test plans for the services as a next step. This report includes: * Detailed description of what the compiler flag does * Code example that shows how does it work to prevent attacks * If there is a change in the binary, we create a microbenchmark that shows us how the flag impact the performance https://github.com/VictorRodriguez/hobbies/tree/master/c_programing_exercise... As a result of the microbenchmark, the performance impact is not relevant ( less than 1% ) using an Ubuntu x86 system ( GCC 5 ) (more details on the HW and SW specification upon requests) The areas of the code we are suggesting on the patches are: * stx-ha * stx-metal * stx-nfv * stx-fault We do take care that these flags are not breaking the following areas after being applied. * Build process of the image * Sanity test cases after the image is created (Ada can give more details on the sanity report of the image generated with these flags) If running the sanity tests are not enough to prove that a change in compiler flags do not affect functionality, please gave us the right path to follow. As mentioned before, this is a preliminary report, and that we will be following up with a test plan for functional & performance test plans for the services as a next step. Hope this email helps to clarify some questions related to the flags and start the follow-up discussion. Thanks for the context Victor, it's very helpful to me. Hi Curtis, glad it helps, it was fun to do the research One thing I want to mention is something the Kata Containers team was talking about at the Berlin OpenStack summit, which is when many small performance hits start to add up. They have to be careful to ensure they don't have a bunch of smallish looking changes that add up to a large performance hit over a longer period of time. You are right, it's a valid point that we need to take care too Overall I'm sure the StarlingX project would like to have some performance testing, if we don't already, though that can be challenging for an open source project. I had mentioned OPNFV's Functest and related projects on the TSC call, but now seeing which components are affected I'm not sure that would be directly helpful. I look forward to further discussions around this area. Thanks for let me know that, I will take a look at OPNFV's functest and other projects before the next TSC of 2019 I will do my best to came up with a proposal for a better performance testing. Thanks Victor Rodriguez Thanks, Curtis Regards Victor Rodriguez _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss -- Blog: serverascode.com<http://serverascode.com>
On Wed, Jan 2, 2019 at 10:35 AM Young, Ken <Ken.Young@windriver.com> wrote:
Victor,
Security work is never completed. There is always a long list of inventive new vulnerabilities and a laundry list of hardening work to be completed. The vulnerability work, considering the severity, is generally urgent. Hardening work is not urgent but important. In this case, we are dealing with a hardening initiative that focuses on a small area of the code.
The challenge is that these small change proposed have larger implications. As was pointed out on the gerrit reviews, performance and / or functional testing is required. My concern is that we affect the timing / behaviour of stx-ha and stx-metal such that they do not work together in some scenarios. This will need to be tested and is certainly larger than a sanity.
Agree, our concern on the last TSC meeting was to come up with a proper framework to measure the performance impact of key changes in the project ( such as new compiler flags or new functionality options). The concern you have about timing /behavior of stx-ha and stx-metal is a key point that I would like to understand more, the idea is to improve security without affecting functionality at all
Also, I am wondering if there is a way to phase the effort. For example, is there a way to break up the flag changes such that the warnings are separated from the flags which change the compiled code? That way, we are not trying to jam everything through at once.
We could came up with a V2 of the patches with just the warning flags and the fixes to those warnings, is that ok?
Hope this helps. Happy to discuss when you return from Holliday.
Sure, thanks for the feedback ( I will be fully back Monday )
Regards,
Ken Y
From: Victor Rodriguez <vm.rod25@gmail.com> Date: Friday, December 28, 2018 at 7:34 PM To: Curtis <serverascode@gmail.com> Cc: "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security
On Fri, Dec 21, 2018, 07:08 Curtis <serverascode@gmail.com wrote:
On Thu, Dec 20, 2018 at 3:47 PM Victor Rodriguez <vm.rod25@gmail.com> wrote:
Hi StarlingX community
We can all agree that security is an important feature to be taken into consideration in any SW project. In the aim of improving the security of the StarlingX project, we have been taking the task to propose the use of some compiler flags that prevent and detect some security holes, especially by buffer overflow that could lead into ROP attacks.
The list of flags that we are proposing are :
Stack-based Buffer Overrun Detection: CFLAGS=”-fstack-protector-strong”
Fortify source: CFLAGS="-O2 -D_FORTIFY_SOURCE=2" Format string vulnerabilities: CFLAGS="-Wformat -Wformat-security" Stack execution protection: LDFLAGS="-z noexecstack" Data relocation and protection (RELRO): LDLFAGS="-z relro -z now"
These are being analyzed in the following Gerrit reviews (thanks a lot for all the good feedback)
https://review.openstack.org/#/c/623608/ https://review.openstack.org/#/c/623603/ https://review.openstack.org/#/c/623601/ https://review.openstack.org/#/c/623599/
As requested in the Gerrit reviews, there is a proper need to first understand what these compiler flags do and what is the impact they have at the functional and performance area of the project. This is a preliminary report, we will be following up with a test plan for functional & performance test plans for the services as a next step. This report includes:
* Detailed description of what the compiler flag does * Code example that shows how does it work to prevent attacks * If there is a change in the binary, we create a microbenchmark that shows us how the flag impact the performance
https://github.com/VictorRodriguez/hobbies/tree/master/c_programing_exercise...
As a result of the microbenchmark, the performance impact is not relevant ( less than 1% ) using an Ubuntu x86 system ( GCC 5 ) (more details on the HW and SW specification upon requests)
The areas of the code we are suggesting on the patches are:
* stx-ha * stx-metal * stx-nfv * stx-fault
We do take care that these flags are not breaking the following areas after being applied.
* Build process of the image * Sanity test cases after the image is created (Ada can give more details on the sanity report of the image generated with these flags)
If running the sanity tests are not enough to prove that a change in compiler flags do not affect functionality, please gave us the right path to follow.
As mentioned before, this is a preliminary report, and that we will be following up with a test plan for functional & performance test plans for the services as a next step.
Hope this email helps to clarify some questions related to the flags and start the follow-up discussion.
Thanks for the context Victor, it's very helpful to me.
Hi Curtis, glad it helps, it was fun to do the research
One thing I want to mention is something the Kata Containers team was talking about at the Berlin OpenStack summit, which is when many small performance hits start to add up. They have to be careful to ensure they don't have a bunch of smallish looking changes that add up to a large performance hit over a longer period of time.
You are right, it's a valid point that we need to take care too
Overall I'm sure the StarlingX project would like to have some performance testing, if we don't already, though that can be challenging for an open source project. I had mentioned OPNFV's Functest and related projects on the TSC call, but now seeing which components are affected I'm not sure that would be directly helpful. I look forward to further discussions around this area.
Thanks for let me know that, I will take a look at OPNFV's functest and other projects before the next TSC of 2019
I will do my best to came up with a proposal for a better performance testing.
Thanks
Victor Rodriguez
Thanks,
Curtis
Regards
Victor Rodriguez
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
--
Blog: serverascode.com
On Wed, 2019-01-02 at 14:40 -0600, Victor Rodriguez wrote:
On Wed, Jan 2, 2019 at 10:35 AM Young, Ken <Ken.Young@windriver.com> wrote:
Victor,
Security work is never completed. There is always a long list of inventive new vulnerabilities and a laundry list of hardening work to be completed. The vulnerability work, considering the severity, is generally urgent. Hardening work is not urgent but important. In this case, we are dealing with a hardening initiative that focuses on a small area of the code.
I don't entirely agree with the hardening urgency, as hardening can prevent vulnerabilities. The lack of reported vulnerabilities not necessary means that software is secure, most probably is that the software hasn't been used/tested well enough. Prevention tends to be more cost-effective than mitigation. I'd prefer to treat the hardening as high priority.
The challenge is that these small change proposed have larger implications. As was pointed out on the gerrit reviews, performance and / or functional testing is required. My concern is that we affect the timing / behaviour of stx-ha and stx-metal such that they do not work together in some scenarios. This will need to be tested and is certainly larger than a sanity.
I agree, Also I think that in order to improve the discussion and solution of these issues it would be helpful to understand the critical/specific use cases that people is worried about and the thresholds that we shouldn't cross. Then, find a way to measure them and start from there. Otherwise we will deal with a high level of ambiguity that could cause delays on solving these issues.
Agree, our concern on the last TSC meeting was to come up with a proper framework to measure the performance impact of key changes in the project ( such as new compiler flags or new functionality options). The concern you have about timing /behavior of stx-ha and stx-metal is a key point that I would like to understand more, the idea is to improve security without affecting functionality at all
Also, I am wondering if there is a way to phase the effort. For example, is there a way to break up the flag changes such that the warnings are separated from the flags which change the compiled code? That way, we are not trying to jam everything through at once.
We could came up with a V2 of the patches with just the warning flags and the fixes to those warnings, is that ok?
Agree with apply the warning patch first to have some progress.
Hope this helps. Happy to discuss when you return from Holliday.
Sure, thanks for the feedback ( I will be fully back Monday )
Regards,
Ken Y
From: Victor Rodriguez <vm.rod25@gmail.com> Date: Friday, December 28, 2018 at 7:34 PM To: Curtis <serverascode@gmail.com> Cc: "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists .starlingx.io> Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security
On Fri, Dec 21, 2018, 07:08 Curtis <serverascode@gmail.com wrote:
On Thu, Dec 20, 2018 at 3:47 PM Victor Rodriguez <vm.rod25@gmail.co m> wrote:
Hi StarlingX community
We can all agree that security is an important feature to be taken into consideration in any SW project. In the aim of improving the security of the StarlingX project, we have been taking the task to propose the use of some compiler flags that prevent and detect some security holes, especially by buffer overflow that could lead into ROP attacks.
The list of flags that we are proposing are :
Stack-based Buffer Overrun Detection: CFLAGS=”-fstack-protector- strong”
Fortify source: CFLAGS="-O2 -D_FORTIFY_SOURCE=2" Format string vulnerabilities: CFLAGS="-Wformat -Wformat- security" Stack execution protection: LDFLAGS="-z noexecstack" Data relocation and protection (RELRO): LDLFAGS="-z relro -z now"
These are being analyzed in the following Gerrit reviews (thanks a lot for all the good feedback)
https://review.openstack.org/#/c/623608/ https://review.openstack.org/#/c/623603/ https://review.openstack.org/#/c/623601/ https://review.openstack.org/#/c/623599/
As requested in the Gerrit reviews, there is a proper need to first understand what these compiler flags do and what is the impact they have at the functional and performance area of the project. This is a preliminary report, we will be following up with a test plan for functional & performance test plans for the services as a next step. This report includes:
* Detailed description of what the compiler flag does * Code example that shows how does it work to prevent attacks * If there is a change in the binary, we create a microbenchmark that shows us how the flag impact the performance
https://github.com/VictorRodriguez/hobbies/tree/master/c_programing _exercises/cflags_security
As a result of the microbenchmark, the performance impact is not relevant ( less than 1% ) using an Ubuntu x86 system ( GCC 5 ) (more details on the HW and SW specification upon requests)
The areas of the code we are suggesting on the patches are:
* stx-ha * stx-metal * stx-nfv * stx-fault
We do take care that these flags are not breaking the following areas after being applied.
* Build process of the image * Sanity test cases after the image is created (Ada can give more details on the sanity report of the image generated with these flags)
If running the sanity tests are not enough to prove that a change in compiler flags do not affect functionality, please gave us the right path to follow.
As mentioned before, this is a preliminary report, and that we will be following up with a test plan for functional & performance test plans for the services as a next step.
Hope this email helps to clarify some questions related to the flags and start the follow-up discussion.
Thanks for the context Victor, it's very helpful to me.
Hi Curtis, glad it helps, it was fun to do the research
One thing I want to mention is something the Kata Containers team was talking about at the Berlin OpenStack summit, which is when many small performance hits start to add up. They have to be careful to ensure they don't have a bunch of smallish looking changes that add up to a large performance hit over a longer period of time.
You are right, it's a valid point that we need to take care too
Overall I'm sure the StarlingX project would like to have some performance testing, if we don't already, though that can be challenging for an open source project. I had mentioned OPNFV's Functest and related projects on the TSC call, but now seeing which components are affected I'm not sure that would be directly helpful. I look forward to further discussions around this area.
Thanks for let me know that, I will take a look at OPNFV's functest and other projects before the next TSC of 2019
I will do my best to came up with a proposal for a better performance testing.
Thanks
Victor Rodriguez
Thanks,
Curtis
Regards
Victor Rodriguez
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus s
--
Blog: serverascode.com
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi: Thank you all for your feedback! I also agree to first add the "warning as errors" flags, I abandoned the old patches and proposed new ones: https://review.openstack.org/#/c/629329/ https://review.openstack.org/#/c/629331/ https://review.openstack.org/#/c/629332/ Once them are merged, we may add the remaining security flags Best Regards Luis Botello -----Original Message----- From: Cordoba Malibran, Erich [mailto:erich.cordoba.malibran@intel.com] Sent: Monday, January 7, 2019 4:14 PM To: vm.rod25@gmail.com; Ken.Young@windriver.com Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security On Wed, 2019-01-02 at 14:40 -0600, Victor Rodriguez wrote:
On Wed, Jan 2, 2019 at 10:35 AM Young, Ken <Ken.Young@windriver.com> wrote:
Victor,
Security work is never completed. There is always a long list of inventive new vulnerabilities and a laundry list of hardening work to be completed. The vulnerability work, considering the severity, is generally urgent. Hardening work is not urgent but important. In this case, we are dealing with a hardening initiative that focuses on a small area of the code.
I don't entirely agree with the hardening urgency, as hardening can prevent vulnerabilities. The lack of reported vulnerabilities not necessary means that software is secure, most probably is that the software hasn't been used/tested well enough. Prevention tends to be more cost-effective than mitigation. I'd prefer to treat the hardening as high priority.
The challenge is that these small change proposed have larger implications. As was pointed out on the gerrit reviews, performance and / or functional testing is required. My concern is that we affect the timing / behaviour of stx-ha and stx-metal such that they do not work together in some scenarios. This will need to be tested and is certainly larger than a sanity.
I agree, Also I think that in order to improve the discussion and solution of these issues it would be helpful to understand the critical/specific use cases that people is worried about and the thresholds that we shouldn't cross. Then, find a way to measure them and start from there. Otherwise we will deal with a high level of ambiguity that could cause delays on solving these issues.
Agree, our concern on the last TSC meeting was to come up with a proper framework to measure the performance impact of key changes in the project ( such as new compiler flags or new functionality options). The concern you have about timing /behavior of stx-ha and stx-metal is a key point that I would like to understand more, the idea is to improve security without affecting functionality at all
Also, I am wondering if there is a way to phase the effort. For example, is there a way to break up the flag changes such that the warnings are separated from the flags which change the compiled code? That way, we are not trying to jam everything through at once.
We could came up with a V2 of the patches with just the warning flags and the fixes to those warnings, is that ok?
Agree with apply the warning patch first to have some progress.
Hope this helps. Happy to discuss when you return from Holliday.
Sure, thanks for the feedback ( I will be fully back Monday )
Regards,
Ken Y
From: Victor Rodriguez <vm.rod25@gmail.com> Date: Friday, December 28, 2018 at 7:34 PM To: Curtis <serverascode@gmail.com> Cc: "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists .starlingx.io> Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security
On Fri, Dec 21, 2018, 07:08 Curtis <serverascode@gmail.com wrote:
On Thu, Dec 20, 2018 at 3:47 PM Victor Rodriguez <vm.rod25@gmail.co m> wrote:
Hi StarlingX community
We can all agree that security is an important feature to be taken into consideration in any SW project. In the aim of improving the security of the StarlingX project, we have been taking the task to propose the use of some compiler flags that prevent and detect some security holes, especially by buffer overflow that could lead into ROP attacks.
The list of flags that we are proposing are :
Stack-based Buffer Overrun Detection: CFLAGS=”-fstack-protector- strong”
Fortify source: CFLAGS="-O2 -D_FORTIFY_SOURCE=2" Format string vulnerabilities: CFLAGS="-Wformat -Wformat- security" Stack execution protection: LDFLAGS="-z noexecstack" Data relocation and protection (RELRO): LDLFAGS="-z relro -z now"
These are being analyzed in the following Gerrit reviews (thanks a lot for all the good feedback)
https://review.openstack.org/#/c/623608/ https://review.openstack.org/#/c/623603/ https://review.openstack.org/#/c/623601/ https://review.openstack.org/#/c/623599/
As requested in the Gerrit reviews, there is a proper need to first understand what these compiler flags do and what is the impact they have at the functional and performance area of the project. This is a preliminary report, we will be following up with a test plan for functional & performance test plans for the services as a next step. This report includes:
* Detailed description of what the compiler flag does * Code example that shows how does it work to prevent attacks * If there is a change in the binary, we create a microbenchmark that shows us how the flag impact the performance
https://github.com/VictorRodriguez/hobbies/tree/master/c_programing _exercises/cflags_security
As a result of the microbenchmark, the performance impact is not relevant ( less than 1% ) using an Ubuntu x86 system ( GCC 5 ) (more details on the HW and SW specification upon requests)
The areas of the code we are suggesting on the patches are:
* stx-ha * stx-metal * stx-nfv * stx-fault
We do take care that these flags are not breaking the following areas after being applied.
* Build process of the image * Sanity test cases after the image is created (Ada can give more details on the sanity report of the image generated with these flags)
If running the sanity tests are not enough to prove that a change in compiler flags do not affect functionality, please gave us the right path to follow.
As mentioned before, this is a preliminary report, and that we will be following up with a test plan for functional & performance test plans for the services as a next step.
Hope this email helps to clarify some questions related to the flags and start the follow-up discussion.
Thanks for the context Victor, it's very helpful to me.
Hi Curtis, glad it helps, it was fun to do the research
One thing I want to mention is something the Kata Containers team was talking about at the Berlin OpenStack summit, which is when many small performance hits start to add up. They have to be careful to ensure they don't have a bunch of smallish looking changes that add up to a large performance hit over a longer period of time.
You are right, it's a valid point that we need to take care too
Overall I'm sure the StarlingX project would like to have some performance testing, if we don't already, though that can be challenging for an open source project. I had mentioned OPNFV's Functest and related projects on the TSC call, but now seeing which components are affected I'm not sure that would be directly helpful. I look forward to further discussions around this area.
Thanks for let me know that, I will take a look at OPNFV's functest and other projects before the next TSC of 2019
I will do my best to came up with a proposal for a better performance testing.
Thanks
Victor Rodriguez
Thanks,
Curtis
Regards
Victor Rodriguez
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus s
--
Blog: serverascode.com
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Tue, Jan 8, 2019 at 5:44 PM Botello Ortega, Luis <luis.botello.ortega@intel.com> wrote:
Hi:
Thank you all for your feedback! I also agree to first add the "warning as errors" flags, I abandoned the old patches and proposed new ones: https://review.openstack.org/#/c/629329/ https://review.openstack.org/#/c/629331/ https://review.openstack.org/#/c/629332/
Thanks Luis, can you post the result of the sanity tests after these patches ? to check that image build and basic tests pass Regards
Once them are merged, we may add the remaining security flags
Best Regards Luis Botello
-----Original Message----- From: Cordoba Malibran, Erich [mailto:erich.cordoba.malibran@intel.com] Sent: Monday, January 7, 2019 4:14 PM To: vm.rod25@gmail.com; Ken.Young@windriver.com Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security
On Wed, 2019-01-02 at 14:40 -0600, Victor Rodriguez wrote:
On Wed, Jan 2, 2019 at 10:35 AM Young, Ken <Ken.Young@windriver.com> wrote:
Victor,
Security work is never completed. There is always a long list of inventive new vulnerabilities and a laundry list of hardening work to be completed. The vulnerability work, considering the severity, is generally urgent. Hardening work is not urgent but important. In this case, we are dealing with a hardening initiative that focuses on a small area of the code.
I don't entirely agree with the hardening urgency, as hardening can prevent vulnerabilities. The lack of reported vulnerabilities not necessary means that software is secure, most probably is that the software hasn't been used/tested well enough.
Prevention tends to be more cost-effective than mitigation. I'd prefer to treat the hardening as high priority.
The challenge is that these small change proposed have larger implications. As was pointed out on the gerrit reviews, performance and / or functional testing is required. My concern is that we affect the timing / behaviour of stx-ha and stx-metal such that they do not work together in some scenarios. This will need to be tested and is certainly larger than a sanity.
I agree, Also I think that in order to improve the discussion and solution of these issues it would be helpful to understand the critical/specific use cases that people is worried about and the thresholds that we shouldn't cross. Then, find a way to measure them and start from there.
Otherwise we will deal with a high level of ambiguity that could cause delays on solving these issues.
Agree, our concern on the last TSC meeting was to come up with a proper framework to measure the performance impact of key changes in the project ( such as new compiler flags or new functionality options). The concern you have about timing /behavior of stx-ha and stx-metal is a key point that I would like to understand more, the idea is to improve security without affecting functionality at all
Also, I am wondering if there is a way to phase the effort. For example, is there a way to break up the flag changes such that the warnings are separated from the flags which change the compiled code? That way, we are not trying to jam everything through at once.
We could came up with a V2 of the patches with just the warning flags and the fixes to those warnings, is that ok?
Agree with apply the warning patch first to have some progress.
Hope this helps. Happy to discuss when you return from Holliday.
Sure, thanks for the feedback ( I will be fully back Monday )
Regards,
Ken Y
From: Victor Rodriguez <vm.rod25@gmail.com> Date: Friday, December 28, 2018 at 7:34 PM To: Curtis <serverascode@gmail.com> Cc: "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists .starlingx.io> Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security
On Fri, Dec 21, 2018, 07:08 Curtis <serverascode@gmail.com wrote:
On Thu, Dec 20, 2018 at 3:47 PM Victor Rodriguez <vm.rod25@gmail.co m> wrote:
Hi StarlingX community
We can all agree that security is an important feature to be taken into consideration in any SW project. In the aim of improving the security of the StarlingX project, we have been taking the task to propose the use of some compiler flags that prevent and detect some security holes, especially by buffer overflow that could lead into ROP attacks.
The list of flags that we are proposing are :
Stack-based Buffer Overrun Detection: CFLAGS=”-fstack-protector- strong”
Fortify source: CFLAGS="-O2 -D_FORTIFY_SOURCE=2" Format string vulnerabilities: CFLAGS="-Wformat -Wformat- security" Stack execution protection: LDFLAGS="-z noexecstack" Data relocation and protection (RELRO): LDLFAGS="-z relro -z now"
These are being analyzed in the following Gerrit reviews (thanks a lot for all the good feedback)
https://review.openstack.org/#/c/623608/ https://review.openstack.org/#/c/623603/ https://review.openstack.org/#/c/623601/ https://review.openstack.org/#/c/623599/
As requested in the Gerrit reviews, there is a proper need to first understand what these compiler flags do and what is the impact they have at the functional and performance area of the project. This is a preliminary report, we will be following up with a test plan for functional & performance test plans for the services as a next step. This report includes:
* Detailed description of what the compiler flag does * Code example that shows how does it work to prevent attacks * If there is a change in the binary, we create a microbenchmark that shows us how the flag impact the performance
https://github.com/VictorRodriguez/hobbies/tree/master/c_programing _exercises/cflags_security
As a result of the microbenchmark, the performance impact is not relevant ( less than 1% ) using an Ubuntu x86 system ( GCC 5 ) (more details on the HW and SW specification upon requests)
The areas of the code we are suggesting on the patches are:
* stx-ha * stx-metal * stx-nfv * stx-fault
We do take care that these flags are not breaking the following areas after being applied.
* Build process of the image * Sanity test cases after the image is created (Ada can give more details on the sanity report of the image generated with these flags)
If running the sanity tests are not enough to prove that a change in compiler flags do not affect functionality, please gave us the right path to follow.
As mentioned before, this is a preliminary report, and that we will be following up with a test plan for functional & performance test plans for the services as a next step.
Hope this email helps to clarify some questions related to the flags and start the follow-up discussion.
Thanks for the context Victor, it's very helpful to me.
Hi Curtis, glad it helps, it was fun to do the research
One thing I want to mention is something the Kata Containers team was talking about at the Berlin OpenStack summit, which is when many small performance hits start to add up. They have to be careful to ensure they don't have a bunch of smallish looking changes that add up to a large performance hit over a longer period of time.
You are right, it's a valid point that we need to take care too
Overall I'm sure the StarlingX project would like to have some performance testing, if we don't already, though that can be challenging for an open source project. I had mentioned OPNFV's Functest and related projects on the TSC call, but now seeing which components are affected I'm not sure that would be directly helpful. I look forward to further discussions around this area.
Thanks for let me know that, I will take a look at OPNFV's functest and other projects before the next TSC of 2019
I will do my best to came up with a proposal for a better performance testing.
Thanks
Victor Rodriguez
Thanks,
Curtis
Regards
Victor Rodriguez
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus s
--
Blog: serverascode.com
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Wed, Jan 2, 2019 at 10:35 AM Young, Ken <Ken.Young@windriver.com> wrote:
Victor,
Security work is never completed. There is always a long list of inventive new vulnerabilities and a laundry list of hardening work to be completed. The vulnerability work, considering the severity, is generally urgent. Hardening work is not urgent but important. In this case, we are dealing with a hardening initiative that focuses on a small area of the code.
The challenge is that these small change proposed have larger implications. As was pointed out on the gerrit reviews, performance and / or functional testing is required.
Hi Ken Just to follow the idea of this mail after hollliday break, you mention that: My concern is that we affect the timing / behaviour of stx-ha and stx-metal such that they do not work together in some scenarios. This will need to be tested and is certainly larger than a sanity. Could you please help to describe n human words, ( I can do the script ) how a good test to probe this would look like? If you provide me with a basic description of the security test I could help writing the first draft of a code test that help us to prove if the flags break the funcionality thanks Victor R
Also, I am wondering if there is a way to phase the effort. For example, is there a way to break up the flag changes such that the warnings are separated from the flags which change the compiled code? That way, we are not trying to jam everything through at once.
Hope this helps. Happy to discuss when you return from Holliday.
Regards,
Ken Y
From: Victor Rodriguez <vm.rod25@gmail.com> Date: Friday, December 28, 2018 at 7:34 PM To: Curtis <serverascode@gmail.com> Cc: "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security
On Fri, Dec 21, 2018, 07:08 Curtis <serverascode@gmail.com wrote:
On Thu, Dec 20, 2018 at 3:47 PM Victor Rodriguez <vm.rod25@gmail.com> wrote:
Hi StarlingX community
We can all agree that security is an important feature to be taken into consideration in any SW project. In the aim of improving the security of the StarlingX project, we have been taking the task to propose the use of some compiler flags that prevent and detect some security holes, especially by buffer overflow that could lead into ROP attacks.
The list of flags that we are proposing are :
Stack-based Buffer Overrun Detection: CFLAGS=”-fstack-protector-strong”
Fortify source: CFLAGS="-O2 -D_FORTIFY_SOURCE=2" Format string vulnerabilities: CFLAGS="-Wformat -Wformat-security" Stack execution protection: LDFLAGS="-z noexecstack" Data relocation and protection (RELRO): LDLFAGS="-z relro -z now"
These are being analyzed in the following Gerrit reviews (thanks a lot for all the good feedback)
https://review.openstack.org/#/c/623608/ https://review.openstack.org/#/c/623603/ https://review.openstack.org/#/c/623601/ https://review.openstack.org/#/c/623599/
As requested in the Gerrit reviews, there is a proper need to first understand what these compiler flags do and what is the impact they have at the functional and performance area of the project. This is a preliminary report, we will be following up with a test plan for functional & performance test plans for the services as a next step. This report includes:
* Detailed description of what the compiler flag does * Code example that shows how does it work to prevent attacks * If there is a change in the binary, we create a microbenchmark that shows us how the flag impact the performance
https://github.com/VictorRodriguez/hobbies/tree/master/c_programing_exercise...
As a result of the microbenchmark, the performance impact is not relevant ( less than 1% ) using an Ubuntu x86 system ( GCC 5 ) (more details on the HW and SW specification upon requests)
The areas of the code we are suggesting on the patches are:
* stx-ha * stx-metal * stx-nfv * stx-fault
We do take care that these flags are not breaking the following areas after being applied.
* Build process of the image * Sanity test cases after the image is created (Ada can give more details on the sanity report of the image generated with these flags)
If running the sanity tests are not enough to prove that a change in compiler flags do not affect functionality, please gave us the right path to follow.
As mentioned before, this is a preliminary report, and that we will be following up with a test plan for functional & performance test plans for the services as a next step.
Hope this email helps to clarify some questions related to the flags and start the follow-up discussion.
Thanks for the context Victor, it's very helpful to me.
Hi Curtis, glad it helps, it was fun to do the research
One thing I want to mention is something the Kata Containers team was talking about at the Berlin OpenStack summit, which is when many small performance hits start to add up. They have to be careful to ensure they don't have a bunch of smallish looking changes that add up to a large performance hit over a longer period of time.
You are right, it's a valid point that we need to take care too
Overall I'm sure the StarlingX project would like to have some performance testing, if we don't already, though that can be challenging for an open source project. I had mentioned OPNFV's Functest and related projects on the TSC call, but now seeing which components are affected I'm not sure that would be directly helpful. I look forward to further discussions around this area.
Thanks for let me know that, I will take a look at OPNFV's functest and other projects before the next TSC of 2019
I will do my best to came up with a proposal for a better performance testing.
Thanks
Victor Rodriguez
Thanks,
Curtis
Regards
Victor Rodriguez
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
--
Blog: serverascode.com
See inline. On 2019-01-17, 5:34 PM, "Victor Rodriguez" <vm.rod25@gmail.com> wrote: On Wed, Jan 2, 2019 at 10:35 AM Young, Ken <Ken.Young@windriver.com> wrote: > > Victor, > > > > Security work is never completed. There is always a long list of inventive new vulnerabilities and a laundry list of hardening work to be completed. The vulnerability work, considering the severity, is generally urgent. Hardening work is not urgent but important. In this case, we are dealing with a hardening initiative that focuses on a small area of the code. > > > > The challenge is that these small change proposed have larger implications. As was pointed out on the gerrit reviews, performance and / or functional testing is required. Hi Ken Just to follow the idea of this mail after hollliday break, you mention that: My concern is that we affect the timing / behaviour of stx-ha and stx-metal such that they do not work together in some scenarios. This will need to be tested and is certainly larger than a sanity. Could you please help to describe n human words, ( I can do the script ) how a good test to probe this would look like? If you provide me with a basic description of the security test I could help writing the first draft of a code test that help us to prove if the flags break the functionality Victor, At a high level, we need to regress the behaviour of stx-ha and stx-metal to ensure that there is functional issues introduced by the change to the compiler. As well, we need to look at the system behaviour of ha and metal to ensure no changes have been introduced which affect has behaviour: - SWACT detection and time - Multinode failure avoidance - Heartbeat loss - lock / unlock - etc I believe that Ada has the test for ha and metal. Please review. Regards, Ken Y thanks Victor R > > > > Also, I am wondering if there is a way to phase the effort. For example, is there a way to break up the flag changes such that the warnings are separated from the flags which change the compiled code? That way, we are not trying to jam everything through at once. > > > > Hope this helps. Happy to discuss when you return from Holliday. > > > > Regards, > > Ken Y > > > > From: Victor Rodriguez <vm.rod25@gmail.com> > Date: Friday, December 28, 2018 at 7:34 PM > To: Curtis <serverascode@gmail.com> > Cc: "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> > Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security > > > > > > On Fri, Dec 21, 2018, 07:08 Curtis <serverascode@gmail.com wrote: > > > > > > On Thu, Dec 20, 2018 at 3:47 PM Victor Rodriguez <vm.rod25@gmail.com> wrote: > > Hi StarlingX community > > We can all agree that security is an important feature to be taken > into consideration in any SW project. In the aim of improving the > security of the StarlingX project, we have been taking the task to > propose the use of some compiler flags that prevent and detect some > security holes, especially by buffer overflow that could lead into ROP > attacks. > > The list of flags that we are proposing are : > > Stack-based Buffer Overrun Detection: CFLAGS=”-fstack-protector-strong” > > Fortify source: CFLAGS="-O2 -D_FORTIFY_SOURCE=2" > Format string vulnerabilities: CFLAGS="-Wformat -Wformat-security" > Stack execution protection: LDFLAGS="-z noexecstack" > Data relocation and protection (RELRO): LDLFAGS="-z relro -z now" > > > These are being analyzed in the following Gerrit reviews (thanks a lot > for all the good feedback) > > https://review.openstack.org/#/c/623608/ > https://review.openstack.org/#/c/623603/ > https://review.openstack.org/#/c/623601/ > https://review.openstack.org/#/c/623599/ > > As requested in the Gerrit reviews, there is a proper need to first > understand what these compiler flags do and what is the impact they > have at the functional and performance area of the project. This is a > preliminary report, we will be following up with a test plan for > functional & performance test plans for the services as a next step. > This report includes: > > * Detailed description of what the compiler flag does > * Code example that shows how does it work to prevent attacks > * If there is a change in the binary, we create a microbenchmark that > shows us how the flag impact the performance > > https://github.com/VictorRodriguez/hobbies/tree/master/c_programing_exercise... > > As a result of the microbenchmark, the performance impact is not > relevant ( less than 1% ) using an Ubuntu x86 system ( GCC 5 ) (more > details on the HW and SW specification upon requests) > > The areas of the code we are suggesting on the patches are: > > * stx-ha > * stx-metal > * stx-nfv > * stx-fault > > We do take care that these flags are not breaking the following areas > after being applied. > > * Build process of the image > * Sanity test cases after the image is created > (Ada can give more details on the sanity report of the image generated > with these flags) > > If running the sanity tests are not enough to prove that a change in > compiler flags do not affect functionality, please gave us the right > path to follow. > > As mentioned before, this is a preliminary report, and that we will be > following up with a test plan for functional & performance test plans > for the services as a next step. > > Hope this email helps to clarify some questions related to the flags > and start the follow-up discussion. > > > > Thanks for the context Victor, it's very helpful to me. > > > > Hi Curtis, glad it helps, it was fun to do the research > > > > One thing I want to mention is something the Kata Containers team was talking about at the Berlin OpenStack summit, which is when many small performance hits start to add up. They have to be careful to ensure they don't have a bunch of smallish looking changes that add up to a large performance hit over a longer period of time. > > > > You are right, it's a valid point that we need to take care too > > > > Overall I'm sure the StarlingX project would like to have some performance testing, if we don't already, though that can be challenging for an open source project. I had mentioned OPNFV's Functest and related projects on the TSC call, but now seeing which components are affected I'm not sure that would be directly helpful. I look forward to further discussions around this area. > > > > Thanks for let me know that, I will take a look at OPNFV's functest and other projects before the next TSC of 2019 > > > > I will do my best to came up with a proposal for a better performance testing. > > > > Thanks > > > > Victor Rodriguez > > > > Thanks, > > Curtis > > > > > Regards > > Victor Rodriguez > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss > > > > -- > > Blog: serverascode.com
Comments inline
-----Original Message----- From: Young, Ken [mailto:Ken.Young@windriver.com] Sent: Friday, January 18, 2019 9:34 AM To: Victor Rodriguez <vm.rod25@gmail.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security
See inline.
On 2019-01-17, 5:34 PM, "Victor Rodriguez" <vm.rod25@gmail.com> wrote:
On Wed, Jan 2, 2019 at 10:35 AM Young, Ken <Ken.Young@windriver.com> wrote: > > Victor, > > > > Security work is never completed. There is always a long list of inventive new vulnerabilities and a laundry list of hardening work to be completed. The vulnerability work, considering the severity, is generally urgent. Hardening work is not urgent but important. In this case, we are dealing with a hardening initiative that focuses on a small area of the code. > > > > The challenge is that these small change proposed have larger implications. As was pointed out on the gerrit reviews, performance and / or functional testing is required.
Hi Ken
Just to follow the idea of this mail after hollliday break, you mention that:
My concern is that we affect the timing / behaviour of stx-ha and stx-metal such that they do not work together in some scenarios. This will need to be tested and is certainly larger than a sanity.
Could you please help to describe n human words, ( I can do the script ) how a good test to probe this would look like? If you provide me with a basic description of the security test I could help writing the first draft of a code test that help us to prove if the flags break the functionality
Victor,
At a high level, we need to regress the behaviour of stx-ha and stx-metal to ensure that there is functional issues introduced by the change to the compiler. As well, we need to look at the system behaviour of ha and metal to ensure no changes have been introduced which affect has behaviour:
- SWACT detection and time - Multinode failure avoidance - Heartbeat loss - lock / unlock - etc
I believe that Ada has the test for ha and metal. Please review.
Yes, we executed several test cases covering what Ken mentions (manually). What I'm not sure is about heartbeat loss, but let me check. What we can do is to build a test plan and submit it for revision. When do you need it (and please, don't say tomorrow)? Ada
Regards, Ken Y
thanks
Victor R > > > > Also, I am wondering if there is a way to phase the effort. For example, is there a way to break up the flag changes such that the warnings are separated from the flags which change the compiled code? That way, we are not trying to jam everything through at once. > > > > Hope this helps. Happy to discuss when you return from Holliday. > > > > Regards, > > Ken Y > > > > From: Victor Rodriguez <vm.rod25@gmail.com> > Date: Friday, December 28, 2018 at 7:34 PM > To: Curtis <serverascode@gmail.com> > Cc: "starlingx-discuss@lists.starlingx.io" <starlingx- discuss@lists.starlingx.io> > Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security > > > > > > On Fri, Dec 21, 2018, 07:08 Curtis <serverascode@gmail.com wrote: > > > > > > On Thu, Dec 20, 2018 at 3:47 PM Victor Rodriguez <vm.rod25@gmail.com> wrote: > > Hi StarlingX community > > We can all agree that security is an important feature to be taken > into consideration in any SW project. In the aim of improving the > security of the StarlingX project, we have been taking the task to > propose the use of some compiler flags that prevent and detect some > security holes, especially by buffer overflow that could lead into ROP > attacks. > > The list of flags that we are proposing are : > > Stack-based Buffer Overrun Detection: CFLAGS=”-fstack-protector- strong” > > Fortify source: CFLAGS="-O2 -D_FORTIFY_SOURCE=2" > Format string vulnerabilities: CFLAGS="-Wformat -Wformat- security" > Stack execution protection: LDFLAGS="-z noexecstack" > Data relocation and protection (RELRO): LDLFAGS="-z relro -z now" > > > These are being analyzed in the following Gerrit reviews (thanks a lot > for all the good feedback) > > https://review.openstack.org/#/c/623608/ > https://review.openstack.org/#/c/623603/ > https://review.openstack.org/#/c/623601/ > https://review.openstack.org/#/c/623599/ > > As requested in the Gerrit reviews, there is a proper need to first > understand what these compiler flags do and what is the impact they > have at the functional and performance area of the project. This is a > preliminary report, we will be following up with a test plan for > functional & performance test plans for the services as a next step. > This report includes: > > * Detailed description of what the compiler flag does > * Code example that shows how does it work to prevent attacks > * If there is a change in the binary, we create a microbenchmark that > shows us how the flag impact the performance > > https://github.com/VictorRodriguez/hobbies/tree/master/c_programing_ex ercises/cflags_security > > As a result of the microbenchmark, the performance impact is not > relevant ( less than 1% ) using an Ubuntu x86 system ( GCC 5 ) (more > details on the HW and SW specification upon requests) > > The areas of the code we are suggesting on the patches are: > > * stx-ha > * stx-metal > * stx-nfv > * stx-fault > > We do take care that these flags are not breaking the following areas > after being applied. > > * Build process of the image > * Sanity test cases after the image is created > (Ada can give more details on the sanity report of the image generated > with these flags) > > If running the sanity tests are not enough to prove that a change in > compiler flags do not affect functionality, please gave us the right > path to follow. > > As mentioned before, this is a preliminary report, and that we will be > following up with a test plan for functional & performance test plans > for the services as a next step. > > Hope this email helps to clarify some questions related to the flags > and start the follow-up discussion. > > > > Thanks for the context Victor, it's very helpful to me. > > > > Hi Curtis, glad it helps, it was fun to do the research > > > > One thing I want to mention is something the Kata Containers team was talking about at the Berlin OpenStack summit, which is when many small performance hits start to add up. They have to be careful to ensure they don't have a bunch of smallish looking changes that add up to a large performance hit over a longer period of time. > > > > You are right, it's a valid point that we need to take care too > > > > Overall I'm sure the StarlingX project would like to have some performance testing, if we don't already, though that can be challenging for an open source project. I had mentioned OPNFV's Functest and related projects on the TSC call, but now seeing which components are affected I'm not sure that would be directly helpful. I look forward to further discussions around this area. > > > > Thanks for let me know that, I will take a look at OPNFV's functest and other projects before the next TSC of 2019 > > > > I will do my best to came up with a proposal for a better performance testing. > > > > Thanks > > > > Victor Rodriguez > > > > Thanks, > > Curtis > > > > > Regards > > Victor Rodriguez > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss > > > > -- > > Blog: serverascode.com
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Ada, Regarding ....
Yes, we executed several test cases covering what Ken mentions (manually). What I'm not sure is about heartbeat loss, but let me check. What we can do is to build a test plan and submit it for revision. When do you need it (and please, don't say tomorrow)?
What are you not sure about regarding heartbeat loss ? I assume testing and detection. Suggest reboot an in-service (unlocked-enabled-available) node and see that there are heartbeat communication loss and inservice failure alarms, that the system detects and Gracefully Recovers the rebooted host and clears said alarms once the host recovers back in-service. Eric.
-----Original Message----- From: Cabrales, Ada [mailto:ada.cabrales@intel.com] Sent: Monday, January 21, 2019 4:51 PM To: Young, Ken; Victor Rodriguez Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security
Comments inline
-----Original Message----- From: Young, Ken [mailto:Ken.Young@windriver.com] Sent: Friday, January 18, 2019 9:34 AM To: Victor Rodriguez <vm.rod25@gmail.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security
See inline.
On 2019-01-17, 5:34 PM, "Victor Rodriguez" <vm.rod25@gmail.com> wrote:
On Wed, Jan 2, 2019 at 10:35 AM Young, Ken <Ken.Young@windriver.com> wrote: > > Victor, > > > > Security work is never completed. There is always a long list of inventive new vulnerabilities and a laundry list of hardening work to be completed. The vulnerability work, considering the severity, is generally urgent. Hardening work is not urgent but important. In this case, we are dealing with a hardening initiative that focuses on a small area of the code. > > > > The challenge is that these small change proposed have larger implications. As was pointed out on the gerrit reviews, performance and / or functional testing is required.
Hi Ken
Just to follow the idea of this mail after hollliday break, you mention that:
My concern is that we affect the timing / behaviour of stx-ha and stx-metal such that they do not work together in some scenarios. This will need to be tested and is certainly larger than a sanity.
Could you please help to describe n human words, ( I can do the script ) how a good test to probe this would look like? If you provide me with a basic description of the security test I could help writing the first draft of a code test that help us to prove if the flags break the functionality
Victor,
At a high level, we need to regress the behaviour of stx-ha and stx-metal to ensure that there is functional issues introduced by the change to the compiler. As well, we need to look at the system behaviour of ha and metal to ensure no changes have been introduced which affect has behaviour:
- SWACT detection and time - Multinode failure avoidance - Heartbeat loss - lock / unlock - etc
I believe that Ada has the test for ha and metal. Please review.
Yes, we executed several test cases covering what Ken mentions (manually). What I'm not sure is about heartbeat loss, but let me check. What we can do is to build a test plan and submit it for revision. When do you need it (and please, don't say tomorrow)?
Ada
Regards, Ken Y
thanks
Victor R > > > > Also, I am wondering if there is a way to phase the effort. For example, is there a way to break up the flag changes such that the warnings are separated from the flags which change the compiled code? That way, we are not trying to jam everything through at once. > > > > Hope this helps. Happy to discuss when you return from Holliday. > > > > Regards, > > Ken Y > > > > From: Victor Rodriguez <vm.rod25@gmail.com> > Date: Friday, December 28, 2018 at 7:34 PM > To: Curtis <serverascode@gmail.com> > Cc: "starlingx-discuss@lists.starlingx.io" <starlingx- discuss@lists.starlingx.io> > Subject: Re: [Starlingx-discuss] Recommended C/C++ compiler flag for security > > > > > > On Fri, Dec 21, 2018, 07:08 Curtis <serverascode@gmail.com wrote: > > > > > > On Thu, Dec 20, 2018 at 3:47 PM Victor Rodriguez <vm.rod25@gmail.com> wrote: > > Hi StarlingX community > > We can all agree that security is an important feature to be taken > into consideration in any SW project. In the aim of improving the > security of the StarlingX project, we have been taking the task to > propose the use of some compiler flags that prevent and detect some > security holes, especially by buffer overflow that could lead into ROP > attacks. > > The list of flags that we are proposing are : > > Stack-based Buffer Overrun Detection: CFLAGS=”-fstack-protector- strong” > > Fortify source: CFLAGS="-O2 -D_FORTIFY_SOURCE=2" > Format string vulnerabilities: CFLAGS="-Wformat -Wformat- security" > Stack execution protection: LDFLAGS="-z noexecstack" > Data relocation and protection (RELRO): LDLFAGS="-z relro -z now" > > > These are being analyzed in the following Gerrit reviews (thanks a lot > for all the good feedback) > > https://review.openstack.org/#/c/623608/ > https://review.openstack.org/#/c/623603/ > https://review.openstack.org/#/c/623601/ > https://review.openstack.org/#/c/623599/ > > As requested in the Gerrit reviews, there is a proper need to first > understand what these compiler flags do and what is the impact they > have at the functional and performance area of the project. This is a > preliminary report, we will be following up with a test plan for > functional & performance test plans for the services as a next step. > This report includes: > > * Detailed description of what the compiler flag does > * Code example that shows how does it work to prevent attacks > * If there is a change in the binary, we create a microbenchmark that > shows us how the flag impact the performance > > https://github.com/VictorRodriguez/hobbies/tree/master/c_programing_ex ercises/cflags_security > > As a result of the microbenchmark, the performance impact is not > relevant ( less than 1% ) using an Ubuntu x86 system ( GCC 5 ) (more > details on the HW and SW specification upon requests) > > The areas of the code we are suggesting on the patches are: > > * stx-ha > * stx-metal > * stx-nfv > * stx-fault > > We do take care that these flags are not breaking the following areas > after being applied. > > * Build process of the image > * Sanity test cases after the image is created > (Ada can give more details on the sanity report of the image generated > with these flags) > > If running the sanity tests are not enough to prove that a change in > compiler flags do not affect functionality, please gave us the right > path to follow. > > As mentioned before, this is a preliminary report, and that we will be > following up with a test plan for functional & performance test plans > for the services as a next step. > > Hope this email helps to clarify some questions related to the flags > and start the follow-up discussion. > > > > Thanks for the context Victor, it's very helpful to me. > > > > Hi Curtis, glad it helps, it was fun to do the research > > > > One thing I want to mention is something the Kata Containers team was talking about at the Berlin OpenStack summit, which is when many small performance hits start to add up. They have to be careful to ensure they don't have a bunch of smallish looking changes that add up to a large performance hit over a longer period of time. > > > > You are right, it's a valid point that we need to take care too > > > > Overall I'm sure the StarlingX project would like to have some performance testing, if we don't already, though that can be challenging for an open source project. I had mentioned OPNFV's Functest and related projects on the TSC call, but now seeing which components are affected I'm not sure that would be directly helpful. I look forward to further discussions around this area. > > > > Thanks for let me know that, I will take a look at OPNFV's functest and other projects before the next TSC of 2019 > > > > I will do my best to came up with a proposal for a better performance testing. > > > > Thanks > > > > Victor Rodriguez > > > > Thanks, > > Curtis > > > > > Regards > > Victor Rodriguez > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss > > > > -- > > Blog: serverascode.com
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (7)
-
Botello Ortega, Luis
-
Cabrales, Ada
-
Cordoba Malibran, Erich
-
Curtis
-
MacDonald, Eric
-
Victor Rodriguez
-
Young, Ken