[Starlingx-discuss] [multi os] Need of systemd for adoption of flock services in other os
Hi community As part of the multi-OS effort for multi os, the first steps were to make the flock services build and run on other OS such as openSUSE system. The build of the flock services is complete and now OBS is being used for it ( some minor details are under fix ). After that, the runtime dependencies were addressed. One of the characteristics STX flocks have is that If they are managed by Service Management (SM) then sysvinit scripts are used instead of systemd SM is a great component from STX which has cool features such as: * Manages high availability of critical infrastructure and cluster services • Use of multiple messaging paths to avoid split-brain communication failures • Uses cluster connectivity data to determine the communication failures The fact that is based on sysvinit does not prevent us from running other flock services on openSUSE, or that someone else wants to manage the services by something else apart from SM. However we would like to ask the community how important is the standardization of systemd for the adoption of STX flock services on other OS The proposal would be to have most of the existing services handled by SM via systemd in order to deprecate the sysvinit scripts. SM will directly manage the services, via systemd commands. A bit of rewrite to current scripts might be necessary. This way there is a consistent set of scripts/unit/service files. So, SM would need to be extended to understand systemd interactions Any feedback from anyone on the community more than welcome Regards Victor Rodriguez
On 12/09/2019 19:37, Victor Rodriguez wrote:
... The proposal would be to have most of the existing services handled by SM via systemd in order to deprecate the sysvinit scripts. SM will directly manage the services, via systemd commands. A bit of rewrite to current scripts might be necessary. This way there is a consistent set of scripts/unit/service files. So, SM would need to be extended to understand systemd interactions Victor,
all modern distros are using systemD and today if I look at the Warning raised by OpenSUSE build system (OBS), [build error log at the end of this email], it is clear that the current mix of systemD/init.d/direct script calls, will not pass any distro QA as it is. I already had to tweak rpmlintrc to reduce the badness in order to get packages to be accepted to be published by the OBS. But for me the fundamental issue is not on how bad is look but how dangerous it is from a security point of view. SystemD is by far the best way to start a service to be run in isolation. We can restrict the privileges, enforce cgroup limits, isolate via namespaces. When it come to provide a secure system, it's by far the easiest launch and maintain processes with reduced attack surface. Now in StrarlingX we have many processes which are to be started and controlled by the Service Manager (SM). What we should do, is to modify SM in order for SM to use SystemD calls to start services which are to be monitored by SM. For those services, we shall not let systemD deamon starts them automatically, nor to monitor them for potential restart (easy, just a config). We can still use automatic dependency to let systemD start underlying required service (not managed by SM) where it make sense. Replacing direct script calls to start service in SM, by calls to systemD cannot be that complex for someone who know where the calls are hidden in the code. We could do a hack by having an init script to call systemD but that would be a mistake as trouble shooting errors would become very messy, security latency poor. The best would be to use systemD library API calls (requires systemD >= 219 not an issue for OpenSUSE but what about CentOS) in order to do the equivalent of systemctl start service_name@unit_number.service without using system(), popen(), or exec() what would lake the system, safer, quicker and easier to debug. -- Dominig ar Foll Senior Software Architect Intel Open Source Technology Centre ---------- rpmlint report building worker-utils ------------ [ 3s] RPMLINT report: [ 3s] =============== [ 3s] worker-utils.noarch: W: no-reload-entry /etc/init.d/cpumap_functions.sh [ 3s] worker-utils.noarch: W: no-reload-entry /etc/init.d/task_affinity_functions.sh [ 3s] In your init script (/etc/rc.d/init.d/your_file), you don't have a 'reload' [ 3s] entry, which is necessary for good functionality. [ 3s] [ 3s] worker-utils.noarch: W: no-status-entry /etc/init.d/cpumap_functions.sh [ 3s] In your init script (/etc/rc.d/init.d/your_file), you don't have a 'status' [ 3s] entry, which is necessary for good functionality. [ 3s] [ 3s] worker-utils.noarch: W: suse-deprecated-init-script /etc/init.d/affine-platform.sh [ 3s] worker-utils.noarch: W: suse-deprecated-init-script /etc/init.d/affine-tasks.sh [ 3s] worker-utils.noarch: W: suse-deprecated-init-script /etc/init.d/cpumap_functions.sh [ 3s] worker-utils.noarch: W: suse-deprecated-init-script /etc/init.d/task_affinity_functions.sh [ 3s] SysV init scripts are deprecated. Please migrate to systemd service files. [ 3s] [ 3s] worker-utils.noarch: W: suse-missing-rclink affine-platform.sh [ 3s] worker-utils.noarch: W: suse-missing-rclink affine-tasks [ 3s] worker-utils.noarch: W: suse-missing-rclink affine-tasks.sh [ 3s] worker-utils.noarch: W: suse-missing-rclink cpumap_functions.sh [ 3s] worker-utils.noarch: W: suse-missing-rclink task_affinity_functions.sh [ 3s] The package contains an init script or systemd service file but lacks the [ 3s] symlink /usr/sbin/rcFOO -> /usr/sbin/service [ 3s] [ 3s] worker-utils.noarch: W: suse-systemd-shadowed-initscript affine-platform.sh [ 3s] The package contains both an init script and a systemd service file. Please [ 3s] decide for one. [ 3s] [ 3s] 2 packages and 0 specfiles checked; 0 errors, 13 warnings.
On Thu, Sep 12, 2019 at 1:42 PM Dominig ar Foll (Intel Open Source) <dominig.arfoll@fridu.net> wrote:
On 12/09/2019 19:37, Victor Rodriguez wrote:
... The proposal would be to have most of the existing services handled by SM via systemd in order to deprecate the sysvinit scripts. SM will directly manage the services, via systemd commands. A bit of rewrite to current scripts might be necessary. This way there is a consistent set of scripts/unit/service files. So, SM would need to be extended to understand systemd interactions Victor,
all modern distros are using systemD and today if I look at the Warning raised by OpenSUSE build system (OBS), [build error log at the end of this email], it is clear that the current mix of systemD/init.d/direct script calls, will not pass any distro QA as it is. I already had to tweak rpmlintrc to reduce the badness in order to get packages to be accepted to be published by the OBS. But for me the fundamental issue is not on how bad is look but how dangerous it is from a security point of view.
SystemD is by far the best way to start a service to be run in isolation. We can restrict the privileges, enforce cgroup limits, isolate via namespaces. When it come to provide a secure system, it's by far the easiest launch and maintain processes with reduced attack surface.
Agree
Now in StrarlingX we have many processes which are to be started and controlled by the Service Manager (SM). What we should do, is to modify SM in order for SM to use SystemD calls to start services which are to be monitored by SM. For those services, we shall not let systemD deamon starts them automatically, nor to monitor them for potential restart (easy, just a config). We can still use automatic dependency to let systemD start underlying required service (not managed by SM) where it make sense.
Ok, I see that as a valid solution. I would like to ask feedback from Bin and team
Replacing direct script calls to start service in SM, by calls to systemD cannot be that complex for someone who know where the calls are hidden in the code. We could do a hack by having an init script to call systemD but that would be a mistake as trouble shooting errors would become very messy, security latency poor.
The best would be to use systemD library API calls (requires systemD >= 219 not an issue for OpenSUSE but what about CentOS)
We do have 219 in centOS # systemctl --version systemd 219 http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_build... in order to do the
equivalent of systemctl start service_name@unit_number.service without using system(), popen(), or exec() what would lake the system, safer, quicker and easier to debug.
-- Dominig ar Foll Senior Software Architect Intel Open Source Technology Centre
---------- rpmlint report building worker-utils ------------ [ 3s] RPMLINT report: [ 3s] =============== [ 3s] worker-utils.noarch: W: no-reload-entry /etc/init.d/cpumap_functions.sh [ 3s] worker-utils.noarch: W: no-reload-entry /etc/init.d/task_affinity_functions.sh [ 3s] In your init script (/etc/rc.d/init.d/your_file), you don't have a 'reload' [ 3s] entry, which is necessary for good functionality. [ 3s] [ 3s] worker-utils.noarch: W: no-status-entry /etc/init.d/cpumap_functions.sh [ 3s] In your init script (/etc/rc.d/init.d/your_file), you don't have a 'status' [ 3s] entry, which is necessary for good functionality. [ 3s] [ 3s] worker-utils.noarch: W: suse-deprecated-init-script /etc/init.d/affine-platform.sh [ 3s] worker-utils.noarch: W: suse-deprecated-init-script /etc/init.d/affine-tasks.sh [ 3s] worker-utils.noarch: W: suse-deprecated-init-script /etc/init.d/cpumap_functions.sh [ 3s] worker-utils.noarch: W: suse-deprecated-init-script /etc/init.d/task_affinity_functions.sh [ 3s] SysV init scripts are deprecated. Please migrate to systemd service files. [ 3s] [ 3s] worker-utils.noarch: W: suse-missing-rclink affine-platform.sh [ 3s] worker-utils.noarch: W: suse-missing-rclink affine-tasks [ 3s] worker-utils.noarch: W: suse-missing-rclink affine-tasks.sh [ 3s] worker-utils.noarch: W: suse-missing-rclink cpumap_functions.sh [ 3s] worker-utils.noarch: W: suse-missing-rclink task_affinity_functions.sh [ 3s] The package contains an init script or systemd service file but lacks the [ 3s] symlink /usr/sbin/rcFOO -> /usr/sbin/service [ 3s] [ 3s] worker-utils.noarch: W: suse-systemd-shadowed-initscript affine-platform.sh [ 3s] The package contains both an init script and a systemd service file. Please [ 3s] decide for one. [ 3s] [ 3s] 2 packages and 0 specfiles checked; 0 errors, 13 warnings.
Thanks for the log, it is much more clear for me the need for this change. Thanks
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hello, One of the comment/worry that have seen on the instant message was about the potential regression risk related to the move to systemD. I would like to share the exchange as it relevant for all the community concerned by that topic. Question: I am afraid that changing this will impact flocks functionality or big regression testing is required. Response: Adding security and multiOS support will require some retest for sure. But if we add a plugin we could keep different DBs for different OSes. One calling for init.d scripts for legacy support while for multiOS a new DB would call for systemD. With that model, we would minimise risk, as only minimalistic regression test would be required until the DB content is changed for a given OS. -- Dominig ar Foll Senior Software Architect Intel Open Source Technology Centre
Concerning SW upgrade needs, do we need to keep DB version to allow these two versions can still communicate? Sent from my iPhone
On Sep 16, 2019, at 22:46, Dominig ar Foll (Intel Open Source) <dominig.arfoll@fridu.net> wrote:
Hello,
One of the comment/worry that have seen on the instant message was about the potential regression risk related to the move to systemD. I would like to share the exchange as it relevant for all the community concerned by that topic.
Question: I am afraid that changing this will impact flocks functionality or big regression testing is required.
Response: Adding security and multiOS support will require some retest for sure. But if we add a plugin we could keep different DBs for different OSes. One calling for init.d scripts for legacy support while for multiOS a new DB would call for systemD. With that model, we would minimise risk, as only minimalistic regression test would be required until the DB content is changed for a given OS.
-- Dominig ar Foll Senior Software Architect Intel Open Source Technology Centre
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 16/09/2019 17:22, Xie, Cindy wrote:
Concerning SW upgrade needs, do we need to keep DB version to allow these two versions can still communicate?
Obviously moving from one system to an other one would be a major update and would require a cold re-start. But activating many security features would require similar cold restart. This is why having a plugin that would allow to activate the changes from initV to systemD via DB configuration is the way to go. The service DB being unique to a cluster the system would remain consistent until the big change is activated. Hot update is not easy but can be done, hot upgrade, to be honest, has been rarely achieved in the real world. -- Dominig ar Foll Senior Software Architect Intel Open Source Technology Centre
On Mon, Sep 16, 2019 at 9:45 AM Dominig ar Foll (Intel Open Source) <dominig.arfoll@fridu.net> wrote:
Hello,
One of the comment/worry that have seen on the instant message was about the potential regression risk related to the move to systemD. I would like to share the exchange as it relevant for all the community concerned by that topic.
Question: I am afraid that changing this will impact flocks functionality or big regression testing is required.
Response: Adding security and multiOS support will require some retest for sure. But if we add a plugin we could keep different DBs for different OSes. One calling for init.d scripts for legacy support while for multiOS a new DB would call for systemD. With that model, we would minimise risk, as only minimalistic regression test would be required until the DB content is changed for a given OS.
I like the idea, keep the current system working as it is and do not interfere with functionality in order to enable another OS has been part of our goals. The plugin solution is a great idea. The testing part is naturally, the more features we have / os we run the more testing it is needed. Does anyone one on the community think that this change need a spec? Thoughts from other OSs ( Debian / Yocto ) Regards Victor R
-- Dominig ar Foll Senior Software Architect Intel Open Source Technology Centre
Cold restart vs hot restart is one thing, but I am more concerning about one node in system and another node in sysinit call while they are in the same cluster and SM needs to manage those different nodes still. -----Original Message----- From: Victor Rodriguez <vm.rod25@gmail.com> Sent: Tuesday, September 17, 2019 5:11 AM To: Dominig ar Foll (Intel Open Source) <dominig.arfoll@fridu.net> Cc: starlingx <starlingx-discuss@lists.starlingx.io>; Bin.Qian@windriver.com Subject: Re: [Starlingx-discuss] [multi os] Need of systemd for adoption of flock services in other os (regression worries) On Mon, Sep 16, 2019 at 9:45 AM Dominig ar Foll (Intel Open Source) <dominig.arfoll@fridu.net> wrote:
Hello,
One of the comment/worry that have seen on the instant message was about the potential regression risk related to the move to systemD. I would like to share the exchange as it relevant for all the community concerned by that topic.
Question: I am afraid that changing this will impact flocks functionality or big regression testing is required.
Response: Adding security and multiOS support will require some retest for sure. But if we add a plugin we could keep different DBs for different OSes. One calling for init.d scripts for legacy support while for multiOS a new DB would call for systemD. With that model, we would minimise risk, as only minimalistic regression test would be required until the DB content is changed for a given OS.
I like the idea, keep the current system working as it is and do not interfere with functionality in order to enable another OS has been part of our goals. The plugin solution is a great idea. The testing part is naturally, the more features we have / os we run the more testing it is needed. Does anyone one on the community think that this change need a spec? Thoughts from other OSs ( Debian / Yocto ) Regards Victor R
-- Dominig ar Foll Senior Software Architect Intel Open Source Technology Centre
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
This just seems wrong, multiple implementations where there is no technical reason to do so. The work should be done to migrate the sysvint scripts to systemd and have a common implementation across the different OS's. Brent -----Original Message----- From: Victor Rodriguez [mailto:vm.rod25@gmail.com] Sent: Monday, September 16, 2019 5:11 PM To: Dominig ar Foll (Intel Open Source) <dominig.arfoll@fridu.net> Cc: starlingx <starlingx-discuss@lists.starlingx.io>; Qian, Bin <Bin.Qian@windriver.com> Subject: Re: [Starlingx-discuss] [multi os] Need of systemd for adoption of flock services in other os (regression worries) On Mon, Sep 16, 2019 at 9:45 AM Dominig ar Foll (Intel Open Source) <dominig.arfoll@fridu.net> wrote:
Hello,
One of the comment/worry that have seen on the instant message was about the potential regression risk related to the move to systemD. I would like to share the exchange as it relevant for all the community concerned by that topic.
Question: I am afraid that changing this will impact flocks functionality or big regression testing is required.
Response: Adding security and multiOS support will require some retest for sure. But if we add a plugin we could keep different DBs for different OSes. One calling for init.d scripts for legacy support while for multiOS a new DB would call for systemD. With that model, we would minimise risk, as only minimalistic regression test would be required until the DB content is changed for a given OS.
I like the idea, keep the current system working as it is and do not interfere with functionality in order to enable another OS has been part of our goals. The plugin solution is a great idea. The testing part is naturally, the more features we have / os we run the more testing it is needed. Does anyone one on the community think that this change need a spec? Thoughts from other OSs ( Debian / Yocto ) Regards Victor R
-- Dominig ar Foll Senior Software Architect Intel Open Source Technology Centre
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (4)
-
Dominig ar Foll (Intel Open Source)
-
Rowsell, Brent
-
Victor Rodriguez
-
Xie, Cindy