[Starlingx-discuss] pmond and systemd handling for fm-rest-api
Hi, I'm converting the fm-api init script to be handled completely by systemd, however I'm seeing that, once I created the unit file, pmod is constantly launching fm-api (which causes a failure in fm-api is there is a conflict in the network address usage). So, this is my unit file for fm-api. [Unit] Description=Fault Management REST API Service After=nfscommon.service sw-patch.service After=network-online.target systemd-udev-settle.service [Service] Type=simple RemainAfterExit=yes User=root ExecStart=/usr/bin/fm-api --config-file=/etc/fm/fm.conf PIDFile=/var/run/fm-api.pid [Install] WantedBy=multi-user.target Then I edit the /etc/pmon.d/fm-api.conf file to this: [process] process = fm-api pidfile = /var/run/fm-api.pid service = fm-api.service style = systemd ; ocf or lsb severity = major ; minor, major, critical restarts = 3 ; restarts before error assertion interval = 5 ; number of seconds to wait between restarts debounce = 20 ; number of seconds to wait before degrade clear What I've seen is that pmond is looking for the pidfile, but this is not created as this was done by the deleted init script[0]. Also, according to this comment[1], looking for the pidfile is the last resource but I can see frequent attempts to find this file, see below: $ sudo strace -e trace=file -p 86386 --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- open("/var/run/fm-api.pid", O_RDONLY) = -1 ENOENT (No such file or directory) --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3222970, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3222617, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x7efa, si_overrun=0, si_value={int=6853136, ptr=0x689210}} --- --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- open("/var/run/fm-api.pid", O_RDONLY) = -1 ENOENT (No such file or directory) --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x7ed1, si_overrun=0, si_value={int=6883584, ptr=0x690900}} --- Does anybody know what can be causing this behavior ? Thanks -Erich - [0] https://opendev.org/starlingx/fault/src/branch/master/fm-rest-api/fm/scripts... - [1] https://opendev.org/starlingx/metal/src/branch/master/mtce/src/pmon/pmonHdlr...
Erich, systemD will not create the PID file for you, the PIDFile statement is used by systemD to locate the PID file, created by the launched service, in order to check that the service has started and is running if the monitor option is requested. PIDFile is used in the case of a "type=forked". What ever "type" is used, the PID file is not created by systemD but by the launched service itself. So your issue, as you start fm-api directly, you need to get the PID written in the proper place. An ExecStartPost statement would likely help you. You could try to add the following line to your service file. It should create your missing PID file that is required by pmond. ExecStartPost=/bin/sh -c 'umask 022; pgrep fm-api > /var/run/fm-api.pid' Regards Dominig On 10/09/2019 17:43, Cordoba Malibran, Erich wrote:
Hi,
I'm converting the fm-api init script to be handled completely by systemd, however I'm seeing that, once I created the unit file, pmod is constantly launching fm-api (which causes a failure in fm-api is there is a conflict in the network address usage).
So, this is my unit file for fm-api.
[Unit] Description=Fault Management REST API Service After=nfscommon.service sw-patch.service After=network-online.target systemd-udev-settle.service
[Service] Type=simple RemainAfterExit=yes User=root ExecStart=/usr/bin/fm-api --config-file=/etc/fm/fm.conf PIDFile=/var/run/fm-api.pid
[Install] WantedBy=multi-user.target
Then I edit the /etc/pmon.d/fm-api.conf file to this:
[process] process = fm-api pidfile = /var/run/fm-api.pid service = fm-api.service style = systemd ; ocf or lsb severity = major ; minor, major, critical restarts = 3 ; restarts before error assertion interval = 5 ; number of seconds to wait between restarts debounce = 20 ; number of seconds to wait before degrade clear
What I've seen is that pmond is looking for the pidfile, but this is not created as this was done by the deleted init script[0].
Also, according to this comment[1], looking for the pidfile is the last resource but I can see frequent attempts to find this file, see below:
$ sudo strace -e trace=file -p 86386
--- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- open("/var/run/fm-api.pid", O_RDONLY) = -1 ENOENT (No such file or directory) --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3222970, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3222617, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x7efa, si_overrun=0, si_value={int=6853136, ptr=0x689210}} --- --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- open("/var/run/fm-api.pid", O_RDONLY) = -1 ENOENT (No such file or directory) --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x7ed1, si_overrun=0, si_value={int=6883584, ptr=0x690900}} ---
Does anybody know what can be causing this behavior ?
Thanks
-Erich
- [0] https://opendev.org/starlingx/fault/src/branch/master/fm-rest-api/fm/scripts... - [1] https://opendev.org/starlingx/metal/src/branch/master/mtce/src/pmon/pmonHdlr...
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
-- Dominig ar Foll Senior Software Architect Intel Open Source Technology Centre
You should be able to use $MAINPID for the PID, rather than pgrep. We have a number of cases in StarlingX repos doing something like: ExecStartPost=/bin/bash -c 'echo $MAINPID > /var/run/XXXXX.pid' Note that those cases do not set PIDFile. The following thread has a description of how systemd determines $MAINPID. If PIDFile is set, it would read the value from there, which would conflict with using it to write the pidfile: https://lists.freedesktop.org/archives/systemd-devel/2013-February/009201.ht... -----Original Message----- From: Dominig ar Foll (Intel Open Source) [mailto:dominig.arfoll@fridu.net] Sent: Tuesday, September 10, 2019 2:35 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] pmond and systemd handling for fm-rest-api Erich, systemD will not create the PID file for you, the PIDFile statement is used by systemD to locate the PID file, created by the launched service, in order to check that the service has started and is running if the monitor option is requested. PIDFile is used in the case of a "type=forked". What ever "type" is used, the PID file is not created by systemD but by the launched service itself. So your issue, as you start fm-api directly, you need to get the PID written in the proper place. An ExecStartPost statement would likely help you. You could try to add the following line to your service file. It should create your missing PID file that is required by pmond. ExecStartPost=/bin/sh -c 'umask 022; pgrep fm-api > /var/run/fm-api.pid' Regards Dominig On 10/09/2019 17:43, Cordoba Malibran, Erich wrote:
Hi,
I'm converting the fm-api init script to be handled completely by systemd, however I'm seeing that, once I created the unit file, pmod is constantly launching fm-api (which causes a failure in fm-api is there is a conflict in the network address usage).
So, this is my unit file for fm-api.
[Unit] Description=Fault Management REST API Service After=nfscommon.service sw-patch.service After=network-online.target systemd-udev-settle.service
[Service] Type=simple RemainAfterExit=yes User=root ExecStart=/usr/bin/fm-api --config-file=/etc/fm/fm.conf PIDFile=/var/run/fm-api.pid
[Install] WantedBy=multi-user.target
Then I edit the /etc/pmon.d/fm-api.conf file to this:
[process] process = fm-api pidfile = /var/run/fm-api.pid service = fm-api.service style = systemd ; ocf or lsb severity = major ; minor, major, critical restarts = 3 ; restarts before error assertion interval = 5 ; number of seconds to wait between restarts debounce = 20 ; number of seconds to wait before degrade clear
What I've seen is that pmond is looking for the pidfile, but this is not created as this was done by the deleted init script[0].
Also, according to this comment[1], looking for the pidfile is the last resource but I can see frequent attempts to find this file, see below:
$ sudo strace -e trace=file -p 86386
--- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- open("/var/run/fm-api.pid", O_RDONLY) = -1 ENOENT (No such file or directory) --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3222970, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3222617, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x7efa, si_overrun=0, si_value={int=6853136, ptr=0x689210}} --- --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- open("/var/run/fm-api.pid", O_RDONLY) = -1 ENOENT (No such file or directory) --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x7ed1, si_overrun=0, si_value={int=6883584, ptr=0x690900}} ---
Does anybody know what can be causing this behavior ?
Thanks
-Erich
- [0] https://opendev.org/starlingx/fault/src/branch/master/fm-rest-api/fm/scripts... - [1] https://opendev.org/starlingx/metal/src/branch/master/mtce/src/pmon/pmonHdlr...
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
-- Dominig ar Foll Senior Software Architect Intel Open Source Technology Centre _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Don, Thanks for the hint! It worked, now pmond is not launching the process over an over __ We'll this ExecStartPost line for all converted unit files managed by pmond. -Erich On 9/10/19, 1:43 PM, "Penney, Don" <Don.Penney@windriver.com> wrote: You should be able to use $MAINPID for the PID, rather than pgrep. We have a number of cases in StarlingX repos doing something like: ExecStartPost=/bin/bash -c 'echo $MAINPID > /var/run/XXXXX.pid' Note that those cases do not set PIDFile. The following thread has a description of how systemd determines $MAINPID. If PIDFile is set, it would read the value from there, which would conflict with using it to write the pidfile: https://lists.freedesktop.org/archives/systemd-devel/2013-February/009201.ht... -----Original Message----- From: Dominig ar Foll (Intel Open Source) [mailto:dominig.arfoll@fridu.net] Sent: Tuesday, September 10, 2019 2:35 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] pmond and systemd handling for fm-rest-api Erich, systemD will not create the PID file for you, the PIDFile statement is used by systemD to locate the PID file, created by the launched service, in order to check that the service has started and is running if the monitor option is requested. PIDFile is used in the case of a "type=forked". What ever "type" is used, the PID file is not created by systemD but by the launched service itself. So your issue, as you start fm-api directly, you need to get the PID written in the proper place. An ExecStartPost statement would likely help you. You could try to add the following line to your service file. It should create your missing PID file that is required by pmond. ExecStartPost=/bin/sh -c 'umask 022; pgrep fm-api > /var/run/fm-api.pid' Regards Dominig On 10/09/2019 17:43, Cordoba Malibran, Erich wrote: > Hi, > > I'm converting the fm-api init script to be handled completely by > systemd, however I'm seeing that, once I created the unit file, pmod is > constantly launching fm-api (which causes a failure in fm-api is there > is a conflict in the network address usage). > > So, this is my unit file for fm-api. > > [Unit] > Description=Fault Management REST API Service > After=nfscommon.service sw-patch.service > After=network-online.target systemd-udev-settle.service > > [Service] > Type=simple > RemainAfterExit=yes > User=root > ExecStart=/usr/bin/fm-api --config-file=/etc/fm/fm.conf > PIDFile=/var/run/fm-api.pid > > [Install] > WantedBy=multi-user.target > > > Then I edit the /etc/pmon.d/fm-api.conf file to this: > > [process] > process = fm-api > pidfile = /var/run/fm-api.pid > service = fm-api.service > style = systemd ; ocf or lsb > severity = major ; minor, major, critical > restarts = 3 ; restarts before error assertion > interval = 5 ; number of seconds to wait between restarts > debounce = 20 ; number of seconds to wait before degrade > clear > > > What I've seen is that pmond is looking for the pidfile, but this is > not created as this was done by the deleted init script[0]. > > Also, according to this comment[1], looking for the pidfile is the last > resource but I can see frequent attempts to find this file, see below: > > $ sudo strace -e trace=file -p 86386 > > --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- > open("/var/run/fm-api.pid", O_RDONLY) = -1 ENOENT (No such file or directory) > --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3222970, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- > --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3222617, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- > --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- > --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- > --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x7efa, si_overrun=0, si_value={int=6853136, ptr=0x689210}} --- > --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x20, si_overrun=0, si_value={int=6882944, ptr=0x690680}} --- > open("/var/run/fm-api.pid", O_RDONLY) = -1 ENOENT (No such file or directory) > --- SIGRT_2 {si_signo=SIGRT_2, si_code=SI_TIMER, si_timerid=0x7ed1, si_overrun=0, si_value={int=6883584, ptr=0x690900}} --- > > > Does anybody know what can be causing this behavior ? > > Thanks > > > -Erich > > > - [0] https://opendev.org/starlingx/fault/src/branch/master/fm-rest-api/fm/scripts... > - [1] https://opendev.org/starlingx/metal/src/branch/master/mtce/src/pmon/pmonHdlr... > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss -- Dominig ar Foll Senior Software Architect Intel Open Source Technology Centre _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (3)
-
Cordoba Malibran, Erich
-
Dominig ar Foll (Intel Open Source)
-
Penney, Don