Systemd and process spawning
Don't normally post here but I am ripping my hair out over this one.
I have a Python script that forks when it launches, and is responsible for starting a bunch of other processes.
This script used to be launched at startup via sysvinit, but recently I upgraded to Debian Jessie so have adapted it to launch via systemd.
Unfortunately, I'm running into an issue I can't work out. When you launch the script directly in a user shell, it launches it's child processes correctly, and when the script exits the child processes are orphaned and continue to run.
When launched Via systemd, if the parent process exits, the children all exit too (Well, the Screen's that they launch in die and appear as Dead???)
Ideally I need to be able to restart the parent script without killing all the child processes, is there something that I am missing?
Thanks!
[Unit]
Description=Server commander
After=network.target
[Service]
User=serveruser
Type=forking
PIDFile=/var/Server/Server.pid
ExecStart=/var/Server/Server.py
ExecStop=/bin/kill -s TERM $MAINPID
[Install]
WantedBy=multi-user.target
Edit:
It's probably relevant for me to point out that the Python script is essentially a 'controller' for its child processes. It starts and stops servers in gnu screens as requested from a central server. It is normally always running, it doesn't spawn services and exit. There are cases however where I would like to be able to reload the script without killing child processes, even if that means the processes are orphaned off to pid 1. In fact, it wouldn't even matter if the Python script started off processes as a parent process, if that is even possible.
A better explanation of how it works:
- Systemd spawns /Server.py
- Server.py forks and writes the pid file for Systemd
- Server.py then spawns server processes in gnu screen based on its instructions
- Server.py continues to run to perform any restarts requested from the server
When launching without Systemd, Server.py can be restarted and the gnu screens it launches are unaffected. When launching with Systemd, when Server.py shuts down, instead of those screen processes being orphaned off to pid 1, they are killed.
systemd process-management sysvinit
add a comment |
Don't normally post here but I am ripping my hair out over this one.
I have a Python script that forks when it launches, and is responsible for starting a bunch of other processes.
This script used to be launched at startup via sysvinit, but recently I upgraded to Debian Jessie so have adapted it to launch via systemd.
Unfortunately, I'm running into an issue I can't work out. When you launch the script directly in a user shell, it launches it's child processes correctly, and when the script exits the child processes are orphaned and continue to run.
When launched Via systemd, if the parent process exits, the children all exit too (Well, the Screen's that they launch in die and appear as Dead???)
Ideally I need to be able to restart the parent script without killing all the child processes, is there something that I am missing?
Thanks!
[Unit]
Description=Server commander
After=network.target
[Service]
User=serveruser
Type=forking
PIDFile=/var/Server/Server.pid
ExecStart=/var/Server/Server.py
ExecStop=/bin/kill -s TERM $MAINPID
[Install]
WantedBy=multi-user.target
Edit:
It's probably relevant for me to point out that the Python script is essentially a 'controller' for its child processes. It starts and stops servers in gnu screens as requested from a central server. It is normally always running, it doesn't spawn services and exit. There are cases however where I would like to be able to reload the script without killing child processes, even if that means the processes are orphaned off to pid 1. In fact, it wouldn't even matter if the Python script started off processes as a parent process, if that is even possible.
A better explanation of how it works:
- Systemd spawns /Server.py
- Server.py forks and writes the pid file for Systemd
- Server.py then spawns server processes in gnu screen based on its instructions
- Server.py continues to run to perform any restarts requested from the server
When launching without Systemd, Server.py can be restarted and the gnu screens it launches are unaffected. When launching with Systemd, when Server.py shuts down, instead of those screen processes being orphaned off to pid 1, they are killed.
systemd process-management sysvinit
1
It is hard to provide a solution without having theServer.py
code and a description of how do the launched services fork (if they fork). However, generally speaking, this is a readiness protocol mismatch problem.
– intelfx
May 21 '15 at 22:29
BTW, theExecStop=
is not needed. systemd's default action on stop is to kill processes. You may want to take a look at the documentation forKillMode=
directive.
– intelfx
May 21 '15 at 22:31
1
And, finally... If there is no suitable readiness protocol (one ofsimple
orforking
, actually), the last resort would beType=oneshot
,RemainAfterExit=yes
andKillMode=control-group
.
– intelfx
May 21 '15 at 22:33
@intelfx Essentially the Python script launches a server in a screen using Subprocess.call. It's more complicated than that because the script receives commands from elsewhere telling it which screens to start and which to not. Which screens are available is also dynamic hence why they cant be systemd services in their own right. Ideally I dont want systemd to treat these screens as part of the service at all, but currently they get dumped in the same process group and die with the master if its restarted.
– Bottswana
May 22 '15 at 13:21
add a comment |
Don't normally post here but I am ripping my hair out over this one.
I have a Python script that forks when it launches, and is responsible for starting a bunch of other processes.
This script used to be launched at startup via sysvinit, but recently I upgraded to Debian Jessie so have adapted it to launch via systemd.
Unfortunately, I'm running into an issue I can't work out. When you launch the script directly in a user shell, it launches it's child processes correctly, and when the script exits the child processes are orphaned and continue to run.
When launched Via systemd, if the parent process exits, the children all exit too (Well, the Screen's that they launch in die and appear as Dead???)
Ideally I need to be able to restart the parent script without killing all the child processes, is there something that I am missing?
Thanks!
[Unit]
Description=Server commander
After=network.target
[Service]
User=serveruser
Type=forking
PIDFile=/var/Server/Server.pid
ExecStart=/var/Server/Server.py
ExecStop=/bin/kill -s TERM $MAINPID
[Install]
WantedBy=multi-user.target
Edit:
It's probably relevant for me to point out that the Python script is essentially a 'controller' for its child processes. It starts and stops servers in gnu screens as requested from a central server. It is normally always running, it doesn't spawn services and exit. There are cases however where I would like to be able to reload the script without killing child processes, even if that means the processes are orphaned off to pid 1. In fact, it wouldn't even matter if the Python script started off processes as a parent process, if that is even possible.
A better explanation of how it works:
- Systemd spawns /Server.py
- Server.py forks and writes the pid file for Systemd
- Server.py then spawns server processes in gnu screen based on its instructions
- Server.py continues to run to perform any restarts requested from the server
When launching without Systemd, Server.py can be restarted and the gnu screens it launches are unaffected. When launching with Systemd, when Server.py shuts down, instead of those screen processes being orphaned off to pid 1, they are killed.
systemd process-management sysvinit
Don't normally post here but I am ripping my hair out over this one.
I have a Python script that forks when it launches, and is responsible for starting a bunch of other processes.
This script used to be launched at startup via sysvinit, but recently I upgraded to Debian Jessie so have adapted it to launch via systemd.
Unfortunately, I'm running into an issue I can't work out. When you launch the script directly in a user shell, it launches it's child processes correctly, and when the script exits the child processes are orphaned and continue to run.
When launched Via systemd, if the parent process exits, the children all exit too (Well, the Screen's that they launch in die and appear as Dead???)
Ideally I need to be able to restart the parent script without killing all the child processes, is there something that I am missing?
Thanks!
[Unit]
Description=Server commander
After=network.target
[Service]
User=serveruser
Type=forking
PIDFile=/var/Server/Server.pid
ExecStart=/var/Server/Server.py
ExecStop=/bin/kill -s TERM $MAINPID
[Install]
WantedBy=multi-user.target
Edit:
It's probably relevant for me to point out that the Python script is essentially a 'controller' for its child processes. It starts and stops servers in gnu screens as requested from a central server. It is normally always running, it doesn't spawn services and exit. There are cases however where I would like to be able to reload the script without killing child processes, even if that means the processes are orphaned off to pid 1. In fact, it wouldn't even matter if the Python script started off processes as a parent process, if that is even possible.
A better explanation of how it works:
- Systemd spawns /Server.py
- Server.py forks and writes the pid file for Systemd
- Server.py then spawns server processes in gnu screen based on its instructions
- Server.py continues to run to perform any restarts requested from the server
When launching without Systemd, Server.py can be restarted and the gnu screens it launches are unaffected. When launching with Systemd, when Server.py shuts down, instead of those screen processes being orphaned off to pid 1, they are killed.
systemd process-management sysvinit
systemd process-management sysvinit
edited Jun 4 '15 at 15:35
Bottswana
asked May 21 '15 at 20:11
BottswanaBottswana
131118
131118
1
It is hard to provide a solution without having theServer.py
code and a description of how do the launched services fork (if they fork). However, generally speaking, this is a readiness protocol mismatch problem.
– intelfx
May 21 '15 at 22:29
BTW, theExecStop=
is not needed. systemd's default action on stop is to kill processes. You may want to take a look at the documentation forKillMode=
directive.
– intelfx
May 21 '15 at 22:31
1
And, finally... If there is no suitable readiness protocol (one ofsimple
orforking
, actually), the last resort would beType=oneshot
,RemainAfterExit=yes
andKillMode=control-group
.
– intelfx
May 21 '15 at 22:33
@intelfx Essentially the Python script launches a server in a screen using Subprocess.call. It's more complicated than that because the script receives commands from elsewhere telling it which screens to start and which to not. Which screens are available is also dynamic hence why they cant be systemd services in their own right. Ideally I dont want systemd to treat these screens as part of the service at all, but currently they get dumped in the same process group and die with the master if its restarted.
– Bottswana
May 22 '15 at 13:21
add a comment |
1
It is hard to provide a solution without having theServer.py
code and a description of how do the launched services fork (if they fork). However, generally speaking, this is a readiness protocol mismatch problem.
– intelfx
May 21 '15 at 22:29
BTW, theExecStop=
is not needed. systemd's default action on stop is to kill processes. You may want to take a look at the documentation forKillMode=
directive.
– intelfx
May 21 '15 at 22:31
1
And, finally... If there is no suitable readiness protocol (one ofsimple
orforking
, actually), the last resort would beType=oneshot
,RemainAfterExit=yes
andKillMode=control-group
.
– intelfx
May 21 '15 at 22:33
@intelfx Essentially the Python script launches a server in a screen using Subprocess.call. It's more complicated than that because the script receives commands from elsewhere telling it which screens to start and which to not. Which screens are available is also dynamic hence why they cant be systemd services in their own right. Ideally I dont want systemd to treat these screens as part of the service at all, but currently they get dumped in the same process group and die with the master if its restarted.
– Bottswana
May 22 '15 at 13:21
1
1
It is hard to provide a solution without having the
Server.py
code and a description of how do the launched services fork (if they fork). However, generally speaking, this is a readiness protocol mismatch problem.– intelfx
May 21 '15 at 22:29
It is hard to provide a solution without having the
Server.py
code and a description of how do the launched services fork (if they fork). However, generally speaking, this is a readiness protocol mismatch problem.– intelfx
May 21 '15 at 22:29
BTW, the
ExecStop=
is not needed. systemd's default action on stop is to kill processes. You may want to take a look at the documentation for KillMode=
directive.– intelfx
May 21 '15 at 22:31
BTW, the
ExecStop=
is not needed. systemd's default action on stop is to kill processes. You may want to take a look at the documentation for KillMode=
directive.– intelfx
May 21 '15 at 22:31
1
1
And, finally... If there is no suitable readiness protocol (one of
simple
or forking
, actually), the last resort would be Type=oneshot
, RemainAfterExit=yes
and KillMode=control-group
.– intelfx
May 21 '15 at 22:33
And, finally... If there is no suitable readiness protocol (one of
simple
or forking
, actually), the last resort would be Type=oneshot
, RemainAfterExit=yes
and KillMode=control-group
.– intelfx
May 21 '15 at 22:33
@intelfx Essentially the Python script launches a server in a screen using Subprocess.call. It's more complicated than that because the script receives commands from elsewhere telling it which screens to start and which to not. Which screens are available is also dynamic hence why they cant be systemd services in their own right. Ideally I dont want systemd to treat these screens as part of the service at all, but currently they get dumped in the same process group and die with the master if its restarted.
– Bottswana
May 22 '15 at 13:21
@intelfx Essentially the Python script launches a server in a screen using Subprocess.call. It's more complicated than that because the script receives commands from elsewhere telling it which screens to start and which to not. Which screens are available is also dynamic hence why they cant be systemd services in their own right. Ideally I dont want systemd to treat these screens as part of the service at all, but currently they get dumped in the same process group and die with the master if its restarted.
– Bottswana
May 22 '15 at 13:21
add a comment |
3 Answers
3
active
oldest
votes
I managed to fix this simply by setting KillMode to process instead of control-group (default). Thanks all
add a comment |
I have a Python script that forks when it launches, and is responsible for starting a bunch of other processes.
Which indicates that you are doing it wrongly. More in this in a moment.
when the script exits the child processes are orphaned and continue to run.
This is not correct dæmon behaviour. If the "main" process — in this case the child that you have forked, since you have specified Type=forking
— exits, systemd considers the service to have deactivated and terminates any other still-running processes (in the control group) in order to tidy up.
Sometimes the conversion from System 5 rc
scripts to systemd is not straightforward, because the right way to do things under systemd is quite different. The right way to do (say) OpenVPN, or OpenStack, or OSSEC HIDS in systemd is not the same as one would do it with an rc
script. The fact that you have a script that is forking, then spawning a whole load of grandchildren processes, then exiting expecting those grandchildren to keep running indicates that you are perpetrating the same sort of horror as ossec-control
, albeit with two less levels of forking. If you find yourself writing a "master" script that checks "enable" flags and runs child processes for the "enabled" parts of your system, then you are making the same mistake as the horrendous ossec-control
.
No such home-grown mechanisms are necessary with systemd. It already is a service manager. Per https://unix.stackexchange.com/a/200365/5132, the right way to go about this in systemd is not to have one service that spawns some wacky and confused attempt to have "sub-services". It is to have each child process as a fully fledged systemd service in its own right. Then one enables and disables, and starts and stops, the various parts of the system using the normal systemd controls. As you can see in the OSSEC HIDS case, a simple template service unit covers almost all (one exception is at https://askubuntu.com/a/624871/43344) services, allowing one to do things such as systemctl enable ossec@agentlessd.service
to enable an optional agentlessd
service, without any need at all for the horrendous "master script" mechanism that was needed with System 5 rc
.
There are plenty of cases, not perhaps as extreme as OSSEC HIDS, where such rethinking is necessary. MTSes like exim and sendmail are two such. One might have had a single rc
script that spawns a queue runner, an SMTP Submission dæmon, and an SMTP Relay dæmon, with a bunch of ad hoc shell variables in a configuration file to control exactly which are run. But the right way to do this with systemd is to have three proper service units (two of which have associated socket units) and no ad hoc stuff at all, just the regular mechanisms of the service manager.
I appreciate the feedback on this. While I agree having subset services makes sense, it was done in Python for a reason which I can't go into. My only solution is to find a way to make this method work. Thanks, though. I'd love to do it properly.
– Bottswana
May 22 '15 at 13:15
The 'sub'services which the script launches are just servers running in gnu screen as a specific user. These servers change a lot, some are added, some are removed and this is controlled elsewhere, so they can't really be actual services in systemd, because that adds to much complexity, and cant be managed centrally. Also the same script is also used on non systemd servers.
– Bottswana
May 22 '15 at 13:24
add a comment |
In the parent process, you could just wait on the children instead of dying.
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f204922%2fsystemd-and-process-spawning%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
I managed to fix this simply by setting KillMode to process instead of control-group (default). Thanks all
add a comment |
I managed to fix this simply by setting KillMode to process instead of control-group (default). Thanks all
add a comment |
I managed to fix this simply by setting KillMode to process instead of control-group (default). Thanks all
I managed to fix this simply by setting KillMode to process instead of control-group (default). Thanks all
answered Jun 4 '15 at 15:36
BottswanaBottswana
131118
131118
add a comment |
add a comment |
I have a Python script that forks when it launches, and is responsible for starting a bunch of other processes.
Which indicates that you are doing it wrongly. More in this in a moment.
when the script exits the child processes are orphaned and continue to run.
This is not correct dæmon behaviour. If the "main" process — in this case the child that you have forked, since you have specified Type=forking
— exits, systemd considers the service to have deactivated and terminates any other still-running processes (in the control group) in order to tidy up.
Sometimes the conversion from System 5 rc
scripts to systemd is not straightforward, because the right way to do things under systemd is quite different. The right way to do (say) OpenVPN, or OpenStack, or OSSEC HIDS in systemd is not the same as one would do it with an rc
script. The fact that you have a script that is forking, then spawning a whole load of grandchildren processes, then exiting expecting those grandchildren to keep running indicates that you are perpetrating the same sort of horror as ossec-control
, albeit with two less levels of forking. If you find yourself writing a "master" script that checks "enable" flags and runs child processes for the "enabled" parts of your system, then you are making the same mistake as the horrendous ossec-control
.
No such home-grown mechanisms are necessary with systemd. It already is a service manager. Per https://unix.stackexchange.com/a/200365/5132, the right way to go about this in systemd is not to have one service that spawns some wacky and confused attempt to have "sub-services". It is to have each child process as a fully fledged systemd service in its own right. Then one enables and disables, and starts and stops, the various parts of the system using the normal systemd controls. As you can see in the OSSEC HIDS case, a simple template service unit covers almost all (one exception is at https://askubuntu.com/a/624871/43344) services, allowing one to do things such as systemctl enable ossec@agentlessd.service
to enable an optional agentlessd
service, without any need at all for the horrendous "master script" mechanism that was needed with System 5 rc
.
There are plenty of cases, not perhaps as extreme as OSSEC HIDS, where such rethinking is necessary. MTSes like exim and sendmail are two such. One might have had a single rc
script that spawns a queue runner, an SMTP Submission dæmon, and an SMTP Relay dæmon, with a bunch of ad hoc shell variables in a configuration file to control exactly which are run. But the right way to do this with systemd is to have three proper service units (two of which have associated socket units) and no ad hoc stuff at all, just the regular mechanisms of the service manager.
I appreciate the feedback on this. While I agree having subset services makes sense, it was done in Python for a reason which I can't go into. My only solution is to find a way to make this method work. Thanks, though. I'd love to do it properly.
– Bottswana
May 22 '15 at 13:15
The 'sub'services which the script launches are just servers running in gnu screen as a specific user. These servers change a lot, some are added, some are removed and this is controlled elsewhere, so they can't really be actual services in systemd, because that adds to much complexity, and cant be managed centrally. Also the same script is also used on non systemd servers.
– Bottswana
May 22 '15 at 13:24
add a comment |
I have a Python script that forks when it launches, and is responsible for starting a bunch of other processes.
Which indicates that you are doing it wrongly. More in this in a moment.
when the script exits the child processes are orphaned and continue to run.
This is not correct dæmon behaviour. If the "main" process — in this case the child that you have forked, since you have specified Type=forking
— exits, systemd considers the service to have deactivated and terminates any other still-running processes (in the control group) in order to tidy up.
Sometimes the conversion from System 5 rc
scripts to systemd is not straightforward, because the right way to do things under systemd is quite different. The right way to do (say) OpenVPN, or OpenStack, or OSSEC HIDS in systemd is not the same as one would do it with an rc
script. The fact that you have a script that is forking, then spawning a whole load of grandchildren processes, then exiting expecting those grandchildren to keep running indicates that you are perpetrating the same sort of horror as ossec-control
, albeit with two less levels of forking. If you find yourself writing a "master" script that checks "enable" flags and runs child processes for the "enabled" parts of your system, then you are making the same mistake as the horrendous ossec-control
.
No such home-grown mechanisms are necessary with systemd. It already is a service manager. Per https://unix.stackexchange.com/a/200365/5132, the right way to go about this in systemd is not to have one service that spawns some wacky and confused attempt to have "sub-services". It is to have each child process as a fully fledged systemd service in its own right. Then one enables and disables, and starts and stops, the various parts of the system using the normal systemd controls. As you can see in the OSSEC HIDS case, a simple template service unit covers almost all (one exception is at https://askubuntu.com/a/624871/43344) services, allowing one to do things such as systemctl enable ossec@agentlessd.service
to enable an optional agentlessd
service, without any need at all for the horrendous "master script" mechanism that was needed with System 5 rc
.
There are plenty of cases, not perhaps as extreme as OSSEC HIDS, where such rethinking is necessary. MTSes like exim and sendmail are two such. One might have had a single rc
script that spawns a queue runner, an SMTP Submission dæmon, and an SMTP Relay dæmon, with a bunch of ad hoc shell variables in a configuration file to control exactly which are run. But the right way to do this with systemd is to have three proper service units (two of which have associated socket units) and no ad hoc stuff at all, just the regular mechanisms of the service manager.
I appreciate the feedback on this. While I agree having subset services makes sense, it was done in Python for a reason which I can't go into. My only solution is to find a way to make this method work. Thanks, though. I'd love to do it properly.
– Bottswana
May 22 '15 at 13:15
The 'sub'services which the script launches are just servers running in gnu screen as a specific user. These servers change a lot, some are added, some are removed and this is controlled elsewhere, so they can't really be actual services in systemd, because that adds to much complexity, and cant be managed centrally. Also the same script is also used on non systemd servers.
– Bottswana
May 22 '15 at 13:24
add a comment |
I have a Python script that forks when it launches, and is responsible for starting a bunch of other processes.
Which indicates that you are doing it wrongly. More in this in a moment.
when the script exits the child processes are orphaned and continue to run.
This is not correct dæmon behaviour. If the "main" process — in this case the child that you have forked, since you have specified Type=forking
— exits, systemd considers the service to have deactivated and terminates any other still-running processes (in the control group) in order to tidy up.
Sometimes the conversion from System 5 rc
scripts to systemd is not straightforward, because the right way to do things under systemd is quite different. The right way to do (say) OpenVPN, or OpenStack, or OSSEC HIDS in systemd is not the same as one would do it with an rc
script. The fact that you have a script that is forking, then spawning a whole load of grandchildren processes, then exiting expecting those grandchildren to keep running indicates that you are perpetrating the same sort of horror as ossec-control
, albeit with two less levels of forking. If you find yourself writing a "master" script that checks "enable" flags and runs child processes for the "enabled" parts of your system, then you are making the same mistake as the horrendous ossec-control
.
No such home-grown mechanisms are necessary with systemd. It already is a service manager. Per https://unix.stackexchange.com/a/200365/5132, the right way to go about this in systemd is not to have one service that spawns some wacky and confused attempt to have "sub-services". It is to have each child process as a fully fledged systemd service in its own right. Then one enables and disables, and starts and stops, the various parts of the system using the normal systemd controls. As you can see in the OSSEC HIDS case, a simple template service unit covers almost all (one exception is at https://askubuntu.com/a/624871/43344) services, allowing one to do things such as systemctl enable ossec@agentlessd.service
to enable an optional agentlessd
service, without any need at all for the horrendous "master script" mechanism that was needed with System 5 rc
.
There are plenty of cases, not perhaps as extreme as OSSEC HIDS, where such rethinking is necessary. MTSes like exim and sendmail are two such. One might have had a single rc
script that spawns a queue runner, an SMTP Submission dæmon, and an SMTP Relay dæmon, with a bunch of ad hoc shell variables in a configuration file to control exactly which are run. But the right way to do this with systemd is to have three proper service units (two of which have associated socket units) and no ad hoc stuff at all, just the regular mechanisms of the service manager.
I have a Python script that forks when it launches, and is responsible for starting a bunch of other processes.
Which indicates that you are doing it wrongly. More in this in a moment.
when the script exits the child processes are orphaned and continue to run.
This is not correct dæmon behaviour. If the "main" process — in this case the child that you have forked, since you have specified Type=forking
— exits, systemd considers the service to have deactivated and terminates any other still-running processes (in the control group) in order to tidy up.
Sometimes the conversion from System 5 rc
scripts to systemd is not straightforward, because the right way to do things under systemd is quite different. The right way to do (say) OpenVPN, or OpenStack, or OSSEC HIDS in systemd is not the same as one would do it with an rc
script. The fact that you have a script that is forking, then spawning a whole load of grandchildren processes, then exiting expecting those grandchildren to keep running indicates that you are perpetrating the same sort of horror as ossec-control
, albeit with two less levels of forking. If you find yourself writing a "master" script that checks "enable" flags and runs child processes for the "enabled" parts of your system, then you are making the same mistake as the horrendous ossec-control
.
No such home-grown mechanisms are necessary with systemd. It already is a service manager. Per https://unix.stackexchange.com/a/200365/5132, the right way to go about this in systemd is not to have one service that spawns some wacky and confused attempt to have "sub-services". It is to have each child process as a fully fledged systemd service in its own right. Then one enables and disables, and starts and stops, the various parts of the system using the normal systemd controls. As you can see in the OSSEC HIDS case, a simple template service unit covers almost all (one exception is at https://askubuntu.com/a/624871/43344) services, allowing one to do things such as systemctl enable ossec@agentlessd.service
to enable an optional agentlessd
service, without any need at all for the horrendous "master script" mechanism that was needed with System 5 rc
.
There are plenty of cases, not perhaps as extreme as OSSEC HIDS, where such rethinking is necessary. MTSes like exim and sendmail are two such. One might have had a single rc
script that spawns a queue runner, an SMTP Submission dæmon, and an SMTP Relay dæmon, with a bunch of ad hoc shell variables in a configuration file to control exactly which are run. But the right way to do this with systemd is to have three proper service units (two of which have associated socket units) and no ad hoc stuff at all, just the regular mechanisms of the service manager.
edited Apr 13 '17 at 12:37
Community♦
1
1
answered May 21 '15 at 22:29
JdeBPJdeBP
34.5k469163
34.5k469163
I appreciate the feedback on this. While I agree having subset services makes sense, it was done in Python for a reason which I can't go into. My only solution is to find a way to make this method work. Thanks, though. I'd love to do it properly.
– Bottswana
May 22 '15 at 13:15
The 'sub'services which the script launches are just servers running in gnu screen as a specific user. These servers change a lot, some are added, some are removed and this is controlled elsewhere, so they can't really be actual services in systemd, because that adds to much complexity, and cant be managed centrally. Also the same script is also used on non systemd servers.
– Bottswana
May 22 '15 at 13:24
add a comment |
I appreciate the feedback on this. While I agree having subset services makes sense, it was done in Python for a reason which I can't go into. My only solution is to find a way to make this method work. Thanks, though. I'd love to do it properly.
– Bottswana
May 22 '15 at 13:15
The 'sub'services which the script launches are just servers running in gnu screen as a specific user. These servers change a lot, some are added, some are removed and this is controlled elsewhere, so they can't really be actual services in systemd, because that adds to much complexity, and cant be managed centrally. Also the same script is also used on non systemd servers.
– Bottswana
May 22 '15 at 13:24
I appreciate the feedback on this. While I agree having subset services makes sense, it was done in Python for a reason which I can't go into. My only solution is to find a way to make this method work. Thanks, though. I'd love to do it properly.
– Bottswana
May 22 '15 at 13:15
I appreciate the feedback on this. While I agree having subset services makes sense, it was done in Python for a reason which I can't go into. My only solution is to find a way to make this method work. Thanks, though. I'd love to do it properly.
– Bottswana
May 22 '15 at 13:15
The 'sub'services which the script launches are just servers running in gnu screen as a specific user. These servers change a lot, some are added, some are removed and this is controlled elsewhere, so they can't really be actual services in systemd, because that adds to much complexity, and cant be managed centrally. Also the same script is also used on non systemd servers.
– Bottswana
May 22 '15 at 13:24
The 'sub'services which the script launches are just servers running in gnu screen as a specific user. These servers change a lot, some are added, some are removed and this is controlled elsewhere, so they can't really be actual services in systemd, because that adds to much complexity, and cant be managed centrally. Also the same script is also used on non systemd servers.
– Bottswana
May 22 '15 at 13:24
add a comment |
In the parent process, you could just wait on the children instead of dying.
add a comment |
In the parent process, you could just wait on the children instead of dying.
add a comment |
In the parent process, you could just wait on the children instead of dying.
In the parent process, you could just wait on the children instead of dying.
answered 10 mins ago
Craig HicksCraig Hicks
24419
24419
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f204922%2fsystemd-and-process-spawning%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
It is hard to provide a solution without having the
Server.py
code and a description of how do the launched services fork (if they fork). However, generally speaking, this is a readiness protocol mismatch problem.– intelfx
May 21 '15 at 22:29
BTW, the
ExecStop=
is not needed. systemd's default action on stop is to kill processes. You may want to take a look at the documentation forKillMode=
directive.– intelfx
May 21 '15 at 22:31
1
And, finally... If there is no suitable readiness protocol (one of
simple
orforking
, actually), the last resort would beType=oneshot
,RemainAfterExit=yes
andKillMode=control-group
.– intelfx
May 21 '15 at 22:33
@intelfx Essentially the Python script launches a server in a screen using Subprocess.call. It's more complicated than that because the script receives commands from elsewhere telling it which screens to start and which to not. Which screens are available is also dynamic hence why they cant be systemd services in their own right. Ideally I dont want systemd to treat these screens as part of the service at all, but currently they get dumped in the same process group and die with the master if its restarted.
– Bottswana
May 22 '15 at 13:21