Yrurtj

The setup: I have a python (3.6) script (call it "operator") that executes a bash script (call it "plunger") in standard subprocess ways, collecting and logging stdout from the child. The plunger script itself is simple and invokes other scripts/programs to do three moderately complicated things: a) shut down a bunch of daemon processes, b) do some housekeeping, c) fire up some new daemon processes then exit. There's nothing especially strange about the system itself: plain old CentOS running with standard rpms.

The problem: When the plunger script runs just parts a and b, everything works as expected - plunger (without c) runs to completion, and operator collects all the output and continues on doing the rest of its job. When I include step c, however, plunger runs correctly, operator collects all output (if I'm reading a little at a time), but then never notices that plunger has exited and never finishes reading the output, so control never gets passed back to the operator script.

Trivial Example:

return subprocess.check_output("plunger")  # doesn't complete with the real plunger script

Observations:

• running plunger in an interactive shell always works properly


• the plunger process does everything it is supposed to do AND exits


• running ps shows the plunger bash process as a zombie ("plunger ")


• using Popen and reading line by line indicates all expected lines are output and properly terminated with a newline


• using Popen and checking process status with poll() only emits None


• it is behaving like the child isn't ending or there are bytes remaining to be read, even when it has exited and the only PIPE stream is stdout... and reading from stdout blocks.

Conjecture:
The best guess I have is that the final step's spawning of new background (daemon) processes is somehow inheriting and keeping open the stdout stream so that even though the executed plunger script outputs and exits, some unknown process continues to hold onto it and thus doesn't allow the operator script to continue.

Questions:
Is my conjecture likely (or possible)? If not, what else might I look for? If so, how could I protect operator and/or plunger from downstream abuse of my streams?

Postscript:
My horrible hacky fugly workaround is for plunger to echo a distinctive line after it has done its job, and when operator sees it, kill the plunger process. I feel dirty just typing that.