How do I call a command using Python, capturing stderr and stdout, without waiting for stderr / stdout to be closed?

advertisements

Right now I have some code that uses Popen.communicate() from subprocess (setting stdin=PIPE and stderr=PIPE) to run a command and capture both stderr and stdout.

The problem is that communicate() not only waits for the command to exit, it waits for stdout and stderr to be closed. The command I'm running spawns a child process which keeps stderr open, so even though the command is finished running (and shows as "defunct" in ps) communicate() is still hung.

I want to only wait for the command to finish without waiting on stderr/stdout. But I still want to capture any stderr/stdout output given while the command was running. The documentation for wait() is accompanied by a red box with a disclaimer:

This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

Obviously, I also want to avoid deadlocks.

What is the right way to accomplish this task?


"shows as "defunct" in ps" implies that you might be on a system where select or fcntl would work i.e., you can read stdout/stderr without blocking easily.

Example: A starts B (cmd, the child), B starts C (grandchild), A reads output until B exits or EOF:

#!/usr/bin/env python
import os
from select import select
from subprocess import Popen, PIPE

p = Popen(cmd, stdout=PIPE, stderr=PIPE, bufsize=0)
read_set = [p.stdout, p.stderr]
pipename = {p.stdout: "stdout", p.stderr: "stderr"}
timeout = 0.5 # ugly but it works
while read_set and p.poll() is None: # while subprocess is running or until EOF
    for pipe in select(read_set, [], [], timeout)[0]:
        data = os.read(pipe.fileno(), 1<<30)
        if data:
            print("got from %s: %r" % (pipename[pipe], data))
        else: # EOF
            pipe.close()
            read_set.remove(pipe)
print("exit code %s" % (p.wait(),))

# child exited, wait for grandchild to print
for pipe in read_set:
    print("read the rest of %s: %r" % (pipename[pipe], pipe.read()))
    pipe.close()

where cmd:

import sys
from textwrap import dedent

cmd = [sys.executable, '-u', '-c', dedent("""
    # inception style
    import os
    import sys
    from subprocess import Popen
    from textwrap import dedent

    Popen([sys.executable, '-u', '-c', dedent('''
        import os
        import sys
        import time

        time.sleep(60)
        print("grandchild %d done" % os.getpid())
        sys.stderr.write("grandchild stderr")
        sys.exit(20)
    ''')]) # stdout/stderr are not redirected

    print('child %d done' % os.getpid())
    sys.stderr.write('child stderr')
    sys.exit(19)
""")]