runners

class invoke.runners.Runner(context)

Partially-abstract core command-running API.

This class is not usable by itself and must be subclassed, implementing a number of methods such as start, wait and returncode. For a subclass implementation example, see the source code for Local.

New in version 1.0.

__init__(context)

Create a new runner with a handle on some Context.

Parameters:context –

a Context instance, used to transmit default options and provide access to other contextualized information (e.g. a remote-oriented Runner might want a Context subclass holding info about hostnames and ports.)

Note

The Context given to Runner instances must contain default config values for the Runner class in question. At a minimum, this means values for each of the default Runner.run keyword arguments such as echo and warn.
Raises:exceptions.ValueError – if not all expected default values are found in context.
context = None

The Context given to the same-named argument of __init__.

program_finished = None

A threading.Event signaling program completion.

Typically set after wait returns. Some IO mechanisms rely on this to know when to exit an infinite read loop.

read_chunk_size = 1000

How many bytes (at maximum) to read per iteration of stream reads.

input_sleep = 0.01

How many seconds to sleep on each iteration of the stdin read loop and other otherwise-fast loops.

warned_about_pty_fallback = None

Whether pty fallback warning has been emitted.

watchers = None

A list of StreamWatcher instances for use by respond. Is filled in at runtime by run.

run(command, **kwargs)

Execute command, returning an instance of Result.

Note

All kwargs will default to the values found in this instance’s context attribute, specifically in its configuration’s run subtree (e.g. run.echo provides the default value for the echo keyword, etc). The base default values are described in the parameter list below.

Parameters:
  • command (str) – The shell command to execute.
  • shell (str) – Which shell binary to use. Default: /bin/bash (on Unix; COMSPEC or cmd.exe on Windows.)
  • warn (bool) –

    Whether to warn and continue, instead of raising UnexpectedExit, when the executed command exits with a nonzero status. Default: False.

    Note

    This setting has no effect on exceptions, which will still be raised, typically bundled in ThreadException objects if they were raised by the IO worker threads.

    Similarly, WatcherError exceptions raised by StreamWatcher instances will also ignore this setting, and will usually be bundled inside Failure objects (in order to preserve the execution context).

  • hide –

    Allows the caller to disable run’s default behavior of copying the subprocess’ stdout and stderr to the controlling terminal. Specify hide='out' (or 'stdout') to hide only the stdout stream, hide='err' (or 'stderr') to hide only stderr, or hide='both' (or True) to hide both streams.

    The default value is None, meaning to print everything; False will also disable hiding.

    Note

    Stdout and stderr are always captured and stored in the Result object, regardless of hide’s value.

    Note

    hide=True will also override echo=True if both are given (either as kwargs or via config/CLI).

  • pty (bool) –

    By default, run connects directly to the invoked process and reads its stdout/stderr streams. Some programs will buffer (or even behave) differently in this situation compared to using an actual terminal or pseudoterminal (pty). To use a pty instead of the default behavior, specify pty=True.

    Warning

    Due to their nature, ptys have a single output stream, so the ability to tell stdout apart from stderr is not possible when pty=True. As such, all output will appear on out_stream (see below) and be captured into the stdout result attribute. err_stream and stderr will always be empty when pty=True.

  • fallback (bool) – Controls auto-fallback behavior re: problems offering a pty when pty=True. Whether this has any effect depends on the specific Runner subclass being invoked. Default: True.
  • echo (bool) –

    Controls whether run prints the command string to local stdout prior to executing it. Default: False.

    Note

    hide=True will override echo=True if both are given.

  • env (dict) –

    By default, subprocesses recieve a copy of Invoke’s own environment (i.e. os.environ). Supply a dict here to update that child environment.

    For example, run('command', env={'PYTHONPATH': '/some/virtual/env/maybe'}) would modify the PYTHONPATH env var, with the rest of the child’s env looking identical to the parent.

    See also

    replace_env for changing ‘update’ to ‘replace’.

  • replace_env (bool) – When True, causes the subprocess to receive the dictionary given to env as its entire shell environment, instead of updating a copy of os.environ (which is the default behavior). Default: False.
  • encoding (str) – Override auto-detection of which encoding the subprocess is using for its stdout/stderr streams (which defaults to the return value of default_encoding).
  • out_stream – A file-like stream object to which the subprocess’ standard output should be written. If None (the default), sys.stdout will be used.
  • err_stream – Same as out_stream, except for standard error, and defaulting to sys.stderr.
  • in_stream –

    A file-like stream object to used as the subprocess’ standard input. If None (the default), sys.stdin will be used.

    If False, will disable stdin mirroring entirely (though other functionality which writes to the subprocess’ stdin, such as autoresponding, will still function.) Disabling stdin mirroring can help when sys.stdin is a misbehaving non-stream object, such as under test harnesses or headless command runners.

  • watchers –

    A list of StreamWatcher instances which will be used to scan the program’s stdout or stderr and may write into its stdin (typically str or bytes objects depending on Python version) in response to patterns or other heuristics.

    See Automatically responding to program output for details on this functionality.

    Default: [].

  • echo_stdin (bool) –

    Whether to write data from in_stream back to out_stream.

    In other words, in normal interactive usage, this parameter controls whether Invoke mirrors what you type back to your terminal.

    By default (when None), this behavior is triggered by the following:

    • Not using a pty to run the subcommand (i.e. pty=False), as ptys natively echo stdin to stdout on their own;
    • And when the controlling terminal of Invoke itself (as per in_stream) appears to be a valid terminal device or TTY. (Specifically, when isatty yields a True result when given in_stream.)

      Note

      This property tends to be False when piping another program’s output into an Invoke session, or when running Invoke within another program (e.g. running Invoke from itself).

    If both of those properties are true, echoing will occur; if either is false, no echoing will be performed.

    When not None, this parameter will override that auto-detection and force, or disable, echoing.
Returns:

Result, or a subclass thereof.
Raises:

UnexpectedExit, if the command exited nonzero and warn was False.
Raises:

Failure, if the command didn’t even exit cleanly, e.g. if a StreamWatcher raised WatcherError.
Raises:

ThreadException (if the background I/O threads encountered exceptions other than WatcherError).

New in version 1.0.

generate_result(**kwargs)

Create & return a suitable Result instance from the given kwargs.

Subclasses may wish to override this in order to manipulate things or generate a Result subclass (e.g. ones containing additional metadata besides the default).

New in version 1.0.

read_proc_output(reader)

Iteratively read & decode bytes from a subprocess’ out/err stream.

Parameters:reader –

A literal reader function/partial, wrapping the actual stream object in question, which takes a number of bytes to read, and returns that many bytes (or None).

reader should be a reference to either read_proc_stdout or read_proc_stderr, which perform the actual, platform/library specific read calls.
Returns:A generator yielding Unicode strings (unicode on Python 2; str on Python 3).

Specifically, each resulting string is the result of decoding read_chunk_size bytes read from the subprocess’ out/err stream.

New in version 1.0.

write_our_output(stream, string)

Write string to stream.

Also calls .flush() on stream to ensure that real terminal streams don’t buffer.

Parameters:
  • stream – A file-like stream object, mapping to the out_stream or err_stream parameters of run.
  • string – A Unicode string object.
Returns:

None.

New in version 1.0.

handle_stdout(buffer_, hide, output)

Read process’ stdout, storing into a buffer & printing/parsing.

Intended for use as a thread target. Only terminates when all stdout from the subprocess has been read.

Parameters:
  • buffer – The capture buffer shared with the main thread.
  • hide (bool) – Whether or not to replay data into output.
  • output – Output stream (file-like object) to write data into when not hiding.
Returns:

None.

New in version 1.0.

handle_stderr(buffer_, hide, output)

Read process’ stderr, storing into a buffer & printing/parsing.

Identical to handle_stdout except for the stream read from; see its docstring for API details.

New in version 1.0.

read_our_stdin(input_)

Read & decode bytes from a local stdin stream.

Parameters:input – Actual stream object to read from. Maps to in_stream in run, so will often be sys.stdin, but might be any stream-like object.
Returns:A Unicode string, the result of decoding the read bytes (this might be the empty string if the pipe has closed/reached EOF); or None if stdin wasn’t ready for reading yet.

New in version 1.0.

handle_stdin(input_, output, echo)

Read local stdin, copying into process’ stdin as necessary.

Intended for use as a thread target.

Note

Because real terminal stdin streams have no well-defined “end”, if such a stream is detected (based on existence of a callable .fileno()) this method will wait until program_finished is set, before terminating.

When the stream doesn’t appear to be from a terminal, the same semantics as handle_stdout are used - the stream is simply read() from until it returns an empty value.

Parameters:
  • input – Stream (file-like object) from which to read.
  • output – Stream (file-like object) to which echoing may occur.
  • echo (bool) – User override option for stdin-stdout echoing.
Returns:

None.

New in version 1.0.

should_echo_stdin(input_, output)

Determine whether data read from input_ should echo to output.

Used by handle_stdin; tests attributes of input_ and output.

Parameters:
  • input – Input stream (file-like object).
  • output – Output stream (file-like object).
Returns:

A bool.

New in version 1.0.

respond(buffer_)

Write to the program’s stdin in response to patterns in buffer_.

The patterns and responses are driven by the StreamWatcher instances from the watchers kwarg of run - see Automatically responding to program output for a conceptual overview.

Parameters:buffer – The capture buffer for this thread’s particular IO stream.
Returns:None.

New in version 1.0.

generate_env(env, replace_env)

Return a suitable environment dict based on user input & behavior.

Parameters:
  • env (dict) – Dict supplying overrides or full env, depending.
  • replace_env (bool) – Whether env updates, or is used in place of, the value of os.environ.
Returns:

A dictionary of shell environment vars.

New in version 1.0.

should_use_pty(pty, fallback)

Should execution attempt to use a pseudo-terminal?

Parameters:
  • pty (bool) – Whether the user explicitly asked for a pty.
  • fallback (bool) – Whether falling back to non-pty execution should be allowed, in situations where pty=True but a pty could not be allocated.

New in version 1.0.

has_dead_threads

Detect whether any IO threads appear to have terminated unexpectedly.

Used during process-completion waiting (in wait) to ensure we don’t deadlock our child process if our IO processing threads have errored/died.

Returns:True if any threads appear to have terminated with an exception, False otherwise.

New in version 1.0.

wait()

Block until the running command appears to have exited.

Returns:None.

New in version 1.0.

write_proc_stdin(data)

Write encoded data to the running process’ stdin.

Parameters:data – A Unicode string.
Returns:None.

New in version 1.0.

decode(data)

Decode some data bytes, returning Unicode.

New in version 1.0.

process_is_finished

Determine whether our subprocess has terminated.

Note

The implementation of this method should be nonblocking, as it is used within a query/poll loop.

Returns:True if the subprocess has finished running, False otherwise.

New in version 1.0.

start(command, shell, env)

Initiate execution of command (via shell, with env).

Typically this means use of a forked subprocess or requesting start of execution on a remote system.

In most cases, this method will also set subclass-specific member variables used in other methods such as wait and/or returncode.

New in version 1.0.

read_proc_stdout(num_bytes)

Read num_bytes from the running process’ stdout stream.

Parameters:num_bytes (int) – Number of bytes to read at maximum.
Returns:A string/bytes object.

New in version 1.0.

read_proc_stderr(num_bytes)

Read num_bytes from the running process’ stderr stream.

Parameters:num_bytes (int) – Number of bytes to read at maximum.
Returns:A string/bytes object.

New in version 1.0.

default_encoding()

Return a string naming the expected encoding of subprocess streams.

This return value should be suitable for use by encode/decode methods.

New in version 1.0.

send_interrupt(interrupt)

Submit an interrupt signal to the running subprocess.

In almost all implementations, the default behavior is what will be desired: submit  to the subprocess’ stdin pipe. However, we leave this as a public method in case this default needs to be augmented or replaced.

Parameters:interrupt – The locally-sourced KeyboardInterrupt causing the method call.
Returns:None.

New in version 1.0.

returncode()

Return the numeric return/exit code resulting from command execution.

Returns:int

New in version 1.0.

stop()

Perform final cleanup, if necessary.

This method is called within a finally clause inside the main run method. Depending on the subclass, it may be a no-op, or it may do things such as close network connections or open files.

Returns:None

New in version 1.0.

__weakref__

list of weak references to the object (if defined)

class invoke.runners.Local(context)

Execute a command on the local system in a subprocess.

Note

When Invoke itself is executed without a controlling terminal (e.g. when sys.stdin lacks a useful fileno), it’s not possible to present a handle on our PTY to local subprocesses. In such situations, Local will fallback to behaving as if pty=False (on the theory that degraded execution is better than none at all) as well as printing a warning to stderr.

To disable this behavior, say fallback=False.

New in version 1.0.

class invoke.runners.Result(stdout='', stderr='', encoding=None, command='', shell='', env=None, exited=0, pty=False, hide=())

A container for information about the result of a command execution.

All params are exposed as attributes of the same name and type.

Parameters:
  • stdout (str) – The subprocess’ standard output.
  • stderr (str) – Same as stdout but containing standard error (unless the process was invoked via a pty, in which case it will be empty; see Runner.run.)
  • encoding (str) – The string encoding used by the local shell environment.
  • command (str) – The command which was executed.
  • shell (str) – The shell binary used for execution.
  • env (dict) – The shell environment used for execution. (Default is the empty dict, {}, not None as displayed in the signature.)
  • exited (int) – An integer representing the subprocess’ exit/return code.
  • pty (bool) – A boolean describing whether the subprocess was invoked with a pty or not; see Runner.run.
  • hide (tuple) –

    A tuple of stream names (none, one or both of ('stdout', 'stderr')) which were hidden from the user when the generating command executed; this is a normalized value derived from the hide parameter of Runner.run.

    For example, run('command', hide='stdout') will yield a Result where result.hide == ('stdout',); hide=True or hide='both' results in result.hide == ('stdout', 'stderr'); and hide=False (the default) generates result.hide == () (the empty tuple.)

Note

Result objects’ truth evaluation is equivalent to their ok attribute’s value. Therefore, quick-and-dirty expressions like the following are possible:

if run("some shell command"):
    do_something()
else:
    handle_problem()

However, remember Zen of Python #2.

New in version 1.0.

return_code

An alias for .exited.

New in version 1.0.

ok

A boolean equivalent to exited == 0.

New in version 1.0.

__weakref__

list of weak references to the object (if defined)

failed

The inverse of ok.

I.e., True if the program exited with a nonzero return code, and False otherwise.

New in version 1.0.