Configuration

Introduction

Invoke offers a fully fleshed out configuration mechanism allowing you to configure both its core behavior (and that of your tasks) via a hierarchy of configuration files, environment variables, task namespaces and CLI flags.

The end result of configuration seeking, loading, parsing & merging, is a Config object, which behaves like a (nested) Python dictionary. Invoke references this object when it runs (determining the default behavior of methods like Context.run) and exposes it to users’ tasks as Context.config and as shorthand attribute access on the Context itself.

The configuration hierarchy

In brief, the order in which configuration values are loaded (and overridden - each new level overrides the one above it) is as follows:

1. Internal default values for behaviors which are controllable via configuration. See Default configuration values for details.

2. Collection-driven configurations defined in tasks modules via Collection.configure. (See Collection-based configuration below for details.)

   • Sub-collections’ configurations get merged into the top level collection and the final result forms the basis of the overall configuration setup.
   • Since the root collection is loaded at runtime, configuration settings modifying the load process itself obviously won’t take effect if defined at this level.

3. System-level configuration file stored in /etc/, such as /etc/invoke.yaml. (See Configuration files for details on this and the other config-file entries.)

4. User-level configuration file found in the running user’s home directory, e.g. ~/.invoke.yaml.

5. Project-level configuration file living next to your top level tasks.py. For example, if your run of Invoke loads /home/user/myproject/tasks.py (see our docs on the load process), this might be /home/user/myproject/invoke.yaml.

6. Environment variables found in the invoking shell environment.

   • These aren’t as strongly hierarchical as the rest, nor is the shell environment namespace owned wholly by Invoke, so we must rely on slightly verbose prefixing instead - see Environment variables for details.

7. Runtime configuration file whose path is given to -f, e.g. inv -f /random/path/to/config_file.yaml.

8. Command-line flags for certain core settings, such as -e.

Default configuration values

Below is a list of all the configuration values and/or section Invoke itself uses to control behaviors such as Context.run‘s echo and pty flags, task deduplication, and so forth.

For convenience, we refer to nested setting names with a dotted syntax, so e.g. foo.bar refers to what would be (in a Python config context) {'foo': {'bar': <value here>}}. Typically, these can be read or set on Config and Context objects using attribute syntax, which looks nearly identical: ctx.foo.bar.

• The tasks config tree holds settings relating to task execution.

  • tasks.dedupe controls Task deduplication and defaults to True. It can also be overridden at runtime via --no-dedupe.

• The run tree controls the behavior of Runner.run. Each member of this tree (such as e.g. run.echo or run.pty) maps directly to a Runner.run keyword argument of the same name; see that method’s docstring for details on what these settings do & what their default values are.

• A top level config setting, debug, controls whether debug-level output is logged; it defaults to False.

  debug can be toggled via the -d CLI flag, which enables debugging after CLI parsing runs. It can also be toggled via the INVOKE_DEBUG environment variable which - unlike regular env vars - is honored from the start of execution and is thus useful for troubleshooting parsing and/or config loading.

Configuration files

Loading

For each configuration file location mentioned in the previous section, we search for files ending in .yaml, .json or .py (in that order!), load the first one we find, and ignore any others that might exist.

For example, if Invoke is run on a system containing both /etc/invoke.yaml and /etc/invoke.json, only the YAML file will be loaded. This helps keep things simple, both conceptually and in the implementation.

Format

Invoke’s configuration allows arbitrary nesting, and thus so do our config file formats. All three of the below examples result in a configuration equivalent to {'debug': True, 'run': {'echo': True}}:

• YAML

  debug: true
  run:
      echo: true
  

• JSON

  {
      "debug": true,
      "run": {
          "echo": true
      }
  }
  

• Python:

  debug = True
  run = {
      "echo": True
  }
  

For further details, see these languages’ own documentation.

Environment variables

Environment variables are a bit different from other configuration-setting methods, since they don’t provide a clean way to nest configuration keys, and are also implicitly shared amongst the entire system’s installed application base.

In addition, due to implementation concerns, env vars must be pre-determined by the levels below them in the config hierarchy (in other words - env vars may only be used to override existing config values). If you need Invoke to understand a FOOBAR environment variable, you must first declare a foobar setting in a configuration file or in your task collections.

Basic rules

To mitigate the shell namespace problem, we simply prefix all our env vars with INVOKE_.

Nesting is performed via underscore separation, so a setting that looks like e.g. {'run': {'echo': True}} at the Python level becomes INVOKE_RUN_ECHO=1 in a typical shell. See Nesting vs underscored names below for more on this.

Type casting

Since env vars can only be used to override existing settings, the previous value of a given setting is used as a guide in casting the strings we get back from the shell:

• If the current value is a string or Unicode object, it is replaced with the value from the environment, with no casting whatsoever;

  • Depending on interpreter and environment, this means that a setting defaulting to a non-Unicode string type (eg a str on Python 2) may end up replaced with a Unicode string, or vice versa. This is intentional as it prevents users from accidentally limiting themselves to non-Unicode strings.

• If the current value is None, it too is replaced with the string from the environment;

• Booleans are set as follows: 0 and the empty value/string (e.g. SETTING=, or unset SETTING, or etc) evaluate to False, and any other value evaluates to True.

• Lists and tuples are currently unsupported and will raise an exception;

  • In the future we may implement convenience transformations, such as splitting on commas to form a list; however since users can always perform such operations themselves, it may not be a high priority.

• All other types - integers, longs, floats, etc - are simply used as constructors for the incoming value.

  • For example, a foobar setting whose default value is the integer 1 will run all env var inputs through int, and thus FOOBAR=5 will result in the Python value 5, not "5".

Nesting vs underscored names

Since environment variable keys are single strings, we must use some form of string parsing to allow access to nested configuration settings. As mentioned above, in basic use cases this just means using an underscore character: {'run': {'echo': True}} becomes INVOKE_RUN_ECHO=1.

However, ambiguity is introduced when the settings names themselves contain underscores: is INVOKE_FOO_BAR=baz equivalent to {'foo': {'bar': 'baz'}}, or to {'foo_bar': 'baz'}? Thankfully, because env vars can only be used to modify settings declared at the Python level or in config files, we simply look at the current state of the config to determine the answer.

There is still a corner case where both possible interpretations exist as valid config paths (e.g. {'foo': {'bar': 'default'}, 'foo_bar': 'otherdefault'}). In this situation, we honor the Zen of Python and refuse to guess; an error is raised instead counseling users to modify their configuration layout or avoid using env vars for the setting in question.

Collection-based configuration

Collection objects may contain a config mapping, set via Collection.configure, and (as per the hierarchy) this typically forms the lowest level of configuration in the system.

When collections are nested, configuration is merged ‘downwards’ by default: when conflicts arise, outer namespaces closer to the root will win, versus inner ones closer to the task being invoked.

Note

‘Inner’ tasks here are specifically those on the path from the root to the one housing the invoked task. ‘Sibling’ subcollections are ignored.

A quick example of what this means:

from invoke import Collection, ctask as task

# This task & collection could just as easily come from another module
# somewhere.
@task
def mytask(ctx):
    print(ctx['conflicted'])
inner = Collection('inner', mytask)
inner.configure({'conflicted': 'default value'})

# Our project's root namespace.
ns = Collection(inner)
ns.configure({'conflicted': 'override value'})

The result of calling inner.mytask:

$ inv inner.mytask
override value

Example

Setup

As an example, we’ll start out with some semi-realistic, non-contextualized tasks that hardcode their values, and build up to using the various configuration mechanisms. A small module for building Sphinx docs might start out like this:

from invoke import task, run

@task
def clean():
    run("rm -rf docs/_build")

@task
def build():
    run("sphinx-build docs docs/_build")

Then maybe you refactor the build target:

target = "docs/_build"

@task
def clean():
    run("rm -rf {0}".format(target))

@task
def build():
    run("sphinx-build docs {0}".format(target))

We can also allow runtime parameterization:

default_target = "docs/_build"

@task
def clean(target=default_target):
    run("rm -rf {0}".format(target))

@task
def build(target=default_target):
    run("sphinx-build docs {0}".format(target))

This task module works for a single set of users, but what if we want to allow reuse? Somebody may want to use this module with a different default target. You can kludge it using non-contextualized tasks, but using a context to configure these settings is usually the better solution [1].

Switching to contexts

The configuration setting and getting APIs make it easy to move otherwise ‘hardcoded’ default values into a config structure which downstream users are free to redefine. Let’s apply this to our example. First we switch to using contextualized tasks and add an explicit namespace object:

from invoke import Collection, ctask as task

default_target = "docs/_build"

@task
def clean(ctx, target=default_target):
    ctx.run("rm -rf {0}".format(target))

@task
def build(ctx, target=default_target):
    ctx.run("sphinx-build docs {0}".format(target))

ns = Collection(clean, build)

Then we can move the default build target value into the collection’s default configuration, and refer to it via the context. At this point we also change our kwarg default value to be None so we can determine whether or not a runtime value was given. The result:

@task
def clean(ctx, target=None):
    ctx.run("rm -rf {0}".format(target or ctx.sphinx.target))

@task
def build(ctx, target=None):
    ctx.run("sphinx-build docs {0}".format(target or ctx.sphinx.target))

ns = Collection(clean, build)
ns.configure({'sphinx': {'target': "docs/_build"}})

The result isn’t significantly more complex than what we began with, and as we’ll see next, it’s now trivial for users to override your defaults in various ways.

Configuration overriding

The lowest-level override is, of course, just modifying the local Collection tree into which a distributed module has been imported. E.g. if the above module is distributed as myproject.docs, someone can define a tasks.py that does this:

from invoke import Collection, ctask as task
from myproject import docs

@task
def mylocaltask(ctx):
    # Some local stuff goes here
    pass

# Add 'docs' to our local root namespace, plus our own task
ns = Collection(mylocaltask, docs)

And then they can simply add this to the bottom:

ns.configure({'sphinx': {'target': "built_docs"}}) # Our docs live here

Now we have a docs sub-namespace whose build target defaults to built_docs instead of docs/_build.

If you prefer configuration files over in-Python tweaking of your namespace tree, that works just as well; instead of adding the line above to the previous snippet, instead drop this into a file next to tasks.py named invoke.yaml:

sphinx:
    target: built_docs

For this example, that sort of local-to-project conf file makes the most sense, but don’t forget that the config hierarchy offers additional configuration methods which may be suitable depending on your needs.

Footnotes

[1]	Copying and modifying the file breaks code reuse; overriding the module-level default_path variable won’t play well with concurrency; wrapping the tasks with different default arguments works but is fragile and adds boilerplate.