Command Reference

This page serves as a reference for commands of the DevAssistant Yaml DSL. These commands are also callable from PingPong scripts. Every command consists of a command_type and command_input. After it gets executed, it sets the LAST_LRES and LAST_RES variables. These are also its return values, similar to Expressions logical result and result.

  • LAST_LRES is the logical result of the run - True/False if successful/unsuccessful
  • LAST_RES is the “return value” - e.g. a computed value

In the Yaml DSL, commands are called like this:

- command_type: command_input

This reference summarizes commands included in DevAssistant itself in the following format:

command_type - some optional info

  • Input: what should the input look like?
  • RES: what is LAST_RES set to after this command?
  • LRES: what is LAST_LRES set to after this command?
  • Example: example usage

Note: if a command explanation says that command “raises exception” under some circumstances, it means that a critical error has occured and assistant execution has to be interrupted immediately. See documentation for exceptions in run sections for details on how this reflects on command line and in GUI. In terms of the underlying Python source code, this means that exceptions.CommandException has been raised.

Missing something? Commands are your entry point for extending DevAssistant. If you’re missing some functionality in run sections, just write a command runner and either include it with your assistant or send us a pull request to get it merged in DevAssistant core.

Builtin Commands

There are three builtin commands that are inherent part of DevAssistant Yaml DSL:

  • variable assignment
  • condition
  • loop

All of these builtin commands utilize expressions in some way - these must follow rules in Expressions.

Variable Assignment

Assign result (and possibly also logical result) of Expressions to a variable(s).

$<var1>[, $<var2>] - if one variable is given, result of expression (command input) is assigned. If two variables are given, the first gets assigned logical result and the second result.

  • Input: an expression

  • RES: result of the expression

  • LRES: logical result of the expression

  • Example:

    - $foo: "bar"
    - $spam:
      - spam
      - spam
      - spam
    - $bar: $baz
    - $success, $list~: $(ls "$foo")
    

Condition

Conditional execution.

if <expression>, else - conditionally execute one or the other section (if can stand alone, of course)

  • Input: a subsection to run

  • RES: RES of last command in the subsection, if this clause is invoked. If not invoked, RES remains untouched.

  • LRES: LRES of last command in the subsection, if this clause is invoked. If not invoked, LRES remains untouched.

  • Example:

    - if defined $foo:
      - log_i: Foo is $foo!
    - else:
      - log_i: Foo is not defined!
    

Loop

A simple for loop.

for <var>[, <var>] [word_in,in] <expression> - loop over result of the expression. If word_in is used and <expression> is a string, it will be split on whitespaces and iterated over; with in, string will be split to single characters and iterated over. For iterations over lists and mappings, word_in and in behave the same. When iterating over mapping, two control variables may be provided to get both key and its value.

  • Input: a subsection to repeat in loop

  • RES: RES of last command of last iteration in the subsection. If there are no interations, RES is untouched.

  • LRES: LRES of last command of last iteration in the subsection. If there are no interations, RES remains untouched.

  • Example:

    - for $i word_in $(ls):
      - log_i: File: $i
    
    - $foo:
        1: one
        2: two
    - for $k, $v in $foo:
      - log_i: $k, $v
    

Ask Commands

User interaction commands, let you ask for password and various other input.

ask_confirm

  • Input: mapping containing prompt (short prompt for user) and message (a longer description of what the user should confirm)

  • RES: the confirmation (True or False)

  • LRES: same as RES

  • Example:

    - $confirmed~:
      - ask_confirm:
          message: "Do you think DevAssistant is great?"
          prompt: "Please select yes."
    

ask_input

  • Input: mapping containing prompt (short prompt for user)

  • RES: the string that was entered by the user

  • LRES: True if non-empty string was provided

  • Example:

    - $variable:
      - ask_input:
          prompt: "Your name"
    

ask_password

  • Input: mapping containing prompt (short prompt for user)

  • This command works the same way as ask_input, but the entered text is hidden (displayed as bullets)

  • RES: the password

  • LRES: True if non-empty password was provided

  • Example:

    - $passwd:
      - ask_password:
          prompt: "Please provide your password"
    

Command Line Commands

Run commands in subprocesses and receive their output.

cl, cl_[i,r] (these do the same, but appending i logs the command output on INFO level and appending r runs command as root; appending p makes DevAssistant pass subcommand error, e.g. execution continues normally even if subcommand return code is non-zero)

  • Input: a string, possibly containing variables and references to files

  • RES: stdout + stdin interleaved as they were returned by the executed process

  • LRES: always True, raises exception on non-zero return code

  • Example:

    - cl: mkdir ${name}
    - cl: cp *file ${name}/foo
    - cl_i: echo "Hey!"
    - cl_ir: echo "Echoing this as root"
    - cl_r: mkdir /var/lib/foo
    - $lres, $res:
      - cl_ip: cmd -this -will -log -in -realtime -and -save -lres -and -res -and -then -continue
    

If you need to set environment variables for multiple subsequent commands, consult Modifying Subprocess Environment Variables.

Note: when using r, it’s job of DevAssistant core to figure out what to use as authentication method. Consider this an implementation detail.

A note on changing current working directory: Due to the way Python interpreter works, DevAssistant has to specialcase “cd <dir>” command, since it needs to call a special Python method for changing current working directory of the running interpreter. Therefore you must always use “cd <dir>” as a single command (do not use “ls foo && cd foo”); also, using pushd/popd is not supported for now.

Modifying Subprocess Environment Variables

Globaly set/unset shell variables for subprocesses invoked by Command Line Commands and in Expressions.

env_set, env_unset

  • Input: a mapping of variables to set if using env_set, name (string) or names (list) of variables to unset if using env_unset

  • RES: mapping of newly set variable name(s) to their new values (for env_set) or unset variables to their last values (for env_unset)

  • LRES: always True

  • Example:

    - env_set:
        FOO: bar
    # If FOO is not in local DevAssistant context, DevAssistant does no substitution.
    #  This measn that the shell still gets "echo $FOO" to execute and prints "bar".
    - cl_i: echo $FOO
    - env_unset: FOO
    

Note: If some variables to be unset are not defined, their names are just ignored.

Dependencies Command

Install dependencies from given command input.

dependencies

  • Input: list of mappings, similar to Dependencies section, but without conditions and usage of sections from snippets etc.

  • RES: command input, but with expanded variables

  • LRES: always True if everything is ok, raises exception otherwise

  • Example:

    - if $foo:
      - $rpmdeps: [foo, bar]
    - else:
      - $rpmdeps: []
    
    - dependencies:
      - rpm: $rpmdeps
    

.devassistant Commands

Commands that operate with .devassistant file.

dda_c - creates a .devassistant file, should only be used in creator assistants

  • Input: directory where the file is supposed to be created

  • RES: always True, terminates DevAssistant if something goes wrong

  • LRES: always empty string

  • Example:

    - dda_c: ${path}/to/project
    

dda_r - reads an existing .devassistant file, should be used by tweak and preparer assistants.Sets some global variables accordingly, most importantly original_kwargs (arguments used when the project was created) - these are also made available with dda__ prefix (yes, that’s double underscore).

  • Input: directory where the file is supposed to be

  • RES: always empty string

  • LRES: True, raises exception if something goes wrong

  • Example:

    - dda_r: ${path}/to/project
    

dda_w - writes a mapping (dict in Python terms) to .devassistant

  • Input: list with directory with .devassistant file as a first item and the mapping to write as the second item. Variables in the mapping will be substituted, you have to use $$foo (two dollars instead of one) to get them as variables in .devassistant.

  • RES: always empty string

  • LRES: True, raises exception if something goes wrong

  • Example:

    - dda_w:
      - ${path}/to/project
      - run:
        - $$foo: $name # name will get substituted from current variable
        - log_i: $$foo
    

dda_dependencies - installs dependencies from .devassistant file, should be used by preparer assistants. Utilizes both dependencies of creator assistants that created this project plus dependencies from dependencies section, if present (this section is evaluated in the context of current assistant, not the creator).

  • Input: directory where the file is supposed to be

  • RES: always empty string

  • LRES: True, raises exception if something goes wrong

  • Example:

    - dda_dependencies: ${path}/to/project
    

dda_run - run run section from from .devassistant file, should be used by preparer assistants. This section is evaluated in the context of current assistant, not the creator.

  • Input: directory where the file is supposed to be

  • RES: always empty string

  • LRES: True, raises exception if something goes wrong

  • Example:

    - dda_run: ${path}/to/project
    

Github Command

Manipulate Github repositories. Two factor authentication is supported out of the box.

Github command (github) has many “subcommands”. Subcommands are part of the command input, see below.

  • Input: a string with a subcommand or a two item list, where the first item is a subcommand and the second item is a mapping that explicitly specifies parameters for the subcommand.

  • RES: if command succeeds, either a string with URL of manipulated repo or empty string is returned (depends on subcommand), else a string with problem description (it is already logged at WARNING level)

  • LRES: True if the Github operation succeeds, False otherwise

  • Example:

    - github: create_repo
    
    - github:
      - create_and_push
      - login: bkabrda
        reponame: devassistant
    
    - github: push
    
    - github: create_fork
    

Explanation of individual subcommands follows. Each subcommand takes defined arguments, whose default values are taken from global context. E.g. create_and_push takes an argument login. If it is not specified, assistant variable github is used.

create_repo
Creates a repo with given reponame (defaults to var name) for a user with given login (defaults to var github). Optionally accepts private argument to create repo as private (defaults to var github_private).
create_and_push
Same as create_repo, but it also adds a proper git remote to repository in current working dir and pushes to Github.
push
Just does git push -u origin master, no arguments needed.
create_fork
Creates a fork of repo at given repo_url (defaults ot var url) under user specified by login (defaults to var github).

Jinja2 Render Command

Render a Jinja2 template.

jinja_render, jinja_render_dir - render a single template or a directory containing more templates

  • Input: a mapping containing

    • template - a reference to file (or a directory if using jinja_render_dir) in files section
    • destination - directory where to place rendered template (or rendered directory)
    • data - a mapping of values used to render the template itself
    • overwrite (optional) - overwrite the file if it exists? (defaults to false)
    • output (optional) - specify a filename of the rendered template (see below for information on how the filename is costructed if not provided), not used with jinja_render_dir
  • RES: always success string

  • LRES: True, raises exception if something goes wrong

  • Example:

    - jinja_render:
        template: *somefile
        destination: ${dest}/foo
        overwrite: yes
        output: filename.foo
        data:
          foo: bar
          spam: spam
    
    - jinja_render_dir:
        template: *somedir
        destination: ${dest}/somedir
        data:
          foo: foo!
          spam: my_spam
    

The filename of the rendered template is created in this way (the first step is omitted with jinja_render_dir:

  • if output is provided, use that as the filename
  • else if name of the template endswith .tpl, strip .tpl and use it
  • else use the template name

For template syntax reference, see Jinja2 documentation.

Logging Commands

Log commands on various levels. Logging on ERROR or CRITICAL logs the message and then terminates the execution.

log_[d,i,w,e,c] (the letters stand for DEBUG, INFO, WARNING, ERROR, CRITICAL)

  • Input: a string, possibly containing variables and references to files

  • RES: the logged message (with expanded variables and files)

  • LRES: always True

  • Example:

    - log_i: Hello $name!
    - log_e: Yay, something has gone wrong, exiting.
    

Docker Commands

Control Docker from assistants.

docker_[build,cc,start,stop,attach,find_img,container_ip,container_name]

  • Input:

    • attach - list or string with names/hashes of container(s) (if string is provided, it’s split on whitespaces to get names/hashes)
    • build - mapping with arguments same as build method from docker_py_api, but path is required and fileobj is ignored
    • cc - mapping with arguments same as create_container method from docker_py_api, image is required
    • container_ip - string (container hash/name)
    • container_name - string (container hash)
    • find_img - string (a start of hash of image to find)
    • start - mapping with arguments same as start method from docker_py_api, container is required
    • stop - mapping with arguments same as stop method from docker_py_api, container is required
  • LRES and RES:

    • attach - LRES is True if all attached containers end with success, False otherwise; RES is always a string composed of outputs of all containers
    • build - True and hash of built image on success, otherwise raises exception
    • cc - True and hash of created container, otherwise raises exception
    • container_ip - True and IPv4 container address on success, otherwise raises exception
    • container_name - True and container name on success, otherwise raises exception
    • find_img - True and image hash on success if there is only one image that starts with provided input; False and string with space separated image hashes if there are none or more than one images
    • start - True and container hash on success, raises exception otherwise
    • stop - True and container hash on success, raises exception otherwise
  • Example (build an image, create container, start it and attach to output; stop it on DevAssistant shutdown):

    run:
    # build image
    - $image~:
      - docker_build:
          path: .
    # create container
    - $container~:
      - docker_cc:
          image: $image
    # start container
    - docker_start:
        container: $container
    - log_i~:
      - docker_container_ip: $container
    # register container to be shutdown on DevAssistant exit
    - atexit:
      - docker_stop:
          container: $container
          timeout: 3
    # attach to container output - this can be interrupted by Ctrl+C in terminal,
    #  but currently not in GUI, see https://github.com/devassistant/devassistant/issues/284
    - docker_attach: $container
    

Vagrant-Docker Commands

Control Docker using Vagrant from assistants.

vagrant_docker

  • Input: string with vagrant command to run, must start with one of up, halt, destroy, reload

  • RES: hashes/names of containers from Vagrantfile (not all of these were necessarily manipulated with, for example if you use halt, all container hashes are returned even if no containers were previously running)

  • LRES: True, raises exception if something goes wrong

  • Example:

    - vagrant_docker: halt
    - vagrant_docker: up
    

SCL Command

Run subsection in SCL environment.

scl [args to scl command] (note: you must use the scriptlet name - usually enable - because it might vary)

  • Input: a subsection

  • RES: RES of the last command in the given section

  • LRES: LRES of the last command in the given section

  • Example:

    - scl enable python33 postgresql92:
      - cl_i: python --version
      - cl_i: pgsql --version
    

Note: currently, this command can’t be nested, e.g. you can’t run scl enable in another scl enable.

Running Commands as Another User

Run subsection as a different user (how this command runner does this is considered an implementation detail). as <username> (note: use as root, to run subsection under superuser)

  • Input: a subsection

  • RES: output of the whole subsection

  • LRES: LRES of the last command in the given section

  • Example:

    - as root:
      - cl: ls /root
    - as joe:
      - log_i~: $(echo "this is run as joe")
    

Note: This command invokes DevAssistant under another user and passes the whole section to it. This means some behaviour differences from e.g. scl command, where each command is run in current assistant. Most importantly, RES of this command is RES of all commands from given subsection.

Using Another Section

Runs a section specified by command input at this place.

use This can be used to run:

  • another section of this assistant (e.g. use: self.run_foo)

  • section of superassistant (e.g. use: super.run) - searches all superassistants (parent of this, parent of the parent, etc.) and runs the first found section of given name

  • section from snippet (e.g. use: snippet_name.run_foo)

  • Input: a string with section name

  • RES: RES of the last command in the given section

  • LRES: LRES of the last command in the given section

  • Example:

    - use: self.run_foo
    - use: super.run
    - use: a_snippet.run_spam
    

This way, the whole context (all variables) are passed into the section run (by value, so they don’t get modified).

Another, more function-like usage is also available:

- use:
    sect: self.run_foo
      args:
        foo: $bar
        baz: $spam

Using this approach, the assistant/snippet and section name is taken from sect and only arguments listed in args are passed to the section (plus all “magic” variables, e.g. those starting and ending with double underscore).

Normalizing User Input

Replace “weird characters” (whitespace, colons, equals...) by underscores and unicode chars by their ascii counterparts.

  • Input: a string or a mapping containing keys what and ok_chars (ok_chars is a string containing characters that should not be normalized)

  • RES: a string with weird characters (e.g. brackets/braces, whitespace, etc) replaced by underscores

  • LRES: True

  • Example:

    - $dir~:
      - normalize: foo!@#$%^bar_ěšč
    - cl: mkdir $dir  # creates dir named foo______bar_esc
    - $dir~:
      - normalize:
          what: f-o.o-@#$baz
          ok_chars: "-."
    - cl: mkdir $dir  # creates dir named f-o.o-___baz
    

Setting up Project Directory

Creates a project directory (possibly with a directory containing it) and sets some global variables.

  • Input: a mapping of input options, see below

  • RES: path of project directory or a directory containing it, if create_topdir is False

  • LRES: True, raises exception if something goes wrong

  • Example:

    - $dir: foo/bar/baz
    - setup_project_dir:
        from: $dir
        create_topdir: normalized
    

Note: as a side effect, this command runner sets 3 global variables for you (their names can be altered by using arguments contdir_var, topdir_var and topdir_normalized_var):

  • contdir - the dir containing project directory (e.g. foo/bar in the example above)
  • topdir - the project directory (e.g. baz in the example above)
  • topdir_normalized - normalized name (by Normalizing User Input) of the project directory

Arguments:

  • from (required) - a string or a variable containing string with directory name (possibly a path)
  • create_topdir - one of True (default), False, normalized - if False, only creates the directory containing the project, not the project directory itself (e.g. it would create only foo/bar in example above, but not the baz directory); if True, it also creates the project directory itself; if normalized, it creates the project directory itself, but runs it’s name through Normalizing User Input first
  • normalize_ok_chars - string containing characters that should not be normalized, assuming that create_topdir: normalized is used
  • contdir_var, topdir_var, topdir_normalized_var - names to which the global variables should be assigned to - note: you have to use variable names without dollar sign here
  • accept_path - either True (default) or False - if False, this will terminate DevAssistant if a path is provided
  • on_existing - one of fail (default), pass - if fail, this will terminate DevAssistant if directory specified by from already exists; if pass, nothing will happen; note, that this is always considered pass, if create_topdir is False (in which case the assistant is in full control and responsible for checking everything itself)

Running Commands After Assistant Exits

Register commands to be run when assistant exits (this is not necessarily DevAssistant exit).

  • Input: section (list of commands to run)

  • RES: the passed list of commands (raw, unformatted)

  • LRES: True

  • Example:

    - $server: $(get server pid)
    - atexit:
      - cl: kill $server
      - log_i: Server gets killed even if the assistant failed at some point.'
    

Sections registered by atexit are run at the very end of assistant execution even after the post_run section. There are some differencies compared to post_run:

  • atexit command creates a “closure”, meaning the values of variables in time of the actual section invocation are the same as they were at the time the atexit command was used (meaning that even if you change variable values during the run section after running atexit, the values are preserved).
  • You can use multiple atexit command calls to register multiple sections. These are run in the order in which they were registered.
  • Even if some of the sections registered with atexit fail, the others are still invoked.

DevAssistant PingPong

Run DevAssistant PingPong scripts.

  • Input: a string to line on commandlie

  • RES: Result computed by the PingPong script

  • LRES: Logical result computed by the PingPong script

  • Example:

    - pingpong: python3 *file_from_files_section
    

Loading Custom Command Runners

Load DevAssistant command runner(s) from a file.

  • Input: string or mapping, see below

  • RES: List of classnames of loaded command runners

  • LRES: True if at least one command runner was loaded, False otherwise

  • Example:

    files:
      my_cr: &my_cr
        source: cr.py
    
    run:
    - load_cmd: *my_cr
    # assuming that there is a command runner that runs "mycommand" in the file,
    #  we can do this as of now until the end of this assistant
    #  this is equivalent of
    #  - load_cmd:
    #      from_file: *my_cr
    - mycommand: foo
    
    # load command runner from file provided in hierarchy of a different assistant
    # - make it prefixed to make sure it doesn't conflict with any core command runners
    # - load only BlahCommandRunner even if the file includes more runners
    - load_cmd:
        from_file: crt/someotherassistant/crs.py
        prefix: foo
        only: BlahCommandRunner
    - foo.blah: input  # runs ok
    - blah: input  # will fail, the command runner was registered with "foo" prefix
    

Note: since command runners loaded by load_cmd have higher priority than DevAssistant builtin command runners, you can use this to override the builtins. E.g. you can have a command runner that overrides log_i. If someone wants to use this command runner of yours but also keep the original one, he can provide a prefix, so that your logging command is only available as some_prefix.log_i.