Command Hooks
Hooks are pre- or -post functions that are attached to a command pattern, which is a regular expression (regex). Anytime Watiba encounters a command
that matches the pattern for the hook, the hook function is called.
All commands, spawned, remote, or local, can have Python functions executed before exection, by default, or post hooks that are run after the command. (Note: Post hooks are not run for spwaned commands because the resolver function is a post hook itself.) These functions can be passed arguments, too.
Command Hook Expressions
# Run before commands that match that pattern
hook-cmd "pattern" hook-function parms
# Run before commands that match that pattern, but is non-recursive
hook-cmd-nr "pattern" hook-function parms
# Run after commands that match that pattern
post-hook-cmd "pattern" hook-function parms
# Run after commands that match that pattern, but is non-recursive
post-hook-cmd-nr "pattern" hook-function parms
Hook Recursion
Hooks, which are nothing more than Python functions called before or after a command is run, can issue their own commands and, thus, cause the hook
to be recursively called. However, if the command in the hook block of code matches a command pattern that causes that same hook function to be run again,
an infinte loop can occur. To prevent that, use the -nr suffix on the Watiba hook expression. (-nr stands for non-recursive.) This will ensure that
the hook cannot be re-invoked for any commands that are within it.
To attach a hook:
1. Code one or more Python functions that will be the hooks. At the end of each hook, you must return True if the hook was successful, or False
if something wrong.
2. Use the _hook-cmd_ expression to attach those hooks to a command
pattern, which is a regular expression
3. To remove the hooks, use the _remove-hooks "pattern"_ expression. If a pattern, i.e. command regex pattern, is omitted, then all command hooks are removed.
hook-cmd "command pattern" function parms
The first parameter always passed to the hook function is the Python match object from the command match. This is provided so the hook has access
to the tokens on the command should it need them.
Example:
def my_hook(match, parms):
print(match.groups())
print(f'Tar file name is {match.group(1)}')
print(parms["parmA"])
print(parms["parmB"])
return True # Successful execution
def your_hook(match, parms):
# This hook doesn't need the match object, so ignores it
print(parms["something"])
if parms["something-else"] != "blah":
return False # Failed execution
return True # Successful excution
# Add first hook to my tar command
hook-cmd "tar -zcvf (\S.*)" my_hook: {"parmA":"A", "parmB":"B"}
# Add another hook to my tar command
hook-cmd "tar -zcvf (\S.*)" your_hook: {"parmD":1, "parmE":"something"}
# Spawn command, but hooks will be invoked first...
spawn `tar -zcvf files.tar.gz /tmp/files/* `:
# Resolver code block
return True # Resolve promise
Your parameters are whatever is valid for Python. These are simply passed to their attached functions, essentially each one's key is the function name, as specified.
Where are the hooks run for spawned commands? All hooks run under the thread of the issuer on the local host, not the target thread.
Where are the hooks run for remote commands? As with spawned commands, all hooks are issued on the local host, not the remote. Note that you
can have remote backticked commands in your hook and that will run those remotely. If your remote command matches a hook(s) pattern, then those hooks will be run. This means if your command pattern for the first remote call runs a hook that contains another remote command that matches that same command pattern, then the hook is run again. Since this can lead to infinte hook loops, Watiba offers a non-recursive definition for the command pattern. Note that this non-recursive setting
only applies to the command pattern and not the hook function itself. So if hookA is run for two different command patterns, say, "ls -lrt" and "ls -laF" you can
make one non-recusrive and still run the same hook for both commands. For the recursive command pattern, the hook has no limit to its recursion. For non-recursive,
it will only be called once during the recursion process.
To set a command pattern as non-recursive, use hook-cmd-nr.
Example using a variation on a previous example:
def my_hook(match, parms)
`tar -zcvf /tmp/files` # my_hook will NOT because for this command even though it matches
print("Will be called only once!")
return True
# Note the "-nr" on the expression. That's for non-recursive
hook-cmd-nr "tar -zcvf (\S.*)" my_hook: {"parmA":"A", "parmB":"B"}
# my_hook will be called before this command runs
` tar -zcvf tarball.tar.gz /home/user/files.*`
Command Chaining
Watiba extends its remote command execution to chaining commands across multiple remote hosts. This is achieved
by the chain expression. This expression will execute the backticked command across a list of hosts, passed by
the user, sequentially, synchronously until the hosts list is exhausted, or the command fails. chain returns a
Python dictionary where the keys are the host names and the values the WTOutput from the command run on that host.
Chain Exception
The chain expression raises a WTChainException on the first failed command. The exception raised
has the following properties:
WTChainException:
Property | Data Type | Description |
---|
|
command | String | Command that failed |
|
host | String | Host where command failed |
|
message | String | Error message |
|
output | WTOutput structure:
| Output from command |
Import this exception to catch it:
from watiba import WTChainException
Examples:
from watiba import WTChainException
try:
out = chain `tar -zcvf backup/file.tar.gz dir/*` {"hosts", ["serverA", "serverB"]}
for host,output in out.items():
print(f'{host} exit code: {output.exit_code}')
for line in output.stderr:
print(line)
except WTChainException(ex):
print(f"Error: {ex.message}")
print(f" host: {ex.host} exit code: {ex.output.exit_code} command: {ex.command})
Command Chain Piping (Experimental)
The chain expression supports piping STDOUT and/or STDERR to other commands executed on remote servers. Complex
arrangements can be constructed through the Python dictionary passed to the chain expression. The dictionary
contents function as follows:
-
"hosts": [server, server, ...] This entry instructions chain on which hosts the backticked command will run.
This is a required entry.
-
"stdout": {server:command, server:command, ...}
This is an optional entry.
-
"stderr": {server:command, server:command, ...}
This is an optional entry.
Just like a chain expression that does not pipe output, the return object is a dictionary of WTOutput object keyed
by the host name from the hosts list and not from the commands recieving the piped output.
If any command fails, a WTChainException is raised. Import this exception to catch it:
from watiba import WTChainException
Note: The piping feature is experimental as of this release, and a better design will eventually
supercede it.
Examples:
from watiba import WTChainException
# This is a simple chain with no piping
try:
args = {"hosts": ["serverA", "serverB", "serverC"]}
out = chain `ls -lrt dir/` args
for host, output in out.items():
print(f'{host} exit code: {output.exit_code}')
except WTChainException as ex:
print(f'ERROR: {ex.message}, {ex.host}, {ex.command}, {ex.output.stderr}')
# This is a more complex chain that runs the "ls -lrt" command on each server listed in "hosts"
# and pipes the STDOUT output from serverC to serverV and serverD, to those commands, and serverB's STDERR
# to serverX and its command
try:
args = {"hosts": ["serverA", "serverB", "serverC"],
"stdout": {"serverC":{"serverV": "grep something", "serverD":"grep somethingelse"}},
"stderr": {"serverB":{"serverX": "cat >> /tmp/serverC.err"}}
}
out = chain `ls -lrt dir/` args
for host, output in out.items():
print(f'{host} exit code: {output.exit_code}')
except WTChainException as ex:
print(f'ERROR: {ex.message}, {ex.host}, {ex.command}, {ex.output.stderr}')
####How does this work?
Watiba will run the backticked command in the expression on each host listed in hosts, in sequence and synchronously.
If there is a "stdout" found in the arguments, then it will name the source host as the key, i.e. the host from which
STDOUT will be read, and fed to each host and command listed under that host. This is true for STDERR as well.
The method in which Watiba feeds the piped output is through a an echo command shell piped to the command to be run
on that host. So, "stdout": {"serverC":{"serverV": "grep something"}} causes Watiba to read each line of STDOUT from
serverC and issue echo "$line" | grep something
on serverV. It is piping from serverC to serverV.
Installation
PIP
If you installed this as a Python package, e.g. pip, then the pre-compiler, watiba-c,
will be placed in your system's PATH by PIP.
GITHUB
If you cloned this from github, you'll still need to install the package with pip, first, for the
watbia module. Follow these steps to install Watiba locally.
# Watiba package required
python3 -m pip install watiba
Pre-compiling
Test that the pre-compiler functions in your environment:
watiba-c version
For example:
rwalk@walkubu:~$ watiba-c version
Watiba 0.3.26
To pre-compile a .wt file:
watiba-c my_file.wt > my_file.py
chmod +x my_file.py
./my_file.py
Where my_file.wt is your Watiba code.
Code Examples
my_file.wt
#!/usr/bin/python3
# Stand alone commands. One with directory context, one without
# This CWD will be active until a subsequent command changes it
`cd /tmp`
# Simple statement utilizing command and results in one statement
print(`cd /tmp`.cwd)
# This will not change the Watiba CWD context, because of the dash prefix, but within
# the command itself the cd is honored. file.txt is created in /home/user/blah but
# this does not impact the CWD of any subsequent commands. They
# are still operating from the previous cd command to /tmp
-`cd /home/user/blah && touch file.txt`
# This will print "/tmp" _not_ /home because of the leading dash on the command
print(f"CWD is not /home: {-`cd /home`.cwd)}"
# This will find text files in /tmp/, not /home/user/blah (CWD context!)
w=`find . -name '*.txt'`
for l in w.stdout:
print(f"File: {l}")
# Embedding commands in print expressions that will print the stderr output, which tar writes to
print(`echo "Some textual comment" > /tmp/blah.txt && tar -zcvf /tmp/blah.tar.gz /tmp`).stdout)
# This will print the first line of stdout from the echo
print(`echo "hello!"`.stdout[0])
# Example of more than one command in a statement line
if len(`ls -lrt`.stdout) > 0 or len(-`cd /tmp`.stdout) > 0:
print("You have stdout or stderr messages")
# Example of a command as a Python varible and
# receiving a Watiba object
cmd = "tar -zcvf /tmp/watiba_test.tar.gz /mnt/data/git/watiba/src"
cmd_results = `$cmd`
if cmd_results.exit_code == 0:
for l in cmd_results.stderr:
print(l)
# Simple reading of command output
# Iterate on the stdout property
for l in `cat blah.txt`.stdout:
print(l)
# Example of a failed command to see its exit code
xc = `lsvv -lrt`.exit_code
print(f"Return code: {xc}")
# Example of running a command asynchronously and resolving promise
spawn `cd /tmp && tar -zxvf tarball.tar.gz`:
for l in promise.output.stderr:
print(l)
return True # Mark promise resolved
# List dirs from CWD, iterate through them, spawn a tar command
# then within the resolver, spawn a move command
# Demonstrates spawns within resolvers
for dir in `ls -d *`.stdout:
tar = "tar -zcvf {}.tar.gz {}"
prom = spawn `$tar` {"dir": dir}:
print(f"{}args['dir'] tar complete")
mv = f"mv -r {args['dir']}/* /tmp/."
spawn `$mv`:
print("Move done")
# Resolve outer promise
promise.resolve_parent()
return True
# Do not resolve this promise yet. Let the inner resolver do it
return False
prom.join()