Introduction
Black boxes and APIs.
Sparrow plugins are underlying, essential part of sparrowdo system. One the one hand they are just a scripts to solve various tasks. Like create user accounts, populate configuration files or remove directories. One other hand they are more or less black boxes with well defined API exposed to external world.
Sparrowdo uses sparrow plugins as building blocks to manage and automate remote servers. In this article I am going to give an informal introduction into sparrow/sparrowdo echo system with the focus on it’s central part, heart of all – sparrow plugins. As much as possible, I will try not to burden the material with low level technical details which might be confusing for unprepared user, however sometimes a simple code examples and diagrams will occur here, hopefully helping you to catch the main ideas and not to gets strayed.
Bottom of the system.
Well. Not to diving too much into technical aspects let me try to explain informally what sparrow plugins are. We start from the very bottom of the system, like if would not wanted know anything about sparrowdo and only wanted to play with sparrow plugins ( indeed it’s possible without sparrowdo itself! ), so these are few basic entities we have to meet first:
* Scenarios
* Stories
* Suites
* Plugins
* Tasks
* Task boxes
Every single step ahead will lead us to the whole picture of sparrow ecosystem.
Scenarios
Scenarios are just scripts written on one of the language of choice – Perl5, Ruby or Bash. Sparrow provide an unified, language agnostic API for script developers so they leverage it:
* Easy scripts configuration ( made in various formats – command line, Config::General, JSON / YAML )
* Multi scripts scenarios – ability to call one script from another with parameters
* Check rules – ability verify scenarios output by embedded DSL
Picture 1. Scenarios & Stories.
Stories
Stories are abstraction for scenarios and its check rules. In terms of sparrow scenarios are always accompanied by a check list file – a list of definitions written in a special DSL to verify script output. In trivial case it could be empty file, thus no checking is taken. If user define some patterns check lost file to validate scenario output, the verification is taken.
Say we have a simple scenario “hello world”:
$ cat story.pl
print "hello world!\n"
In sparrow scenario is executed by story executor called strun which run scripts and checks if exit code is zero, which is treated as successful status. If a check rules are supplied scenario STDOUT gets validates against such a rules. Consider a trivial check rule for the script above:
$ cat story.check
hello world
It just checks that script output contains string , there are much more things you can do when validating like providing regex checking, capturing and handling matching data and so on, follow a Outthentic::DSL module documentation to get more. More or less many of plugins have none or simple check rules. But if you write a monitoring / testing / audit scripts a check rules feature could be extremely useful. Another interesting idea behind check rules is “self-testing scripts“, but I am not going to talk much about this here 🙂
Ok, if we take a look at the picture number 1, we will get a visual summary of all we’ve learned so far.
Let’s go ahead and talk about sparrow suites.
Suites
Picture 2. Suites.
Suites are related stories. One may have many scripts related to a one task or split complex thing into small scripts interacting with each other. A sparrow suite always have a “main” story ( denoted as “FIRST” at the picture number 2), which “call” others in chain. So we end up a tree of stories. A story being called is “downstream” story, a story calling downstream stories is a “upstream” story, obviously the same story could be both upstream/downstream. When a story gets called it might be given a story parameters. A unified API is provided to handle story parameters for whatever language you choose to write a scenario. Like for Bash we could have such a code:
# upstream story
run_story my-story message 'hello world'
# downstream story my-story.bash
message=$(story_var message)
echo $message
The same code in Perl would be:
# upstream story
run_story("my-story", { message => 'hello world' });
# downstream story
my $message = story_var('message');
print $message;
Story parameters being passed could be nested, which are pretty good represented via Perl or Ruby hashes. And even Bash is supported ( with some limitations ):
# set parameters at upstream story:
# Perl
run_story("S1", { message => { hello => 'world' } } );
# Ruby
run_story "S1", { :message => { :hello => 'world' } }
# Bash
run_story message.hello world
# access parameters at downstream story:
# Perl
story_var('message'){'hello'};
# Ruby
story_var('message')['hello'];
# Bash
$(story_var message.hello )
A technical details on sparrow stories could be found at Outthentic module documentation.
The same way as stories accept parameters and handle them using unified API, one may configure a sparrow suites. Say we want to pass some global parameters as suite input. Lets create first a default suite configuration, it could be ( one of the options, see later ) a Config::General format file:
$ cat suite.ini
nginx 80
tomcat 8080
dev_server 3000
$ strun --ini suite.ini
Now we have a unified API to access global parameters:
# access global parameters at story:
# Perl
config('app'){'servers_and_ports'}{nginx};
# Ruby
config('app')['servers_and_ports']['nginx'];
# Bash
$(config app.servers_and_ports.nginx )
We even can override a suite global parameters at run time via command line:
$ strun --param app.servers_and_ports.nginx=81
And finally we could use a JSON/YAML format to store global parameters:
$ cat suite.json
{
"app": {
"servers_and_ports": {
"nginx" : 80,
"tomcat" : 8080
"dev_server": 3000
}
}
}
$ strun --json suite.json
Default configuration and Hash merge.
If we have suite.ini configuration file for our suite it is considered as default configuration file. Thanks to Hash::Merge it is possible to override ( merge two files into one Hash ) a default values using custom configuration file:
$ cat suite.ini
bar = bar-default-value
baz = baz-default-value
$ strun --ini suite.ini # load a default configuration
$ cat suite.my.ini
bar = bar-new-value
$ strun --ini suite.my.ini
# will override bar value to `bar-new-value`
# baz value will remain default.
To know more about suites and their interfaces take a look at Outthentic documentation – https://github.com/melezhik/outthentic
Now let’s see how suites becomes a plugins.
Sparrow plugins
Picture 3. Sparrow plugins distribution system.
Plugins are packaged suites ready for contribution. From the end user point of view plugins act as suites, so they “borrowed” all the features we have learned so far.
But there are some extra values plugins add into the system:
* name and version
* dependencies ( CPAN / RubyGems )
* ownership
Every plugin has a name to be identified in a global sparrow system. This is obvious like having names for every software packages. As well as plugins have a versions so plugin developer may release and plugin users may utilize a various versions of plugins:
$ sparrow plg search nginx # search nginx related plugins
$ sparrow plg install nginx-check # install nginx-check plugin
$ sparrow plg run nginx-check # run nginx-check plugin
$ sparrow plg run nginx-check --version 0.0.8 # installing by version
Sparrow has quite extensive API on managing plugins, we can’t focus on here, please follow a documentation if you are interested. What is important here that plugins are small bits of software which is distributes the same as we see at many package systems like apt, CPAN, rpm, RubyGems so on.
Dependencies
In a spirit of ansible modules sparrow plugins does not depend on other plugins, but we can use any software libraries in our scenarios. Currently a plugin developer can declare CPAN dependence in a cpanfile or RubyGems dependencies in Gemfile, so that such a dependencies will be installed. Sparrow adjusts running environment ( setting library paths for Perl and Ruby ) so that installed libraries will be accessible in running scenario. It’s very handy!
Ownership
To publish plugins into central repository SparrowHub you need to get account there. It is also possible to distribute so called private plugins hosted at remote git repositories.
Metadata
All spoken could be written in simple JSON format. This is how sparrow plugins get registered in sparrow system:
{
"version" : "0.0.7.5",
"name" : "nginx-check",
"description" : "checks if nginx server is healthy by executing low level system checks ( ps, pid, etime )",
"url" : "https://github.com/melezhik/nginx-check"
}
Sparrow tasks
Sparrow plugins bind to a default suite configuration, there is not that much you can do about it, only redefine global parameter at run time:
$ sparrow plg run foo --param a=1 --param b=2
Picture 4. Sparrow plugins and tasks.
Sparrow tasks give you way more agile. Tasks are plugins with custom configurations. Tasks have names and grouped by projects:
$ sparrow task add foo-project foo-task foo
$ sparrow task ini foo-project/foo-task
a = 100
b = 200
$ sparrow task run foo-project/foo-task
There a lot of information about sparrow task at Sparrow documentation pages.
Ok, it’s been a long trip. We are approaching the end of evolution here 🙂 And this is sparrow task boxes.
Task box
Task box is a collection of sparrow tasks, we can write it as JSON:
[
{
"task" : "foo-task",
"plugin" : "foo-plugin",
"global_parameters" : {
"a" : 1,
"b" : 2
},
},
{
"task" : "bar-task",
"plugin" : "bar-plugin",
"global_parameters" : {
"aa" : 1,
"bb" : 2
}
}
]
Sparrow tasks is way to run many sparrow plugins with parameters and consequently. This actually what Sparrowdo does when compiling sparrowdo scenarios:
$ cat sparrowfile
user "zookeeper";directory "/var/data/zoo";
file "/var/data/zoo/birds.txt", %( owner => 'zookeeper' );
A given code gets complied into sparrow task box:
[
{
"plugin" : "user",
"task" : "create user zookeeper",
"data" : { "name" : "zookeeper", "action" : "create" }
},
{
"plugin" : "directory",
"task" : "create directory /var/data/zoo",
"data" : { "path" : "/var/data/zoo", "action" : "create" }
},
{
"plugin" : "file",
"task" : "create file /var/data/zoo/birds.txt",
"data" : {
"owner" : "zookeeper",
"action" : "create",
"target" : "/var/data/zoo/birds.txt"
}
}
]
From the very bottom of the system we have reached a sparrow evolution end point – a high level configuration management scenarios written on Perl6. But under the hood – it’s just a JSON gets pushed to sparrow client, do it will do low level job by executing sparrow plugins 🙂 , see the last picture:
Picture 5. Sparrow plugins evolution.
Summary
Let’s summarize what we’ve learned in this article:
* Sparrow plugins are scripts written on one of language of choice: Perl5/Bash/Ruby
* Outthentic – a core sparrow component – a development and execution kit to enable some frequently used features when writing automation scenarios: testing script output , reuse other scripts and pass script configuration parameters.
* To distribute scripts they are packaged and uploaded into central repository – SparrowHub
* Sparrow client is command line tool to install, configure and run plugins.
* Sparrowdo acts as high level system build upon sparrow plugins to write an automation scenarios in Perl6 language and then execute them as sparrow “plugins-primitives” with the JSON as internal presentation format and scp/ssh as transport.
I hope this was a helpful article, please post your comments, questions, ideas here.
Thanks.
— Alexey Melezhik
