Testing Command Line Applications using Tomty and RakuDist

So you have a command line application written on Raku and shipped as a Raku module to an end user. You’re quite happy with unit tests get run as a part of  zef install process.

But say, you need something  extra. For example, some post deployment tests that check your tool runs properly on a small set of scenarios. Here we enter  Tomty – Raku testing framework.

With a very simple code it’s possuble to add extra tests and run those tests under RakuDist – API to test Raku modules.

Zero Install

As tests are going to  be  run by RakuDist API, no dependencies to install is required, RakuDist will take care about execution environment, you just need to create tests and push them to Git as a part of .rakudist folder,  as this:

mkdir .rakudist/.tomty && git add .rakudist && git commit -a -m "my tests"

You still may want to install Tomty to debug/run tests locally :

zef install Tomty

Hello world example

Every test consists of two parts – high level wrapper written on Raku and underlying script written on a language of your choice (any Sparrow compatible language*)

* Perl,Bash,Ruby,Powershell,Python,Raku for the moment

Raku wrapper:

​nano .rakudist/.tomty/test-01.pl6

task-run "tasks/some-task";

Test script:

nano .rakudist/tasks/some-task/task.bash

echo "Hello World"

In this example it’s just a simple echo command, of course for real modules you would have a command line of  an application.

Let’s give it a run:

cd .rakudist/ && tomty --all --verbose

[test-01] ....... 
23:12:04 02/21/2020 [repository] index updated from file:///home/melezhik/repo/api/v1/index
23:12:04 02/21/2020 [tasks/some-task] Hello World
=========================================
(=: / [1] tests in 1 sec / (1) tests passed

At the very basic level Tomty just runs an external script and checks it exit code.

Task checks

Tomty allows to create so called task checks – Raku regexp based DSL to check a script output, in simplest form it could be just a plain string check:

nano .rakudist/some-task/task.check

Hello World

Now, Tomty not only checks if a script exits with zero exit code but also verifies it’s output:

tomty --all --verbose

[test-01] ....... 
23:16:16 02/21/2020 [repository] index updated from file:///home/melezhik/repo/api/v1/index
23:16:16 02/21/2020 [tasks/some-task] Hello World
[task check] stdout match <Hello World> True
=========================================
(=: / [1] tests in 1 sec / (1) tests passed

Script parameters

Raku wrappers not just call scripts,  they might pass some parameters:

task-run "tasks/some-task", %(  
  name => Raku 
)

With a simple modification a script could handle passed parameters:

.rakudist/tasks/some-task/task.bash

name=$(config name)
echo "Hello {$name}"

You can even handle parameters within task checks:

.rakudist/tasks/some-task/task.check

generator: <<CODE
!bash
  name=$(config name)
  echo "hello ${name}"
CODE

To know more about how to write Sparrow scenarios follow documentation.

More tests

Eventually more tests are added by creating new Raku wrappers inside .rakudist/.tomty directory and respected scripts inside .rakudist/tasks folder:

.tomty/test-02.pl6
.tomty/test-03.pl6
...

The whole test suite as usually is executed by tomty --all command

Conclusion

We’ve ended up with a simple file structure to test command line application distributed as a Raku module:

.rakudist/
├── tasks
│   └── some-task
│       ├── task.bash
│       └── task.check
└── .tomty
    └── test-01.pl6
3 directories, 3 files

If you want to see more examples, take a look at my pull request for App::Mi6 application.

I would love to hear a feedback and ideas on possible usage of the toolset.

Thank you for reading

RakuDist Update – Travis Integration and Post Deployment Tests

Hi! I continue to make updates on RakuDist project aimed to automate a process of testing Raku modules.

Here is a couple of interesting features I have added recently.

Enter Travis CI integration and post deployemt checks.

Travis CI

If one need to run RakuDist through Travis CI it’s as simple as that:

language: minimal

script:
 - curl http://repo.westus.cloudapp.azure.com/rakudist/api/run/$os/$author/$module -s | bash

Where:

$os – tested OS name, for example `debian`
$author – github(*) login
$module – module name

(*) – RakuDist supports GitLab projects as well I just need to write a proper helper for that.

Post deployment scenarios

Quite often Raku modules distributions include scripts that are part of an installation process.

RakuDist allows to test such a scripts at post deployment phase:

nano smokefile:

bash 'your-script', %(
  user => config()<user>,
  expect_stdout => 'this string should be in a script output'
);

 

You could see more sophisticated and interesting examples here.

I would like to hear from Raku modules authors if this feature is useful.

That’s all for @today, stay tuned!

RakuDist Update – Raku Modules Custom Installation.

Custom installation

I continue to play with RakuDist API searching for useful cases for Raku modules authors.

With RakuDist/Sparrow6 one can easily customize an installation and testing process.

Why would people need it? For various reasons:

  • A module requires external libraries not satisfied by zef install
  • A module requires external services – e.g. databases servers running and so on
  • A module requires extra configuration files which are hard to deliver by standard distribution means

To address all the needs RakuDist provides a handy functionality called custom scenarios.

Custom scenarios

Just drop a .rakudist folder into your Raku module source code, with some custom scenario inside:

.rakudist/sparrowfile

Say, you need an installation of an sqlite library as your module relies on it:

cat .rakudist/sparrowfile

package-install "sqlite-dev";

That is it. Now commit and push changes to git:

git add .rakudist && git commit -m "my custom scenario" && git push

Now you may test your distribution hosted on GitHub through API:

POST os=$os project=$author/$project /rakudist/api/run/:github

For example to test Teddy::Bear module:

curl \
-d os=alpine \
-d project=melezhik/rakudist-teddy-bear \
http://repo.westus.cloudapp.azure.com/rakudist/api/run/:github

Once you’ve packed .rakudist/ folder in a CPAN distribution, you can test it as well:

POST os=$os  /rakudist/api/run/$module

For example:

curl \
-d os=alpine \
http://repo.westus.cloudapp.azure.com/rakudist/api/run/Teddy::Bear

 

More examples

1. Custom tasks/scripts

Sparrow6 DSL allows to write scenarios of any complexity and if it still is not enough one could add custom tasks written on many languages to extend the functionality:

cat .rakudist/tasks/hello-world/task.bash

# your Bash code here

And then inside Sparrow scenario just:

cat .rakudist/sparrowfile

task-run "tasks/hello-world";

2. Raku dependencies not declared in META6.json file

Sometimes developers might be not ready to include dependencies in meta6 file for various reasons. For example, when a dependent module itself is not released yet, or if the dependency fails and needs to be installed as --/test exclusively, here is depends.raku for the rescue:

Just create .rakudist/depends.raku file declaring Raku dependencies in simple format:

OpenSSL notest
git://github.com/frithnanth/raku-Math-Libgsl-Constants.git

Here we want to install OpenSSLwithout unit tests and Math::Libgsl::Constants from GitHub sources.

3. Existing modules examples

I’ve recently made a PR to Math::Libgsl::Permutation and to PR to Archive::Libarchive::Raw to support RakuDist test against various OS systems. These modules rely on external libraries gsl and libarchive and so need to be tested against platforms. RakuDist comes very handy in such case.

Conclusion

RakuDist keeps growing, but is still in quite experimental phase, to see the project potential, I would like to have a feedback from Raku modules authors. Thanks.

RakuDist Update. Rakudo Versions Support.

I continue to post updates on RakuDist project aiming to help Raku modules authors to test their distributions on different environments.

Here is very quick ( I lack of time due to my main $work ) but exciting update. Now RakuDist supports  various Rakudo versions.

Long story short

Yes, you can now test your shiny Raku  module install against arbitrary Rakudo version. Now RakuDist expose simple ( don’t blame me if the server will blow under a load 😉 ) yet handy API to test distribution online:

curl -d os=debian -d rakudo_version=SHA $api_url/rakudist/api/run/$module

For example to test Tomty module against rakudo version 40b13322c503808235d9fec782d3767eb8edb899 on Debian say this:

curl -d os=debian \
-d rakudo_version=40b13322c503808235d9fec782d3767eb8edb899 \
http://repo.westus.cloudapp.azure.com/rakudist/api/run/Tomty

 

Low level API

If for some reasons you don’t want to use public API, you can run RakuDist test scenarios on on-premise infrastructure. All you need is a docker container!

Following a simple workflow:

Install Sparrowdo

zef install --/test Sparrowdo

Pull docker images

docker pull debian

Run container

docker run -d -t --rm --name debian-rakudist debian

Write test

Test scenarios are written on Sparrow6 DSL

cat sparrowfile
zef "https://github.com/melezhik/sparrow6-rakudo-install.git", %( 
  force => True 
);

user "test123";

my $path = module-run 'Rakudo::Install', %(
  user => 'test123',
  rakudo-version => '40b13322c503808235d9fec782d3767eb8edb899'
);

# Now you can use installed Rakudo

bash "perl6 --version", %(
  envvars => %( PATH => "{%*ENV<PATH>}:{$path}" ),
  user => "test123",
  debug => True
);

bash "zef install --/test Kind --verbose", %(
  description => "install Kind",
  user => "test123",
  envvars => %( PATH => "{%*ENV<PATH>}:{$path}" ),
);

 

Run test

sparrowdo --bootstrap \
--no_sudo --docker=debian-rakudist \
--repo=http://repo.westus.cloudapp.azure.com


Conclusion

Read more on https://github.com/melezhik/RakuDist

 

RakuDist update

One week ago I started a discussion in Raku community on testing Raku modules suggesting Sparrow as a solution. While the discussion has been being held and various opinions have been being given, I’d like to share my experience and indeed some good results I’ve had with applying Sparrow tool to the task of testing Raku modules.

First results

* It reveals bugs – drforr/raku-OLE-Storage_Lite/issues/1 , FCO/Red/issues/421 , titsuki/raku-Chart-Gnuplot/issues/40 , HomoGlypher/issues/2

* It’s flexible enough to create even complicated tests involving databases and services, see FCO/Red/issues/421

* It’s dead simple.  Taking me around 1-2 minutes to write a test for an average module. See the list of examples

* It’s light. With Alpine OS image weighted 5MB and very small Sparrow6 dependency tree, the overall test of average module would take no more 2-3 minutes. If we run on already bootstrapped instance – it’s even a seconds!

* It’s has clear and understandable syntax. Sparrow6 DSL is dead simple and easy to read and write, just see a following example

Let’s get started

The main workflow is dead easy. Spin up a docker instance, install Sparrowdo, drop a simple scenario and give it a run! Boom, within a couple of minutes you’ll get your report nice and readable. No need to wait in queue in Travis, it’s cheap, it’s easy and it’s yours!

Spin a docker instance.

Choose an alpine OS image, 5 MB. You pull it and run it with literally seconds!

docker pull alpine
docker run -d -t --rm --name alpine-rakudist alpine

 

Bootstrap docker instance

Install Sparrowdo – a thin client to run Sparrow6 tasks on docker instance. It’s really light, just a few Raku modules as dependencies. Then bootstrap your docker instance with Rakudo and Sparrow6 environment, the last one is also extremely light and it won’t take more then a couple of minutes for the whole process and should be done only once.

zef install --/test Sparrowdo
sparrowdo --bootstrap --no_sudo --docker=alpine-rakudist

 

Create your test

Sparrow6 DSL has all you need. Nice wrappers around zef installer, user management to create users if you need to test users installations and git-scm wrapper to fetch a module source code from Git repositories.

The simplest scenario would be something like that:

$ cat sparrowfile:

my $user = config()<user>;

my $directory = "/data/test/{$user}";

my $scm = config()<scm>;

user $user;

directory $directory, %( 
  owner => $user,
  group => $user

);

git-scm $scm, %( 
  to => $directory, 
  user => $user,
);

if os() eq 'alpine' {

  # this is needed for alpine rakudo installation
  unless "/bin/zef".IO ~~ :e {
    copy "/opt/rakudo-pkg/share/perl6/core/bin/zef", "/bin/zef"
  }

  # this is needed for alpine rakudo installation
  unless "/bin/perl6".IO ~~ :e {
    copy "/opt/rakudo-pkg/bin/perl6", "/bin/perl6"
  }

}

zef $directory, %( 
  force => False,
  depsonly => True, 
  user => $user,
  description => "Install module dependencies"
);

bash "cd {$directory} && zef test .", %(
  description => "zef test",
  user => $user
);

 

The scenario takes some parameters from config.pl6 , e.g. source code repository url and user name to run tests under:

%(
  user => "kind",
  scm =>  "https://github.com/Kaiepi/p6-Kind.git"
)

 

Users installations allows to reuse the same docker instance to test various modules. Because dependencies get installed under user’s account and never clash with other modules. Another option would be – relaunch a new docker instance for every test and start from the scratch but this will add time for initialization phase.

Run scenario

I promised it’d be easy, so just launch our  new shiny test on docker alpine instance:

sparrowdo --no_sudo --docker=alpine-rakudist --repo=http://repo.westus.cloudapp.azure.com

And here the report:

16:50:02 01/09/2020 [repository] index updated from http://repo.westus.cloudapp.azure.com/api/v1/index
16:50:07 01/09/2020 [create user kind] uid=1006(kind) gid=1006(kind) groups=1006(kind)
16:50:07 01/09/2020 [create user kind] user kind created
[task check] stdout match <created> True
16:50:11 01/09/2020 [create directory /data/test/kind] directory path: /data/test/kind
16:50:11 01/09/2020 [create directory /data/test/kind] directory owner: <kind>
16:50:11 01/09/2020 [create directory /data/test/kind] directory group: <kind>
16:50:11 01/09/2020 [create directory /data/test/kind] directory access rights: drwxr-xr-x
[task check] stdout match <owner: <kind>> True
[task check] stdout match <group: <kind>> True
16:50:15 01/09/2020 [bash: git checkout https://github.com/Kaiepi/p6-Kind.git] /data/test/kind
16:50:15 01/09/2020 [bash: git checkout https://github.com/Kaiepi/p6-Kind.git] stderr: Cloning into '.'...
16:50:17 01/09/2020 [bash: last commit] commit c08acf1e4a52491a3b09e19e129efc3092cef745
16:50:17 01/09/2020 [bash: last commit] Author: Ben Davies <kaiepi@outlook.com>
16:50:17 01/09/2020 [bash: last commit] Date:   Tue Dec 24 16:28:29 2019 -0400
16:50:17 01/09/2020 [bash: last commit]
16:50:17 01/09/2020 [bash: last commit]     Release v0.1.0
16:50:17 01/09/2020 [bash: last commit]
16:50:17 01/09/2020 [bash: last commit] M       CHANGELOG
16:50:17 01/09/2020 [bash: last commit] M       META6.json
16:50:17 01/09/2020 [bash: last commit] M       lib/Kind.pm6
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] total 32
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] -rw-r--r--. 1 kind kind  191 Jan  9 16:50 CHANGELOG
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] -rw-r--r--. 1 kind kind 8902 Jan  9 16:50 LICENSE
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] -rw-r--r--. 1 kind kind  408 Jan  9 16:50 META6.json
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] -rw-r--r--. 1 kind kind 3230 Jan  9 16:50 README.md
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] -rw-r--r--. 1 kind kind 3337 Jan  9 16:50 README.pod6
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] -rw-r--r--. 1 kind kind  114 Jan  9 16:50 dist.ini
16:50:20 01/09/2020 [bash: cd /data/test/kind && ls -l] drwxr-xr-x. 2 kind kind   22 Jan  9 16:50 lib
16:50:20 01/09/2020 [bash: cd /data/test/kind && ls -l] drwxr-xr-x. 2 kind kind   23 Jan  9 16:50 t
16:50:24 01/09/2020 [bash: Install module dependencies] stderr: All candidates are currently installed
16:50:24 01/09/2020 [bash: Install module dependencies] <empty stdout>
16:50:27 01/09/2020 [bash: zef test] ===> Testing: Kind:ver<0.1.0>
16:50:29 01/09/2020 [bash: zef test] [Kind] t/01-kind.t .. ok
16:50:29 01/09/2020 [bash: zef test] [Kind] All tests successful.
16:50:29 01/09/2020 [bash: zef test] [Kind] Files=1, Tests=3,  2 wallclock secs ( 0.02 usr  0.00 sys +  2.68 cusr  0.21 csys =  2.91 CPU)
16:50:29 01/09/2020 [bash: zef test] [Kind] Result: PASS
16:50:29 01/09/2020 [bash: zef test] ===> Testing [OK] for Kind:ver<0.1.0>

 

Conclusion

Sparrow could be a reasonable alternative to existing CI tools allowing test Raku distributions easy and fast.  Give it a try and say you comments! Follow the RakuDist project for examples of test scenarios and further updates.

 

Sparrowdo on Docker

With the latest Sparrowdo commits it’s now possible to run Sparrowdo tasks on running Docker containers:

Spin up a Docker container:

$ docker pull bitnami/minideb-extras # Debian minimal
$ docker run -d instance0 -it minideb-extras bash

Create some Sparrowdo scenario:

$ cat sparrowfile
use v6;
use Sparrowdo;
task-run 'check disk available space', 'df-check', %( threshold => 80 );
bash 'pwd';

Ran Sparrowdo scenario on running Docker container:

$ sparrowdo --docker=instance0 --bootstrap --no_sudo --format=production

Output:

running sparrow tasks on 127.0.0.1 ... 
target OS is - ubuntu
push [task] check disk available space [plg] df-check OK
push [task] run bash: pwd ... OK
SPL file /opt/sparrow/sparrow.list is empty
get index updates from SparrowHub ... OK
set up task box file - /home/melezhik/.sparrowdo//opt/sparrow/task-box.json - OK
public@df-check is uptodate (0.2.3)
public@bash is uptodate (0.1.6)
running task box from /opt/sparrow/sparrow-cache/task-box.json ... 
2017-09-22 11:12:39 : [task] check disk available space [plg] df-check [path] /
threshhold: 80
2017-09-22 11:12:39 : [task] run bash: pwd ... [path] modules/bash-command/ [params] envvars:

 
Caveats

You should have bash and curl pre installed at your Docker container.

Minoca OS automation with Sparrowdo

Introduction

Hello! Minoca is a new operating system for the world of connected devices. In this post I am going to show you how one can enable configuration management of running Minoca instances by the help of Sparrowdo.

Download the latest Minoca build

$ wget http://www.minocacorp.com/download/nightlies/latest-x86/Minoca-pc.zip
$ unzip Minoca-pc.zip

Start Minoca OS instance

$ qemu-system-x86_64 -enable-kvm -m 2000 -net nic,model=i82559er -net user,hostfwd=tcp::2222-:22,hostfwd=tcp::8888-:80 -hda pc.img

Set up remote ssh access

Inside running Minoca instance:

$ opkg update 
$ opkg install openssh bash
$ /etc/init.d/sshd start
$ passwd # we will use this password when ssh-ing from the Host OS after `ssh-copy-id` gets run

Inside Host OS:

$ ssh-copy-id -p 2222 root@127.0.0.1
$ ssh -p 2222 root@127.0.0.1
$ exit

Create some Sparrowdo scenario

$ nano sparrowfile
use v6;
use Sparrowdo;
package-install ("nano", "zsh", "nginx");
user "alexey";
directory "/var/data/bar", %( owner => "alexey");
service-stop "nginx";
service-start "nginx";
http-ok %( port => 8888 );

Run Sparrowdo scenario for Minoca instance

$ sparrowdo --host=127.0.0.1 --ssh_user=root --ssh_port=2222 --no_sudo --sparrowfile=sparrowfile --bootstrap --format=production

The output:

running sparrow bootstrap for host: 127.0.0.1 ... 
bootstrap for minoca
Downloading http://www.minocacorp.com/packages/0.4/i686/main/Packages.gz.
Inflating http://www.minocacorp.com/packages/0.4/i686/main/Packages.gz.
Updated list of available packages in /var/opkg-lists/main.
/usr/bin/curl
/usr/bin/perl
/usr/bin/cpanm
/usr/bin/sparrow
Outthentic is up to date. (0.3.9)
Sparrow is up to date. (0.2.48)
running sparrow tasks on 127.0.0.1 ... 
target OS is - minoca
push [task] install packages: nano zsh nginx OK
push [task] create user alexey OK
push [task] create directory /var/data/bar OK
push [task] stop service nginx OK
push [task] start service nginx OK
push [task] run bash: curl -fsSLk -D - --retry 3 127.0.0.1:8888 -o /dev/ ... OK
SPL file /opt/sparrow/sparrow.list is empty
get index updates from SparrowHub ... OK
set up task box file - /home/melezhik/.sparrowdo//opt/sparrow/task-box.json - OK
public@package-generic is uptodate (0.3.7)
public@user is uptodate (0.2.1)
public@directory is uptodate (0.1.4)
public@service is uptodate (0.1.13)
public@bash is uptodate (0.1.6)
running task box from /opt/sparrow/sparrow-cache/task-box.json ... 
2017-09-21 02:27:12 : [task] install packages: nano zsh nginx [path] modules/opkg/ [params] action:install package:nano
2017-09-21 02:27:12 : [task] install packages: nano zsh nginx [path] modules/opkg/ [params] action:install package:zsh
2017-09-21 02:27:12 : [task] install packages: nano zsh nginx [path] modules/opkg/ [params] action:install package:nginx
2017-09-21 02:27:13 : [task] create user alexey [path] modules/create/
2017-09-21 02:27:13 : [task] create directory /var/data/bar [path] modules/create/
2017-09-21 02:27:14 : [task] stop service nginx [path] modules/stop/ [params] os:minoca service:nginx
2017-09-21 02:27:14 : [task] start service nginx [path] modules/start/ [params] os:minoca service:nginx
2017-09-21 02:27:14 : [task] run bash: curl -fsSLk -D - --retry 3 127.0.0.1:8888 -o /dev/ ... [path] modules/bash-command/ [params] envvars:

Building Perl6 Applications with Docker and Ducky

Docker containers allow developers run environments and deploy application easy and fast. Dockerfile/Ansible/Chef are means you can configure bootstrapped Docker instances, however there is another way to do this …

Ducky is a lightweight Docker provision tool allow easy deploy Docker containers in just creating JSON scenarios in declarative way:

$ cat ducky.json
[
 {
   "task" : "install perl6",
   "plugin" : "rakudo-install",
   "data" : {
   "url" : "https://github.com/nxadm/rakudo-pkg/releases/download/2017.07/perl6-rakudo-moarvm-CentOS7.3.1611-20170700-01.x86_64.rpm"
  }
 }
]

This is how we bootstrap Rakudo on Docker box by using Ducky and this simple scenario.  Now let’s pull CentOS image and run Docker container based on it:

$ docker pull centos
$ docker run -d -i -t -v $PWD:/var/ducky --name ducky-centos centos

The only requirement here is that the running Docker container should have current working directory, ( which holds Ducky json file ) mounted as /var/ducky.

Ducky picks up ducky.json file placed in the current working directory and executes scenario on running Docker container, named centos-ducky:

$ ducky.bash ducky-centos

Here is a piece of screen shot of Ducky’s output ( only the last lines are shown for the sake of brevity ):

ducky2

Under the hood Ducky installs Sparrow client on the container and then give it a run  to execute tasks defined at Ducky json file.  The tasks are described at  Sparrow task box format.

So Ducky json is just a Sparrow Task Box file. That means you can declare Sparrow plugins with parameters here, aka Sparrow tasks which are executed on a Docker container. Available plugins are listed, documented and stored at the SparrowHub – Sparrow plugins repository.

In this scenario we use rakudo-install plugin to install Rakudo as system pcakge. The plugin documentation is available at the SparrowHub site.

Thus there are more things you could do with Ducky not only installing software,  you’re only limited by existed Sparrow plugins.

A typical use case is to run Test::Harness against a Perl6 project. Let’s do it for  Bailador  which is “A light-weight route-based web application framework for Perl 6” :

$ git clone https://github.com/Bailador/Bailador.git
$ cd Bailador
$ cat ducky.json 

[

  {
    "task" : "install perl6",
    "plugin" : "rakudo-install",
    "data" : {
      "url" : "https://github.com/nxadm/rakudo-pkg/releases/download/2017.07/perl6-rakudo-moarvm-CentOS7.3.1611-20170700-01.x86_64.rpm"
    }
  },
  {
    "task" : "installs Bailador dependencies",
    "plugin" : "zef",
    "data" : {
        "list" : [ "." ],
        "options" : "--deps-only"
    }
  },
  {
    "task" : "run t/ tests",
    "plugin" : "bash",
    "data" : {
        "command" : "prove6 -l",
        "envvars" : {
          "PATH" : "/opt/rakudo/bin:/opt/rakudo/share/perl6/site/bin:/root/.rakudobrew/moar-nom/install/share/perl6/site/bin:$PATH"
        }
    }
  }
]

Ducky json is quite self illustrative, here we define some standard steps to build and test the project:

* Install Raudo
* Install Bailador dependencies  picked from META6 file
* Runs t/ tests with prove6

Ok, let’s give it a run: ( don’t forget that we should first launch Docker container with the current working directory mounted as /var/ducky ):

$ docker run -d -i -t -v $PWD:/var/ducky --name ducky-bailador centos
$ ducky.bash ducky-bailador

Here is the last lines of the Ducky output:

ducky3

In this scenario we use 2 other plugins – bash – to execute arbitrary Bash code and zef – simple wrapper for Zef manager – tool to install Perl6 modules. The plugins documentation is available at the SparrowHub site.

Further thoughts.

Ducky and Sparrow are kind of cross platform tools, meaning you can successfully run the same scenarios on the variety of Linux platforms ( provided that Bash is installed ), for example the last scenario will succeed when running against Alpine Linux   docker image:

$ docker pull melezhik/alpine-perl6
$ docker run -d -i -t -v $PWD:/var/ducky --name ducky-bailador-alpine melezhik/alpine-perl6
$ ducky.bash ducky-bailador-alpine

Thus it becomes extremely  useful when you want to test a project against different environments just sitting at your developer box and running chip docker containers.

And the last but not the least. If for some reasons you’re not satisfied by existed Sparrow plugins you can easy write new one to cover your needs. I have written a plenty of posts of how to do this and you yon may start with this one – Outthentic – quick way to develop user’s scenarios .

Regards and have a fun with your coding and automation.

How to use Chef and Sparrowdo together

Good team member.

Chef is a well recognized configuration management tool which I use extensively at my current work. However I keep pushing to Sparrowdo – Perl6 configuration management tool and find those two tools play nicely together.

In this post I am going to give a few examples on how I use Sparrowdo to simplify and improve Chef cookbooks development workflow.

Running chef client on a target host.

Here is the most useful scenario how I use Chef and Sparrowdo together. My working environment implies launching ec2 Amazon instances get configured by chef. Instead of ssh-ing to an instance and running a chef-client on it, I delegate this task to Sparrowdo using wrapper called Sparrowdo::Chef::Client.

Let’s install the module:

$ zef install Sparrowdo::Chef::Client

And then create a simple Sparrowdo scenario:

$ cat sparrowfile
module_run 'Chef::Client', %(
    run-list => [
      "recipe[foo]",
      "recipe[baz]"
    ],
    log-level => 'info',
    force-formatter => True
);

Here we’re just running two recipes called foo and bar. And define some chef client’s settings, like log level and enabling force-formater option. Now we can run a chef-client on a target host:

$ sparrowdo --host=$target_host

Post deployments checks.

It is always good idea to check a server’s state right after the deployment. There are reasons why I prefer to not keep such a checks inside my Chef scenarios. And it seems there is a trend which is seen as new monitor and audit tools appear at the open source market with InSpec and goss among them, to list a few.

Likewise Sparrowdo has some built-in facilities to quickly test an infrastructure.

Let me give you a few examples.

Check system processes.

Say, we reconfigure an Nginx server by using some Chef recipes, sometimes Chef is not able to ensure that Nginx starts successfully after deploy or even if it does I don’t want to grep huge chef client logs ( sometimes there is a load of them ) to find out whether an Nginx gets started successfully. Happily, here is the dead simple solution – the usage of Sparrowdo asserts:

$ cat healthcheck.pl6
proc-exists 'nginx';

This function checks that file /var/run/nginx.pid exists as well as the related process with the PID taken from the file does. If you need to handle uncommon file paths for pid files, you can always set the path explicitly:

$ cat healthcheck.pl6
proc-exists-by-pid 'nginx server', '/var/run/nginx/ngix.pid';

Moreover if you only know the “name” of the process ( well technically specking this regular expression to match the process command ), simply have this:

$ cat healthcheck.pl6
proc-exists-by-footprint 'nginx web server', 'nginx\s+master';

Having this simple Sparrowdo scenario just run it against a target server to check that Nginx process exists:

$ sparrowdo --host=$target_host --sparrowfile=healthcheck.pl6

I always put Sparrowdo scenarios and Chef cookbook files together and commit them to Git repository:

$ git add sparrowfile healthcheck.pl6
$ git commit -a -m 'sparrowdo scenarios for chef cookbook'

And finally let me give an example of checking web application endpoints by sending http requests. Say, we have Chef recipe which deploys an application that should be accessible by http GET / request. Sparrowdo exposes handy http-ok asserts to deal with such a checks:

$ cat healthcheck.pl6
http-ok;

That is it! This is the simplest form of http-ok function’s call to verify that the web application is responsible by  accepting  requests for GET / route. Under the hood it just:

  1. resolves hostname as those one you run sparrowdo
  2. issues http request using curl utility:
$ curl -f http://$target_host

There are options how you can call http-ok function. For example, you may define endpoints and set http port:

$ cat healthcheck.pl6
http-ok(port  => '8080' , path => '/Foo/Bar' );

Follow Sparrowdo documentation for full description of http asserts function.

Conclusion

Using Sparrowdo and Chef together could be efficient approach when developing and testing server’s configuration scenarios. Sparrowdo is proven to be able to adopt any requirements and play nicely with other ecosystems and frameworks being an intelligent “clue” to bind together various tools and get your work done.