RakuDist Update. Raku Modules Installation Customization.

Installation customization

I continue to play with RakuDist API searching for useful cases for Raku modules authors.

With RakuDist/Sparrow6 one can easily customize installation and testing process.

Why would people need it? For various reasons:

* A module requires external libraries not satisfied by zef install
* A module requires external services – e.g. databases servers running and so on
* A module requires extra configuration files which are hard to deliver by standard distribution needs

To address all the needs RakuDist provides a handy functionality called custom scenarios.

Custom scenario

Just drop a .rakudist folder into your Raku module source code, with some custom scenario inside:

.rakudist/sparrowfile

Say, you need an installation of an sqlite library as your module relies on it:

cat .rakudist/sparrowfile

package-install "sqlite-dev";

That is it. Now commit and push changes to git:

git add .rakudist && git commit -m "my custom scenario" && git push

Now you may test your distribution through API:

POST os=$os project=$author/$project /rakudist/api/run/:github

Here is report for an example Teddy::Bear module:

curl \
-d os=alpine \
-d project=melezhik/rakudist-teddy-bear \
http://repo.westus.cloudapp.azure.com/rakudist/api/run/:github

20:58:45 01/27/2020 [repository] index updated from http://repo.westus.cloudapp.azure.com/api/v1/index
20:58:51 01/27/2020 [create user gh-rakudist-teddy-bear-alpine] uid=1008(gh-rakudist-teddy-bear-alpine) gid=1008(gh-rakudist-teddy-bear-alpine) groups=1008(gh-rakudist-teddy-bear-alpine)
20:58:51 01/27/2020 [create user gh-rakudist-teddy-bear-alpine] user gh-rakudist-teddy-bear-alpine created
[task check] stdout match <created> True
20:58:54 01/27/2020 [delete directory /data/test/gh-rakudist-teddy-bear-alpine] directory-delete-ok
[task check] stdout match <directory-delete-ok> True
20:58:58 01/27/2020 [create directory /data/test/gh-rakudist-teddy-bear-alpine] directory path: /data/test/gh-rakudist-teddy-bear-alpine
20:58:58 01/27/2020 [create directory /data/test/gh-rakudist-teddy-bear-alpine] directory owner: <gh-rakudist-teddy-bear-alpine>
20:58:58 01/27/2020 [create directory /data/test/gh-rakudist-teddy-bear-alpine] directory group: <gh-rakudist-teddy-bear-alpine>
20:58:58 01/27/2020 [create directory /data/test/gh-rakudist-teddy-bear-alpine] directory access rights: drwxr-xr-x
[task check] stdout match <owner: <gh-rakudist-teddy-bear-alpine>> True
[task check] stdout match <group: <gh-rakudist-teddy-bear-alpine>> True
20:59:02 01/27/2020 [bash: remove ~/.perl6 for user gh-rakudist-teddy-bear-al ...] <empty stdout>
20:59:05 01/27/2020 [bash: git checkout https://github.com/melezhik/rakudist- ...] /data/test/gh-rakudist-teddy-bear-alpine
20:59:05 01/27/2020 [bash: git checkout https://github.com/melezhik/rakudist- ...] stderr: Cloning into '.'...
20:59:08 01/27/2020 [bash: last commit] commit 92498b7f57bce83185baef74524f5c848fe25b09
20:59:08 01/27/2020 [bash: last commit] Author: Alexey Melezhik <melezhik@gmail.com>
20:59:08 01/27/2020 [bash: last commit] Date:   Thu Jan 23 15:38:16 2020 -0600
20:59:08 01/27/2020 [bash: last commit] 
20:59:08 01/27/2020 [bash: last commit]     Update README.md
20:59:08 01/27/2020 [bash: last commit] 
20:59:08 01/27/2020 [bash: last commit] M	README.md
Running custom scenario from /data/test/gh-rakudist-teddy-bear-alpine/.rakudist/sparrowfile ...
20:59:10 01/27/2020 [install package(s): sqlite-dev.perl] fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
20:59:10 01/27/2020 [install package(s): sqlite-dev.perl] fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
20:59:10 01/27/2020 [install package(s): sqlite-dev.perl] v3.11.3-19-gb3a750a9f7 [http://dl-cdn.alpinelinux.org/alpine/v3.11/main]
20:59:10 01/27/2020 [install package(s): sqlite-dev.perl] v3.11.3-20-gb834bd1f6e [http://dl-cdn.alpinelinux.org/alpine/v3.11/community]
20:59:10 01/27/2020 [install package(s): sqlite-dev.perl] OK: 11263 distinct packages available
20:59:11 01/27/2020 [install package(s): sqlite-dev.perl] trying to install sqlite-dev ...
20:59:11 01/27/2020 [install package(s): sqlite-dev.perl] installer - apk
20:59:12 01/27/2020 [install package(s): sqlite-dev.perl] (1/3) Installing pkgconf (1.6.3-r0)
20:59:12 01/27/2020 [install package(s): sqlite-dev.perl] (2/3) Installing sqlite-libs (3.30.1-r1)
20:59:12 01/27/2020 [install package(s): sqlite-dev.perl] (3/3) Installing sqlite-dev (3.30.1-r1)
20:59:12 01/27/2020 [install package(s): sqlite-dev.perl] Executing busybox-1.31.1-r8.trigger
20:59:12 01/27/2020 [install package(s): sqlite-dev.perl] OK: 70 MiB in 35 packages
20:59:12 01/27/2020 [install package(s): sqlite-dev.perl] Installed:                                Available:
20:59:12 01/27/2020 [install package(s): sqlite-dev.perl] sqlite-dev-3.30.1-r1                    = 3.30.1-r1 
20:59:12 01/27/2020 [install package(s): sqlite-dev.perl] sqlite-dev
20:59:14 01/27/2020 [bash: cd /data/test/gh-rakudist-teddy-bear-alpine && ls  ...] total 8
20:59:14 01/27/2020 [bash: cd /data/test/gh-rakudist-teddy-bear-alpine && ls  ...] -rw-r--r--    1 gh-rakud gh-rakud       622 Jan 27 20:59 META6.json
20:59:14 01/27/2020 [bash: cd /data/test/gh-rakudist-teddy-bear-alpine && ls  ...] -rw-r--r--    1 gh-rakud gh-rakud       885 Jan 27 20:59 README.md
20:59:14 01/27/2020 [bash: cd /data/test/gh-rakudist-teddy-bear-alpine && ls  ...] drwxr-xr-x    3 gh-rakud gh-rakud        19 Jan 27 20:59 lib
20:59:14 01/27/2020 [bash: cd /data/test/gh-rakudist-teddy-bear-alpine && ls  ...] drwxr-xr-x    2 gh-rakud gh-rakud        23 Jan 27 20:59 t
20:59:20 01/27/2020 [bash: zef install .] ===> Testing: Teddy Bear:ver<0.0.1>
20:59:21 01/27/2020 [bash: zef install .] ===> Testing [OK] for Teddy Bear:ver<0.0.1>
20:59:21 01/27/2020 [bash: zef install .] ===> Installing: Teddy Bear:ver<0.0.1>

Conclusion

I would like to have a feedback from Raku modules authors if this feature is useful. Sparrow6 DSL allows to write scenarios of any complexity and if it still is not enough one could add custom tasks written on many languages to extend the functionality:

cat .rakudist/tasks/hello-world/task.bash

# your Bash code here

And then just:

cat .rakudist/sparrowfile

task-run "tasks/hello-world";

 

RakuDist Update. Rakudo Versions Support.

I continue to post updates on RakuDist project aiming to help Raku modules authors to test their distributions on different environments.

Here is very quick ( I lack of time due to my main $work ) but exciting update. Now RakuDist supports  various Rakudo versions.

Long story short

Yes, you can now test your shiny Raku  module install against arbitrary Rakudo version. Now RakuDist expose simple ( don’t blame me if the server will blow under a load 😉 ) yet handy API to test distribution online:

curl -d os=debian -d rakudo_version=SHA $api_url/rakudist/api/run/$module

For example to test Tomty module against rakudo version 40b13322c503808235d9fec782d3767eb8edb899 on Debian say this:

curl -d os=debian \
-d rakudo_version=40b13322c503808235d9fec782d3767eb8edb899 \
http://repo.westus.cloudapp.azure.com/rakudist/api/run/Tomty

 

Low level API

If for some reasons you don’t want to use public API, you can run RakuDist test scenarios on on-premise infrastructure. All you need is a docker container!

Following a simple workflow:

Install Sparrowdo

zef install --/test Sparrowdo

Pull docker images

docker pull debian

Run container

docker run -d -t --rm --name debian-rakudist debian

Write test

Test scenarios are written on Sparrow6 DSL

cat sparrowfile

zef "https://github.com/melezhik/sparrow6-rakudo-install.git", %( 
  force => True 
);

user "test123";

my $path = module-run 'Rakudo::Install', %(
  user => 'test123',
  rakudo-version => '40b13322c503808235d9fec782d3767eb8edb899'
);

# Now you can use installed Rakudo

bash "perl6 --version", %(
  envvars => %( PATH => "{%*ENV<PATH>}:{$path}" ),
  user => "test123",
  debug => True
);

bash "zef install --/test Kind --verbose", %(
  description => "install Kind",
  user => "test123",
  envvars => %( PATH => "{%*ENV<PATH>}:{$path}" ),
);

 

Run test

sparrowdo --bootstrap \
--no_sudo --docker=debian-rakudist \
--repo=http://repo.westus.cloudapp.azure.com

Conclusion

Read more on https://github.com/melezhik/RakuDist

 

RakuDist update

One week ago I started a discussion in Raku community on testing Raku modules suggesting Sparrow as a solution. While the discussion has been being held and various opinions have been being given, I’d like to share my experience and indeed some good results I’ve had with applying Sparrow tool to the task of testing Raku modules.

First results

* It reveals bugs – drforr/raku-OLE-Storage_Lite/issues/1 , FCO/Red/issues/421 , titsuki/raku-Chart-Gnuplot/issues/40 , HomoGlypher/issues/2

* It’s flexible enough to create even complicated tests involving databases and services, see FCO/Red/issues/421

* It’s dead simple.  Taking me around 1-2 minutes to write a test for an average module. See the list of examples

* It’s light. With Alpine OS image weighted 5MB and very small Sparrow6 dependency tree, the overall test of average module would take no more 2-3 minutes. If we run on already bootstrapped instance – it’s even a seconds!

* It’s has clear and understandable syntax. Sparrow6 DSL is dead simple and easy to read and write, just see a following example

Let’s get started

The main workflow is dead easy. Spin up a docker instance, install Sparrowdo, drop a simple scenario and give it a run! Boom, within a couple of minutes you’ll get your report nice and readable. No need to wait in queue in Travis, it’s cheap, it’s easy and it’s yours!

Spin a docker instance.

Choose an alpine OS image, 5 MB. You pull it and run it with literally seconds!

docker pull alpine
docker run -d -t --rm --name alpine-rakudist alpine

 

Bootstrap docker instance

Install Sparrowdo – a thin client to run Sparrow6 tasks on docker instance. It’s really light, just a few Raku modules as dependencies. Then bootstrap your docker instance with Rakudo and Sparrow6 environment, the last one is also extremely light and it won’t take more then a couple of minutes for the whole process and should be done only once.

zef install --/test Sparrowdo
sparrowdo --bootstrap --no_sudo --docker=alpine-rakudist

 

Create your test

Sparrow6 DSL has all you need. Nice wrappers around zef installer, user management to create users if you need to test users installations and git-scm wrapper to fetch a module source code from Git repositories.

The simplest scenario would be something like that:

$ cat sparrowfile:

my $user = config()<user>;

my $directory = "/data/test/{$user}";

my $scm = config()<scm>;

user $user;

directory $directory, %( 
  owner => $user,
  group => $user

);

git-scm $scm, %( 
  to => $directory, 
  user => $user,
);

if os() eq 'alpine' {

  # this is needed for alpine rakudo installation
  unless "/bin/zef".IO ~~ :e {
    copy "/opt/rakudo-pkg/share/perl6/core/bin/zef", "/bin/zef"
  }

  # this is needed for alpine rakudo installation
  unless "/bin/perl6".IO ~~ :e {
    copy "/opt/rakudo-pkg/bin/perl6", "/bin/perl6"
  }

}

zef $directory, %( 
  force => False,
  depsonly => True, 
  user => $user,
  description => "Install module dependencies"
);

bash "cd {$directory} && zef test .", %(
  description => "zef test",
  user => $user
);

 

The scenario takes some parameters from config.pl6 , e.g. source code repository url and user name to run tests under:

%(
  user => "kind",
  scm =>  "https://github.com/Kaiepi/p6-Kind.git"
)

 

Users installations allows to reuse the same docker instance to test various modules. Because dependencies get installed under user’s account and never clash with other modules. Another option would be – relaunch a new docker instance for every test and start from the scratch but this will add time for initialization phase.

Run scenario

I promised it’d be easy, so just launch our  new shiny test on docker alpine instance:

sparrowdo --no_sudo --docker=alpine-rakudist --repo=http://repo.westus.cloudapp.azure.com

And here the report:

16:50:02 01/09/2020 [repository] index updated from http://repo.westus.cloudapp.azure.com/api/v1/index
16:50:07 01/09/2020 [create user kind] uid=1006(kind) gid=1006(kind) groups=1006(kind)
16:50:07 01/09/2020 [create user kind] user kind created
[task check] stdout match <created> True
16:50:11 01/09/2020 [create directory /data/test/kind] directory path: /data/test/kind
16:50:11 01/09/2020 [create directory /data/test/kind] directory owner: <kind>
16:50:11 01/09/2020 [create directory /data/test/kind] directory group: <kind>
16:50:11 01/09/2020 [create directory /data/test/kind] directory access rights: drwxr-xr-x
[task check] stdout match <owner: <kind>> True
[task check] stdout match <group: <kind>> True
16:50:15 01/09/2020 [bash: git checkout https://github.com/Kaiepi/p6-Kind.git] /data/test/kind
16:50:15 01/09/2020 [bash: git checkout https://github.com/Kaiepi/p6-Kind.git] stderr: Cloning into '.'...
16:50:17 01/09/2020 [bash: last commit] commit c08acf1e4a52491a3b09e19e129efc3092cef745
16:50:17 01/09/2020 [bash: last commit] Author: Ben Davies <kaiepi@outlook.com>
16:50:17 01/09/2020 [bash: last commit] Date:   Tue Dec 24 16:28:29 2019 -0400
16:50:17 01/09/2020 [bash: last commit]
16:50:17 01/09/2020 [bash: last commit]     Release v0.1.0
16:50:17 01/09/2020 [bash: last commit]
16:50:17 01/09/2020 [bash: last commit] M       CHANGELOG
16:50:17 01/09/2020 [bash: last commit] M       META6.json
16:50:17 01/09/2020 [bash: last commit] M       lib/Kind.pm6
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] total 32
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] -rw-r--r--. 1 kind kind  191 Jan  9 16:50 CHANGELOG
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] -rw-r--r--. 1 kind kind 8902 Jan  9 16:50 LICENSE
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] -rw-r--r--. 1 kind kind  408 Jan  9 16:50 META6.json
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] -rw-r--r--. 1 kind kind 3230 Jan  9 16:50 README.md
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] -rw-r--r--. 1 kind kind 3337 Jan  9 16:50 README.pod6
16:50:19 01/09/2020 [bash: cd /data/test/kind && ls -l] -rw-r--r--. 1 kind kind  114 Jan  9 16:50 dist.ini
16:50:20 01/09/2020 [bash: cd /data/test/kind && ls -l] drwxr-xr-x. 2 kind kind   22 Jan  9 16:50 lib
16:50:20 01/09/2020 [bash: cd /data/test/kind && ls -l] drwxr-xr-x. 2 kind kind   23 Jan  9 16:50 t
16:50:24 01/09/2020 [bash: Install module dependencies] stderr: All candidates are currently installed
16:50:24 01/09/2020 [bash: Install module dependencies] <empty stdout>
16:50:27 01/09/2020 [bash: zef test] ===> Testing: Kind:ver<0.1.0>
16:50:29 01/09/2020 [bash: zef test] [Kind] t/01-kind.t .. ok
16:50:29 01/09/2020 [bash: zef test] [Kind] All tests successful.
16:50:29 01/09/2020 [bash: zef test] [Kind] Files=1, Tests=3,  2 wallclock secs ( 0.02 usr  0.00 sys +  2.68 cusr  0.21 csys =  2.91 CPU)
16:50:29 01/09/2020 [bash: zef test] [Kind] Result: PASS
16:50:29 01/09/2020 [bash: zef test] ===> Testing [OK] for Kind:ver<0.1.0>

 

Conclusion

Sparrow could be a reasonable alternative to existing CI tools allowing test Raku distributions easy and fast.  Give it a try and say you comments! Follow the RakuDist project for examples of test scenarios and further updates.

 

Sparrowdo on Docker

With the latest Sparrowdo commits it’s now possible to run Sparrowdo tasks on running Docker containers:

Spin up a Docker container:

$ docker pull bitnami/minideb-extras # Debian minimal
$ docker run -d instance0 -it minideb-extras bash

Create some Sparrowdo scenario:

$ cat sparrowfile
use v6;
use Sparrowdo;
task-run 'check disk available space', 'df-check', %( threshold => 80 );
bash 'pwd';

Ran Sparrowdo scenario on running Docker container:

$ sparrowdo --docker=instance0 --bootstrap --no_sudo --format=production

Output:

running sparrow tasks on 127.0.0.1 ... 
target OS is - ubuntu
push [task] check disk available space [plg] df-check OK
push [task] run bash: pwd ... OK
SPL file /opt/sparrow/sparrow.list is empty
get index updates from SparrowHub ... OK
set up task box file - /home/melezhik/.sparrowdo//opt/sparrow/task-box.json - OK
public@df-check is uptodate (0.2.3)
public@bash is uptodate (0.1.6)
running task box from /opt/sparrow/sparrow-cache/task-box.json ... 
2017-09-22 11:12:39 : [task] check disk available space [plg] df-check [path] /
threshhold: 80
2017-09-22 11:12:39 : [task] run bash: pwd ... [path] modules/bash-command/ [params] envvars:

 
Caveats

You should have bash and curl pre installed at your Docker container.

Minoca OS automation with Sparrowdo

Introduction

Hello! Minoca is a new operating system for the world of connected devices. In this post I am going to show you how one can enable configuration management of running Minoca instances by the help of Sparrowdo.

Download the latest Minoca build

$ wget http://www.minocacorp.com/download/nightlies/latest-x86/Minoca-pc.zip
$ unzip Minoca-pc.zip

Start Minoca OS instance

$ qemu-system-x86_64 -enable-kvm -m 2000 -net nic,model=i82559er -net user,hostfwd=tcp::2222-:22,hostfwd=tcp::8888-:80 -hda pc.img

Set up remote ssh access

Inside running Minoca instance:

$ opkg update 
$ opkg install openssh bash
$ /etc/init.d/sshd start
$ passwd # we will use this password when ssh-ing from the Host OS after `ssh-copy-id` gets run

Inside Host OS:

$ ssh-copy-id -p 2222 root@127.0.0.1
$ ssh -p 2222 root@127.0.0.1
$ exit

Create some Sparrowdo scenario

$ nano sparrowfile
use v6;
use Sparrowdo;
package-install ("nano", "zsh", "nginx");
user "alexey";
directory "/var/data/bar", %( owner => "alexey");
service-stop "nginx";
service-start "nginx";
http-ok %( port => 8888 );

Run Sparrowdo scenario for Minoca instance

$ sparrowdo --host=127.0.0.1 --ssh_user=root --ssh_port=2222 --no_sudo --sparrowfile=sparrowfile --bootstrap --format=production

The output:

running sparrow bootstrap for host: 127.0.0.1 ... 
bootstrap for minoca
Downloading http://www.minocacorp.com/packages/0.4/i686/main/Packages.gz.
Inflating http://www.minocacorp.com/packages/0.4/i686/main/Packages.gz.
Updated list of available packages in /var/opkg-lists/main.
/usr/bin/curl
/usr/bin/perl
/usr/bin/cpanm
/usr/bin/sparrow
Outthentic is up to date. (0.3.9)
Sparrow is up to date. (0.2.48)
running sparrow tasks on 127.0.0.1 ... 
target OS is - minoca
push [task] install packages: nano zsh nginx OK
push [task] create user alexey OK
push [task] create directory /var/data/bar OK
push [task] stop service nginx OK
push [task] start service nginx OK
push [task] run bash: curl -fsSLk -D - --retry 3 127.0.0.1:8888 -o /dev/ ... OK
SPL file /opt/sparrow/sparrow.list is empty
get index updates from SparrowHub ... OK
set up task box file - /home/melezhik/.sparrowdo//opt/sparrow/task-box.json - OK
public@package-generic is uptodate (0.3.7)
public@user is uptodate (0.2.1)
public@directory is uptodate (0.1.4)
public@service is uptodate (0.1.13)
public@bash is uptodate (0.1.6)
running task box from /opt/sparrow/sparrow-cache/task-box.json ... 
2017-09-21 02:27:12 : [task] install packages: nano zsh nginx [path] modules/opkg/ [params] action:install package:nano
2017-09-21 02:27:12 : [task] install packages: nano zsh nginx [path] modules/opkg/ [params] action:install package:zsh
2017-09-21 02:27:12 : [task] install packages: nano zsh nginx [path] modules/opkg/ [params] action:install package:nginx
2017-09-21 02:27:13 : [task] create user alexey [path] modules/create/
2017-09-21 02:27:13 : [task] create directory /var/data/bar [path] modules/create/
2017-09-21 02:27:14 : [task] stop service nginx [path] modules/stop/ [params] os:minoca service:nginx
2017-09-21 02:27:14 : [task] start service nginx [path] modules/start/ [params] os:minoca service:nginx
2017-09-21 02:27:14 : [task] run bash: curl -fsSLk -D - --retry 3 127.0.0.1:8888 -o /dev/ ... [path] modules/bash-command/ [params] envvars:

Building Perl6 Applications with Docker and Ducky

Docker containers allow developers run environments and deploy application easy and fast. Dockerfile/Ansible/Chef are means you can configure bootstrapped Docker instances, however there is another way to do this …

Ducky is a lightweight Docker provision tool allow easy deploy Docker containers in just creating JSON scenarios in declarative way:

$ cat ducky.json
[
 {
   "task" : "install perl6",
   "plugin" : "rakudo-install",
   "data" : {
   "url" : "https://github.com/nxadm/rakudo-pkg/releases/download/2017.07/perl6-rakudo-moarvm-CentOS7.3.1611-20170700-01.x86_64.rpm"
  }
 }
]

This is how we bootstrap Rakudo on Docker box by using Ducky and this simple scenario.  Now let’s pull CentOS image and run Docker container based on it:

$ docker pull centos
$ docker run -d -i -t -v $PWD:/var/ducky --name ducky-centos centos

The only requirement here is that the running Docker container should have current working directory, ( which holds Ducky json file ) mounted as /var/ducky.

Ducky picks up ducky.json file placed in the current working directory and executes scenario on running Docker container, named centos-ducky:

$ ducky.bash ducky-centos

Here is a piece of screen shot of Ducky’s output ( only the last lines are shown for the sake of brevity ):

Under the hood Ducky installs Sparrow client on the container and then give it a run  to execute tasks defined at Ducky json file.  The tasks are described at  Sparrow task box format.

So Ducky json is just a Sparrow Task Box file. That means you can declare Sparrow plugins with parameters here, aka Sparrow tasks which are executed on a Docker container. Available plugins are listed, documented and stored at the SparrowHub – Sparrow plugins repository.

In this scenario we use rakudo-install plugin to install Rakudo as system pcakge. The plugin documentation is available at the SparrowHub site.

Thus there are more things you could do with Ducky not only installing software,  you’re only limited by existed Sparrow plugins.

A typical use case is to run Test::Harness against a Perl6 project. Let’s do it for  Bailador  which is “A light-weight route-based web application framework for Perl 6” :

$ git clone https://github.com/Bailador/Bailador.git
$ cd Bailador
$ cat ducky.json 

[

  {
    "task" : "install perl6",
    "plugin" : "rakudo-install",
    "data" : {
      "url" : "https://github.com/nxadm/rakudo-pkg/releases/download/2017.07/perl6-rakudo-moarvm-CentOS7.3.1611-20170700-01.x86_64.rpm"
    }
  },
  {
    "task" : "installs Bailador dependencies",
    "plugin" : "zef",
    "data" : {
        "list" : [ "." ],
        "options" : "--deps-only"
    }
  },
  {
    "task" : "run t/ tests",
    "plugin" : "bash",
    "data" : {
        "command" : "prove6 -l",
        "envvars" : {
          "PATH" : "/opt/rakudo/bin:/opt/rakudo/share/perl6/site/bin:/root/.rakudobrew/moar-nom/install/share/perl6/site/bin:$PATH"
        }
    }
  }
]

Ducky json is quite self illustrative, here we define some standard steps to build and test the project:

* Install Raudo
* Install Bailador dependencies  picked from META6 file
* Runs t/ tests with prove6

Ok, let’s give it a run: ( don’t forget that we should first launch Docker container with the current working directory mounted as /var/ducky ):

$ docker run -d -i -t -v $PWD:/var/ducky --name ducky-bailador centos
$ ducky.bash ducky-bailador

Here is the last lines of the Ducky output:

In this scenario we use 2 other plugins – bash – to execute arbitrary Bash code and zef – simple wrapper for Zef manager – tool to install Perl6 modules. The plugins documentation is available at the SparrowHub site.

Further thoughts.

Ducky and Sparrow are kind of cross platform tools, meaning you can successfully run the same scenarios on the variety of Linux platforms ( provided that Bash is installed ), for example the last scenario will succeed when running against Alpine Linux   docker image:

$ docker pull melezhik/alpine-perl6
$ docker run -d -i -t -v $PWD:/var/ducky --name ducky-bailador-alpine melezhik/alpine-perl6
$ ducky.bash ducky-bailador-alpine

Thus it becomes extremely  useful when you want to test a project against different environments just sitting at your developer box and running chip docker containers.

And the last but not the least. If for some reasons you’re not satisfied by existed Sparrow plugins you can easy write new one to cover your needs. I have written a plenty of posts of how to do this and you yon may start with this one – Outthentic – quick way to develop user’s scenarios .

Regards and have a fun with your coding and automation.

How to use Chef and Sparrowdo together

Good team member.

Chef is a well recognized configuration management tool which I use extensively at my current work. However I keep pushing to Sparrowdo – Perl6 configuration management tool and find those two tools play nicely together.

In this post I am going to give a few examples on how I use Sparrowdo to simplify and improve Chef cookbooks development workflow.

Running chef client on a target host.

Here is the most useful scenario how I use Chef and Sparrowdo together. My working environment implies launching ec2 Amazon instances get configured by chef. Instead of ssh-ing to an instance and running a chef-client on it, I delegate this task to Sparrowdo using wrapper called Sparrowdo::Chef::Client.

Let’s install the module:

$ zef install Sparrowdo::Chef::Client

And then create a simple Sparrowdo scenario:

$ cat sparrowfile
module_run 'Chef::Client', %(
    run-list => [
      "recipe[foo]",
      "recipe[baz]"
    ],
    log-level => 'info',
    force-formatter => True
);

Here we’re just running two recipes called foo and bar. And define some chef client’s settings, like log level and enabling force-formater option. Now we can run a chef-client on a target host:

$ sparrowdo --host=$target_host

Post deployments checks.

It is always good idea to check a server’s state right after the deployment. There are reasons why I prefer to not keep such a checks inside my Chef scenarios. And it seems there is a trend which is seen as new monitor and audit tools appear at the open source market with InSpec and goss among them, to list a few.

Likewise Sparrowdo has some built-in facilities to quickly test an infrastructure.

Let me give you a few examples.

Check system processes.

Say, we reconfigure an Nginx server by using some Chef recipes, sometimes Chef is not able to ensure that Nginx starts successfully after deploy or even if it does I don’t want to grep huge chef client logs ( sometimes there is a load of them ) to find out whether an Nginx gets started successfully. Happily, here is the dead simple solution – the usage of Sparrowdo asserts:

$ cat healthcheck.pl6
proc-exists 'nginx';

This function checks that file /var/run/nginx.pid exists as well as the related process with the PID taken from the file does. If you need to handle uncommon file paths for pid files, you can always set the path explicitly:

$ cat healthcheck.pl6
proc-exists-by-pid 'nginx server', '/var/run/nginx/ngix.pid';

Moreover if you only know the “name” of the process ( well technically specking this regular expression to match the process command ), simply have this:

$ cat healthcheck.pl6
proc-exists-by-footprint 'nginx web server', 'nginx\s+master';

Having this simple Sparrowdo scenario just run it against a target server to check that Nginx process exists:

$ sparrowdo --host=$target_host --sparrowfile=healthcheck.pl6

I always put Sparrowdo scenarios and Chef cookbook files together and commit them to Git repository:

$ git add sparrowfile healthcheck.pl6
$ git commit -a -m 'sparrowdo scenarios for chef cookbook'

And finally let me give an example of checking web application endpoints by sending http requests. Say, we have Chef recipe which deploys an application that should be accessible by http GET / request. Sparrowdo exposes handy http-ok asserts to deal with such a checks:

$ cat healthcheck.pl6
http-ok;

That is it! This is the simplest form of http-ok function’s call to verify that the web application is responsible by  accepting  requests for GET / route. Under the hood it just:

1. resolves hostname as those one you run sparrowdo
2. issues http request using curl utility:

$ curl -f http://$target_host

There are options how you can call http-ok function. For example, you may define endpoints and set http port:

$ cat healthcheck.pl6
http-ok(port  => '8080' , path => '/Foo/Bar' );

Follow Sparrowdo documentation for full description of http asserts function.

Conclusion

Using Sparrowdo and Chef together could be efficient approach when developing and testing server’s configuration scenarios. Sparrowdo is proven to be able to adopt any requirements and play nicely with other ecosystems and frameworks being an intelligent “clue” to bind together various tools and get your work done.