RakuOps. Issue 1.

RakuOps series – an attempt to show people who write on Raku how to use the language in daily DevOps tasks – automation, configuration management, Docker containers building and so on.

While I don’t know for sure which topics will attract the community interest, I hope that during this series I’ll get some feedback so I could adjust my future posts according actual people needs.

How to Build Docker Containers Using Raku and Sparrow

This is the first post in the series, where I am going to show how to use Raku and Sparrow – Raku automation framework to build Docker images. We will start with simple Dockerfile example and then we’ll see how to use Sparrow to extend image building process.

Why

People usually use Dockerfile DSL to build Docker images. However the usage of Docker file is limited and quickly get cumbersome when it comes to more sophisticated cases. User ends up in extensive shell scripting through various RUN commands or similar way, which is very hard to maintain in the long run.

Moreover if one choose to change an underlying Docker container’s OS they will have to rewrite all the code which often has distro specific RUN commands.

In this post we will see how to use Raku and all battery included Sparrow automation tool to create Docker build scenarios in more portable and easy to maintain way.

As a result one could start using Raku to create high level scenarios gaining an access to all the power of the language. As well as a plenty of Sparrow plugins would reduce efforts to write code when dealing with typical configuration tasks – installing native packages, users, configuration files and so on.

Prerequisites

To build Docker container we will need a following set of tools:

• Rakudo
• Sparrow
• Git
• Docker

Rakudo installation is pretty strait-forward, just follow the instructions on https://rakudo.org/downloads web site.

To install Sparrow toolkit, we need install Sparrow6 Raku module:

zef install --/test Sparrow6

Sparrow bootstrap

To bootstrap Sparrow on Docker instance we need to build a Docker image first. That image should include Rakudo and Sparrow binaries. Thanks to @jjmerelo there is a
jjmelerelo/alpine-raku base Docker image with Alpine Linux with Rakudo binary pre-installed, so our Dockerfile should be pretty simple:

$ mkdir -p RakuOps/docker-sparrow
$ cd RakuOps/docker-sparrow

$ cat Dockerfile

FROM jjmerelo/alpine-raku
RUN zef install --/test Sparrow6

$ docker build --tag rakuops:1.0 .

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM jjmerelo/alpine-raku
latest: Pulling from jjmerelo/alpine-raku
df20fa9351a1: Already exists
a901eee946d8: Pull complete
Digest: sha256:3e22846977d60ccbe2d06a47da4a5e78c6aca7af395d57873d3a907bea811838
Status: Downloaded newer image for jjmerelo/alpine-raku:latest
 ---> c0ecb08ec5db
Step 2/2 : RUN zef install --/test Sparrow6
 ---> Running in ae2a0dc8848f
===> Searching for: Sparrow6
===> Updating cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Searching for missing dependencies: File::Directory::Tree, Hash::Merge, YAMLish, JSON::Tiny, Data::Dump
===> Searching for missing dependencies: MIME::Base64
===> Installing: File::Directory::Tree:auth<labster>
===> Installing: Hash::Merge:ver<1.0.1>:auth<github:scriptkitties>:api<1>
===> Installing: MIME::Base64:ver<1.2.1>:auth<github:retupmoca>
===> Installing: YAMLish:ver<0.0.5>
===> Installing: JSON::Tiny:ver<1.0>
===> Installing: Data::Dump:ver<v.0.0.11>:auth<github:tony-o>
===> Installing: Sparrow6:ver<0.0.24>

1 bin/ script [s6] installed to:
/root/raku-install/share/perl6/site/bin
===> Updated cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Updating p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
Removing intermediate container ae2a0dc8848f
 ---> a2cbc605ec5e
Successfully built a2cbc605ec5e
Successfully tagged rakuops:1.0

$ docker images

REPOSITORY             TAG                 IMAGE ID            CREATED             SIZE
rakuops                1.0                 a2cbc605ec5e        3 minutes ago       139MB

First run

Now having a base Docker image let’s run our very first Sparrow scenario, all we need is to add file called sparrowfile using Docker ADD directive. Our first scenario will be as simple as Bash “Hello World” echo command:

$ cat sparrowfile

bash "echo 'Hello World'", %(
    description => "hello world"
);

As one could notice, Sparrow scenario is just a plain Raku code with some DSL constructions. Let’s modify Dockerfile and rebuild an image.

$ cat Dockerfile

ADD sparrowfile
RUN raku -MSparrow6::DSL sparrowfile

$ docker build --tag rakuops:1.0 .

Sending build context to Docker daemon  5.632kB
Step 1/4 : FROM jjmerelo/alpine-raku
 ---> c0ecb08ec5db
Step 2/4 : RUN zef install --/test Sparrow6
 ---> Using cache
 ---> a2cbc605ec5e
Step 3/4 : ADD sparrowfile .
 ---> 74c7ee71a303
Step 4/4 : RUN raku -MSparrow6::DSL sparrowfile
 ---> Running in c73e1a7d568f
unknown plugin bash
  in method plugin-install at /root/raku-install/share/perl6/site/sources/5D155994EC979DF8EF1FDED7148646312D9073E3 (Sparrow6::Task::Repository::Helpers::Plugin) line 115
  in sub task-run at /root/raku-install/share/perl6/site/sources/DB0BB8A1D70970E848E2F38D2FC0C39E4F904283 (Sparrow6::DSL::Common) line 12
  in sub bash at /root/raku-install/share/perl6/site/sources/7662EE0EFF4206F474B7CC4AEF229F1A86EC8FFF (Sparrow6::DSL::Bash) line 33
  in sub bash at /root/raku-install/share/perl6/site/sources/7662EE0EFF4206F474B7CC4AEF229F1A86EC8FFF (Sparrow6::DSL::Bash) line 7
  in block <unit> at sparrowfile line 1

The very first run has failed with unknown plugin bash error, that means one needs to provision Docker with Sparrow repository – a storage for all dependencies required in Sparrow scenarios.

While there are many ways to do that, for our tutorial use of local file repository seems the easiest one.

Local Sparrow repository

Local Sparrow repository contains all Sparrow plugins, deployed to your local file system. To create one we need to initialize a repository structure first:

$ s6 --repo-init ~/repo

16:41:31 06/29/2020 [repository] repo initialization
16:41:31 06/29/2020 [repository] initialize Sparrow6 repository for /home/scheck/repo

When we have an empty repository let’s populate it with
Sparrow plugins taken from source code . Right now we only need a specific bash plugin, so let’s upload on this one:

$ git clone https://github.com/melezhik/sparrow-plugins ~/sparrow-plugins

$ cd ~/sparrow-plugins/bash

$ s6 --upload
16:41:36 06/29/2020 [repository] upload plugin
16:41:36 06/29/2020 [repository] upload bash@0.2.1

Copy repository to Docker cache

We’re going to use Docker COPY command to copy repository files to a Docker cache. But first we need to copy files to the current working directory so they will be available for the COPY command during Docker build:

$ cp -r ~/repo .

$ cat Dockerfile

RUN apk add bash perl
COPY repo/ /root/repo/
RUN s6 --index-update
RUN raku -MSparrow6::DSL sparrowfile

$ docker build --tag rakuops:1.0 .

Sending build context to Docker daemon  11.26kB
Step 1/7 : FROM jjmerelo/alpine-raku
 ---> c0ecb08ec5db
Step 2/7 : RUN zef install --/test Sparrow6
 ---> Using cache
 ---> a2cbc605ec5e
Step 3/7 : RUN apk add bash perl
 ---> Using cache
 ---> d9011d4e64db
Step 4/7 : ADD sparrowfile .
 ---> Using cache
 ---> adb1df57e1c0
Step 5/7 : COPY repo/ /root/repo/
 ---> Using cache
 ---> 3ed6bfaf4183
Step 6/7 : RUN s6 --index-update
 ---> Running in 6edfc480bde7
17:03:59 06/29/2020 [repository] update local index
17:03:59 06/29/2020 [repository] index updated from file:///root/repo/api/v1/index
Removing intermediate container 6edfc480bde7
 ---> 7eccb5889a80
Step 7/7 : RUN raku -MSparrow6::DSL sparrowfile
 ---> Running in af6eb4b2d9ee
17:04:02 06/29/2020 [repository] installing bash, version 0.002001
17:04:05 06/29/2020 [bash: echo Hello World] Hello World

As we could see from the log, Sparrow scenario successfully finishes printing “Hello World” in stdout. Line installing bash, version 0.002001 means Sparrow plugin has been successfully pulled from Docker cache and installed into container file system.

Build all plugins

To use the rest of Sparrow plugins in Docker build scenarios we need to add the to Docker cache the same way we did for bash plugin:

$ cd ~/sparrow-plugins
$ find  -maxdepth 2 -mindepth 2 -name sparrow.json -execdir s6 --upload \;
17:11:56 06/29/2020 [repository] upload plugin
17:11:56 06/29/2020 [repository] upload ado-read-variable-groups@0.0.1
17:11:56 06/29/2020 [repository] upload plugin
17:11:56 06/29/2020 [repository] upload ambari-hosts@0.0.1
17:11:57 06/29/2020 [repository] upload plugin
17:11:57 06/29/2020 [repository] upload ansible-install@0.0.2
17:11:58 06/29/2020 [repository] upload plugin
17:11:58 06/29/2020 [repository] upload ansible-tutorial@0.0.1
17:11:59 06/29/2020 [repository] upload plugin
17:11:59 06/29/2020 [repository] upload app-cpm-wrapper@0.0.6
... output truncated ...

Now let’s update Docker cache by copy repository file to current working directory, in the next run Docker COPY command will pick files and push to Docker image.

$ cd ~/RakuOps/docker-sparrow/
$ cp -r ~/repo .

Sparrow plugins

Now we’re free to use any plugin we’ve just added. Say, we need to install nano editor on our Docker image. Sparrow provides a cross-platform package-generic plugin to install native packages:

$ cat sparrowfile

package-install "nano";

$ docker build --tag rakuops:1.0 .

Sending build context to Docker daemon  2.012MB
Step 1/7 : FROM jjmerelo/alpine-raku
 ---> c0ecb08ec5db
Step 2/7 : RUN zef install --/test Sparrow6
 ---> Using cache
 ---> a2cbc605ec5e
Step 3/7 : RUN apk add bash perl
 ---> Using cache
 ---> d9011d4e64db
Step 4/7 : ADD sparrowfile .
 ---> 7a3bb7329d46
Step 5/7 : COPY repo/ /root/repo/
 ---> 0c029612c55c
Step 6/7 : RUN s6 --index-update
 ---> Running in 356d29ed8049
17:16:56 06/29/2020 [repository] update local index
17:16:56 06/29/2020 [repository] index updated from file:///root/repo/api/v1/index
Removing intermediate container 356d29ed8049
 ---> 18876a3d6396
Step 7/7 : RUN raku -MSparrow6::DSL sparrowfile
 ---> Running in bd07fecae4f0
17:16:58 06/29/2020 [repository] installing bash, version 0.002001
17:17:00 06/29/2020 [bash: echo Hello World] Hello World
17:17:00 06/29/2020 [repository] installing package-generic, version 0.004001
17:17:02 06/29/2020 [install package(s): nano.perl] fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
17:17:02 06/29/2020 [install package(s): nano.perl] fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
17:17:02 06/29/2020 [install package(s): nano.perl] v3.12.0-103-g1699efe1cd [http://dl-cdn.alpinelinux.org/alpine/v3.12/main]
17:17:02 06/29/2020 [install package(s): nano.perl] v3.12.0-106-g2b11e345c6 [http://dl-cdn.alpinelinux.org/alpine/v3.12/community]
17:17:02 06/29/2020 [install package(s): nano.perl] OK: 12730 distinct packages available
17:17:03 06/29/2020 [install package(s): nano.perl] trying to install nano ...
17:17:03 06/29/2020 [install package(s): nano.perl] installer - apk
17:17:03 06/29/2020 [install package(s): nano.perl] (1/2) Installing libmagic (5.38-r0)
17:17:03 06/29/2020 [install package(s): nano.perl] (2/2) Installing nano (4.9.3-r0)
17:17:03 06/29/2020 [install package(s): nano.perl] Executing busybox-1.31.1-r19.trigger
17:17:03 06/29/2020 [install package(s): nano.perl] OK: 67 MiB in 32 packages
17:17:03 06/29/2020 [install package(s): nano.perl] Installed:                                Available:
17:17:03 06/29/2020 [install package(s): nano.perl] nano-4.9.3-r0                           = 4.9.3-r0
17:17:03 06/29/2020 [install package(s): nano.perl] nano
Removing intermediate container bd07fecae4f0
 ---> 408d35e1e3fd
Successfully built 408d35e1e3fd
Successfully tagged rakuops:1.0

Conclusion

We’ve just seen how one can use Raku and Sparrow to build Docker images. The advantage of the approach one is no more limited by Dockerfile syntax and could leverage all the power of Raku to express any sophisticated build logic. On other hand Sparrow provides a lot of handy primitives and plugins for typical build tasks and some of them I’m going to share in next posts.

Advertisement