RakuOps series – an attempt to show people who write on Raku how to use the language in daily DevOps tasks – automation, configuration management, Docker containers building and so on.
While I don’t know for sure which topics will attract the community interest, I hope that during this series I’ll get some feedback so I could adjust my future posts according actual people needs.
How to Build Docker Containers Using Raku and Sparrow
This is the first post in the series, where I am going to show how to use Raku and Sparrow – Raku automation framework to build Docker images. We will start with simple Dockerfile example and then we’ll see how to use Sparrow to extend image building process.
Why
People usually use Dockerfile DSL to build Docker images. However the usage of Docker file is limited and quickly get cumbersome when it comes to more sophisticated cases. User ends up in extensive shell scripting through various RUN commands or similar way, which is very hard to maintain in the long run.
Moreover if one choose to change an underlying Docker container’s OS they will have to rewrite all the code which often has distro specific RUN commands.
In this post we will see how to use Raku and all battery included Sparrow automation tool to create Docker build scenarios in more portable and easy to maintain way.
As a result one could start using Raku to create high level scenarios gaining an access to all the power of the language. As well as a plenty of Sparrow plugins would reduce efforts to write code when dealing with typical configuration tasks – installing native packages, users, configuration files and so on.
Prerequisites
To build Docker container we will need a following set of tools:
- Rakudo
- Sparrow
- Git
- Docker
Rakudo installation is pretty strait-forward, just follow the instructions on https://rakudo.org/downloads web site.
To install Sparrow toolkit, we need install Sparrow6 Raku module:zef install --/test Sparrow6
Sparrow bootstrap
To bootstrap Sparrow on Docker instance we need to build a Docker image first. That image should include Rakudo and Sparrow binaries. Thanks to @jjmerelo there is ajjmelerelo/alpine-raku base Docker image with Alpine Linux with Rakudo binary pre-installed, so our Dockerfile should be pretty simple:
$ mkdir -p RakuOps/docker-sparrow $ cd RakuOps/docker-sparrow
$ cat Dockerfile
FROM jjmerelo/alpine-raku RUN zef install --/test Sparrow6
$ docker build --tag rakuops:1.0 .
Sending build context to Docker daemon 2.048kB Step 1/2 : FROM jjmerelo/alpine-raku latest: Pulling from jjmerelo/alpine-raku df20fa9351a1: Already exists a901eee946d8: Pull complete Digest: sha256:3e22846977d60ccbe2d06a47da4a5e78c6aca7af395d57873d3a907bea811838 Status: Downloaded newer image for jjmerelo/alpine-raku:latest ---> c0ecb08ec5db Step 2/2 : RUN zef install --/test Sparrow6 ---> Running in ae2a0dc8848f ===> Searching for: Sparrow6 ===> Updating cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json ===> Searching for missing dependencies: File::Directory::Tree, Hash::Merge, YAMLish, JSON::Tiny, Data::Dump ===> Searching for missing dependencies: MIME::Base64 ===> Installing: File::Directory::Tree:auth<labster> ===> Installing: Hash::Merge:ver<1.0.1>:auth<github:scriptkitties>:api<1> ===> Installing: MIME::Base64:ver<1.2.1>:auth<github:retupmoca> ===> Installing: YAMLish:ver<0.0.5> ===> Installing: JSON::Tiny:ver<1.0> ===> Installing: Data::Dump:ver<v.0.0.11>:auth<github:tony-o> ===> Installing: Sparrow6:ver<0.0.24> 1 bin/ script [s6] installed to: /root/raku-install/share/perl6/site/bin ===> Updated cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json ===> Updating p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json ===> Updated p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json Removing intermediate container ae2a0dc8848f ---> a2cbc605ec5e Successfully built a2cbc605ec5e Successfully tagged rakuops:1.0
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE rakuops 1.0 a2cbc605ec5e 3 minutes ago 139MB
First run
Now having a base Docker image let’s run our very first Sparrow scenario, all we need is to add file called sparrowfile using Docker ADD directive. Our first scenario will be as simple as Bash “Hello World” echo command:
$ cat sparrowfile
bash "echo 'Hello World'", %(
description => "hello world"
);
As one could notice, Sparrow scenario is just a plain Raku code with some DSL constructions. Let’s modify Dockerfile and rebuild an image.
$ cat Dockerfile
ADD sparrowfile RUN raku -MSparrow6::DSL sparrowfile
$ docker build --tag rakuops:1.0 .
Sending build context to Docker daemon 5.632kB Step 1/4 : FROM jjmerelo/alpine-raku ---> c0ecb08ec5db Step 2/4 : RUN zef install --/test Sparrow6 ---> Using cache ---> a2cbc605ec5e Step 3/4 : ADD sparrowfile . ---> 74c7ee71a303 Step 4/4 : RUN raku -MSparrow6::DSL sparrowfile ---> Running in c73e1a7d568f unknown plugin bash in method plugin-install at /root/raku-install/share/perl6/site/sources/5D155994EC979DF8EF1FDED7148646312D9073E3 (Sparrow6::Task::Repository::Helpers::Plugin) line 115 in sub task-run at /root/raku-install/share/perl6/site/sources/DB0BB8A1D70970E848E2F38D2FC0C39E4F904283 (Sparrow6::DSL::Common) line 12 in sub bash at /root/raku-install/share/perl6/site/sources/7662EE0EFF4206F474B7CC4AEF229F1A86EC8FFF (Sparrow6::DSL::Bash) line 33 in sub bash at /root/raku-install/share/perl6/site/sources/7662EE0EFF4206F474B7CC4AEF229F1A86EC8FFF (Sparrow6::DSL::Bash) line 7 in block <unit> at sparrowfile line 1
The very first run has failed with unknown plugin bash error, that means one needs to provision Docker with Sparrow repository – a storage for all dependencies required in Sparrow scenarios.
While there are many ways to do that, for our tutorial use of local file repository seems the easiest one.
Local Sparrow repository
Local Sparrow repository contains all Sparrow plugins, deployed to your local file system. To create one we need to initialize a repository structure first:
$ s6 --repo-init ~/repo
16:41:31 06/29/2020 [repository] repo initialization 16:41:31 06/29/2020 [repository] initialize Sparrow6 repository for /home/scheck/repo
When we have an empty repository let’s populate it with
Sparrow plugins taken from source code . Right now we only need a specific bash plugin, so let’s upload on this one:
$ git clone https://github.com/melezhik/sparrow-plugins ~/sparrow-plugins $ cd ~/sparrow-plugins/bash $ s6 --upload 16:41:36 06/29/2020 [repository] upload plugin 16:41:36 06/29/2020 [repository] upload bash@0.2.1
Copy repository to Docker cache
We’re going to use Docker COPY command to copy repository files to a Docker cache. But first we need to copy files to the current working directory so they will be available for the COPY command during Docker build:
$ cp -r ~/repo .
$ cat Dockerfile
RUN apk add bash perl COPY repo/ /root/repo/ RUN s6 --index-update RUN raku -MSparrow6::DSL sparrowfile
$ docker build --tag rakuops:1.0 .
Sending build context to Docker daemon 11.26kB Step 1/7 : FROM jjmerelo/alpine-raku ---> c0ecb08ec5db Step 2/7 : RUN zef install --/test Sparrow6 ---> Using cache ---> a2cbc605ec5e Step 3/7 : RUN apk add bash perl ---> Using cache ---> d9011d4e64db Step 4/7 : ADD sparrowfile . ---> Using cache ---> adb1df57e1c0 Step 5/7 : COPY repo/ /root/repo/ ---> Using cache ---> 3ed6bfaf4183 Step 6/7 : RUN s6 --index-update ---> Running in 6edfc480bde7 17:03:59 06/29/2020 [repository] update local index 17:03:59 06/29/2020 [repository] index updated from file:///root/repo/api/v1/index Removing intermediate container 6edfc480bde7 ---> 7eccb5889a80 Step 7/7 : RUN raku -MSparrow6::DSL sparrowfile ---> Running in af6eb4b2d9ee 17:04:02 06/29/2020 [repository] installing bash, version 0.002001 17:04:05 06/29/2020 [bash: echo Hello World] Hello World
As we could see from the log, Sparrow scenario successfully finishes printing “Hello World” in stdout. Line installing bash, version 0.002001 means Sparrow plugin has been successfully pulled from Docker cache and installed into container file system.
Build all plugins
To use the rest of Sparrow plugins in Docker build scenarios we need to add the to Docker cache the same way we did for bash plugin:
$ cd ~/sparrow-plugins $ find -maxdepth 2 -mindepth 2 -name sparrow.json -execdir s6 --upload \; 17:11:56 06/29/2020 [repository] upload plugin 17:11:56 06/29/2020 [repository] upload ado-read-variable-groups@0.0.1 17:11:56 06/29/2020 [repository] upload plugin 17:11:56 06/29/2020 [repository] upload ambari-hosts@0.0.1 17:11:57 06/29/2020 [repository] upload plugin 17:11:57 06/29/2020 [repository] upload ansible-install@0.0.2 17:11:58 06/29/2020 [repository] upload plugin 17:11:58 06/29/2020 [repository] upload ansible-tutorial@0.0.1 17:11:59 06/29/2020 [repository] upload plugin 17:11:59 06/29/2020 [repository] upload app-cpm-wrapper@0.0.6 ... output truncated ...
Now let’s update Docker cache by copy repository file to current working directory, in the next run Docker COPY command will pick files and push to Docker image.
$ cd ~/RakuOps/docker-sparrow/ $ cp -r ~/repo .
Sparrow plugins
Now we’re free to use any plugin we’ve just added. Say, we need to install nano editor on our Docker image. Sparrow provides a cross-platform package-generic plugin to install native packages:
$ cat sparrowfile
package-install "nano";
$ docker build --tag rakuops:1.0 .
Sending build context to Docker daemon 2.012MB Step 1/7 : FROM jjmerelo/alpine-raku ---> c0ecb08ec5db Step 2/7 : RUN zef install --/test Sparrow6 ---> Using cache ---> a2cbc605ec5e Step 3/7 : RUN apk add bash perl ---> Using cache ---> d9011d4e64db Step 4/7 : ADD sparrowfile . ---> 7a3bb7329d46 Step 5/7 : COPY repo/ /root/repo/ ---> 0c029612c55c Step 6/7 : RUN s6 --index-update ---> Running in 356d29ed8049 17:16:56 06/29/2020 [repository] update local index 17:16:56 06/29/2020 [repository] index updated from file:///root/repo/api/v1/index Removing intermediate container 356d29ed8049 ---> 18876a3d6396 Step 7/7 : RUN raku -MSparrow6::DSL sparrowfile ---> Running in bd07fecae4f0 17:16:58 06/29/2020 [repository] installing bash, version 0.002001 17:17:00 06/29/2020 [bash: echo Hello World] Hello World 17:17:00 06/29/2020 [repository] installing package-generic, version 0.004001 17:17:02 06/29/2020 [install package(s): nano.perl] fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz 17:17:02 06/29/2020 [install package(s): nano.perl] fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz 17:17:02 06/29/2020 [install package(s): nano.perl] v3.12.0-103-g1699efe1cd [http://dl-cdn.alpinelinux.org/alpine/v3.12/main] 17:17:02 06/29/2020 [install package(s): nano.perl] v3.12.0-106-g2b11e345c6 [http://dl-cdn.alpinelinux.org/alpine/v3.12/community] 17:17:02 06/29/2020 [install package(s): nano.perl] OK: 12730 distinct packages available 17:17:03 06/29/2020 [install package(s): nano.perl] trying to install nano ... 17:17:03 06/29/2020 [install package(s): nano.perl] installer - apk 17:17:03 06/29/2020 [install package(s): nano.perl] (1/2) Installing libmagic (5.38-r0) 17:17:03 06/29/2020 [install package(s): nano.perl] (2/2) Installing nano (4.9.3-r0) 17:17:03 06/29/2020 [install package(s): nano.perl] Executing busybox-1.31.1-r19.trigger 17:17:03 06/29/2020 [install package(s): nano.perl] OK: 67 MiB in 32 packages 17:17:03 06/29/2020 [install package(s): nano.perl] Installed: Available: 17:17:03 06/29/2020 [install package(s): nano.perl] nano-4.9.3-r0 = 4.9.3-r0 17:17:03 06/29/2020 [install package(s): nano.perl] nano Removing intermediate container bd07fecae4f0 ---> 408d35e1e3fd Successfully built 408d35e1e3fd Successfully tagged rakuops:1.0
Conclusion
We’ve just seen how one can use Raku and Sparrow to build Docker images. The advantage of the approach one is no more limited by Dockerfile syntax and could leverage all the power of Raku to express any sophisticated build logic. On other hand Sparrow provides a lot of handy primitives and plugins for typical build tasks and some of them I’m going to share in next posts.
