Chasing Rakudo bugs with Sparrow

Since I announced a RICH – Rakudo Issues Confirmation Helper, I’ve been playing a lot with an automation of test cases for known Rakudo bugs.

One thing I’ve found really interesting in this journey is how Sparrow could be a good fit to express issues through automation scenarios.

While I am not pretending on substituting existing Roast test system by Sparrow, I’d like to highlight an alternative approach here, and maybe Rakudo devs will pay an attention on the tool 🙂 and start using it one day.

The following is just some examples and thoughts, and not meant to be a complete “user” guide.

Chasing a bug

It’s all starts with a user describe a bug through a Rakudo GitHub issues page. Let’s take a look at the fresh one, issue #4119:

The Problem

Chaining operators are always iffy, however is assoc<chain> doesn’t make a custom operator iffy (unlike is equiv with a chaining operator, which does).

Expected Behavior

raku -e ‘sub infix:<ab>($a, $b) is assoc<chain> { True }; say 1 !ab 2’

False

Actual Behavior

raku -e ‘sub infix:<ab>($a, $b) is assoc<chain> { True }; say 1 !ab 2’

===SORRY!=== Error while compiling -e

Cannot negate ab because additive operators are not iffy enough

By @leont

So we have 3 essential parts in the issue definition:

* Human readable description
* Expected behavior code + expected output
* Actual behavior code + actual output


Let’s analyze all these piece by piece.

Description

This is just brief description of an issue, not meant to be used in automation process, but helping developers to understand the issue from the high level side. Let’s skip it.


Expected behavior / Actual behavior

This and following bits are most important for test automation purposes, because they are Raku scenarios expressing an issue. The first one as you could guess showing an example code and desired output, the second one is the same code but with some real output and probably unsuccessful exit code.

BDD Approach

BDD paradigm reenforces the idea of close relationship between software users and software developers, it tries to bridge the gap between those two groups. One of the approaches for that users express desired system behavior in runnable scenarios, which are both human readable specification and test code.

A classical Given/When/Then statements criteria is one of these methods.

In a case of Rakudo bug it could written as:

Given: I have this version of Rakudo 
When: I run this code
Then: I should exits successfully and have this output

Let’s Sparrow it!

Here comes the most interesting part. Sparrow has some TDD features by design, so it’s quite easy to implement the idea though this tool. The rest of the post is just an example of Sparrow workflow when automating Rakudo bugs tests.

Given: I have this version of Rakudo

This statement does not need any explicit coding ( but see raku --version in the following script) and is “ensured” by a working environment a test gets run against. Usually users catch bugs on their laptops 🙂

When I run this code

The body of a test is just a simple Bash script that gets executed by Sparrow command line sp6, again in a spirit of the idea, it should literally reproduce a bug, they way you got it. In most of the cases Bash oneliner is enough:

mkdir -p issues/4119

issues/4119/task.bash

set -x
set -e

raku --version

raku -e 'sub infix:<ab>($a, $b) is assoc<chain> { True }; say 1 !ab 2';

Then: I should exits successfully and have this output

Now, let’s just a test a Sparrow task and see a result:

s6 --task-run issues/4119

[sparrowtask] :: run sparrow task issues/4119
[sparrowtask] :: run thing issues/4119
[issues/4119] :: stderr: ++ set -e
++ raku --version
[issues/4119] :: This is Rakudo version 2020.07 built on MoarVM version 2020.07
[issues/4119] :: implementing Raku 6.d.
[issues/4119] :: stderr: ++ raku -e 'sub infix:<ab>($a, $b) is assoc<chain> { True }; say 1 !ab 2'
[issues/4119] :: stderr: ===SORRY!=== Error while compiling -e
Cannot negate ab because additive operators are not iffy enough
at -e:1
[issues/4119] :: stderr: ------> a, $b) is assoc<chain> { True }; say 1 !⏏ab 2
    expecting any of:
        infix
        infix stopper
[issues/4119] :: task exit status: 1
[issues/4119] :: task issues/4119 FAILED

Actual / Expected output

In the given example a sample code exited with error without even producing a desired output, this is a simple form of test. But what if a code exited with 0 yet producing wrong output? This is quite easily with Sparrow task check mechanism.

Let’s take a look at another example of known Rakudo issue #4118 :

issues/4118/task.bash

set -x
set -e

raku --version

raku -e "say qq{===};
  say [⊖] (1,2,3), (1,2,3), (1,2,3);
  say [⊖] (0,1,2), (0,1,2), (0,1,2);
  say qq{===}
" 2>&1;

issues/4118/task.check

begin:
===
regexp: ^^ 'Set()' $$
regexp: ^^ 'Set()' $$
===
end:

s6 --task-run issues/4118/

[sparrowtask] :: run sparrow task issues/4118/
[sparrowtask] :: run thing issues/4118/
[issues/4118/] :: stderr: ++ set -e
[issues/4118/] :: stderr: ++ raku --version
[issues/4118/] :: This is Rakudo version 2020.07 built on MoarVM version 2020.07
[issues/4118/] :: implementing Raku 6.d.
[issues/4118/] :: stderr: ++ raku -e 'say qq{===};
  say [⊖] (1,2,3), (1,2,3), (1,2,3);
  say [⊖] (0,1,2), (0,1,2), (0,1,2);
  say qq{===}
'
[issues/4118/] :: ===
[issues/4118/] :: Set()
[issues/4118/] :: Set(0)
[issues/4118/] :: ===
[task check] stdout match (s) <===> True
[task check] stdout match (s) <^^ 'Set()' $$> True
[task check] stdout match (s) <^^ 'Set()' $$> False
[task check] stdout match (s) <===> False
=================
TASK CHECK FAIL

In this test we ensure that code sample produces Set() twice. Sparrow task checks DSL is very handy in that case.


Conclusion

Sparrow allows to write test scenarios in a way as close as possible to what users get on their environments, it could be an efficient testing and collaboration tool enabling both users and developers interact on Rakudo bugs efficiently without having an unnecessary additional layers.

More Rakudo bugs Sparrow scenarios could be found here – https://github.com/melezhik/RakuPlay/tree/main/issues

—

Thanks for reading

Advertisement