btest: a language agnostic test runner

btest is a minimal, language-agnostic test runner originally written for testing compilers. Brian, an ex- co-worker from Linode, wrote the first implementation in Crystal (a compiled language clone of Ruby) for testing bshift, a compiler project. The tool accomplished exactly what I needed for my own language project, BSDScheme, and had very few dependencies. After some issues with Crystal support in containerized CI environments, and despite some incredible assistance from the Crystal community, we rewrote btest in D to simplify downstream use.

How it works

btest registers a command (or commands) to run and verifies the command output and status for different inputs. btest iterates over files in a directory to discover test groups and individual tests within. It supports a limited template language for easily adjusting a more-or-less similar set of tests. And it supports running test groups and individual tests themselves in parallel. All of this is managed via a simple YAML config.

btest.yaml

btest requires a project-level configuration file to declare the test directory, the command(s) to run per test, etc. Let's say we want to run tests against a python program. We create a btest.yaml file with the following:

test_path: tests

runners:
  - name: Run tests with cpython
    run: python test.py

test_path is the directory in which tests are located. runners is an array of commands to run per test. We hard-code a file to run test.py as a project-level standard file that will get written to disk in an appropriate path for each test-case.

On multiple runners

Using multiple runners is helpful when we want to run all tests with different test commands or test command settings. For instance, we could run tests against cpython and pypy by adding another runner to the runners section.

test_path: tests

runners:
  - name: Run tests with cpython
    run: python test.py
  - name: Run tests with pypy
    run: pypy test.py

An example test config

Let's create a divide-by-zero.yaml file in the tests directory and add the following:

cases:
  - name: Should exit on divide by zero
    status: 1
    stdout: |
      Traceback (most recent call last):
        File "test.py", line 1, in <module>
          4 / 0
      ZeroDivisionError: division by zero
    denominator: 0
templates:
  - test.py: |
      4 / {{ denominator }}

In this example, name will be printed out when the test is run. status is the expected integer returned by running the program. stdout is the entire expected output written by the program during execution. None of these three fields are required. If status or stdout are not provided, btest will skip checking them.

Any additional key-value pairs are treated as template variable values and will be substituted if/where it is referenced in the templates section when the case is run. denominator is the only such variable we use in this example. When this first (and only) case is run, test.py will be written to disk containing 4 / 0.

templates section

The templates section is a dictionary allowing us to specify files to be created with variable substitution. All files are created in the same directory per test case, so if we want to import code we can do so with relative paths.

Here is a simple example of a BSDScheme test that uses this feature.

Running btest

Run btest from the root directory (the directory above tests) and we'll see all the grouped test cases that btest registers and the result of each test:

$ btest
tests/divide-by-zero.yaml
[PASS] Should exit on divide by zero

1 of 1 tests passed for runner: Run tests with cpython

Use in CI environments

In the future we may provide pre-built release binaries. But in the meantime, the CI step involves downloading git and ldc and building/installing btest before calling it.

Circle CI

This is the config file I use for testing BSDScheme:

version: 2
jobs:
  build:
    docker:
      - image: dlanguage/ldc
    steps:
      - checkout
      - run:
          name: Install debian-packaged dependencies
          command: |
            apt update
            apt install -y git build-essential
            ln -s $(which ldc2) /usr/local/bin/ldc
      - run:
          name: Install btest
          command: |
            git clone https://github.com/briansteffens/btest
            cd btest
            make
            make install
      - run:
          name: Install bsdscheme
          command: |
            make
            make install
      - run:
          name: Run bsdscheme tests
          command: btest

Travis CI

This is the config Brian uses for testing BShift:

sudo: required

language: d

d:
    - ldc

script:
    # ldc gets installed as other names sometimes
    - sudo ln -s `which $DC` /usr/local/bin/ldc

    # bshift
    - make
    - sudo ln -s $PWD/bin/bshift /usr/local/bin/bshift
    - sudo ln -s $PWD/lib /usr/local/lib/bshift

    # nasm
    - sudo apt-get install -y nasm

    # basm
    - git clone https://github.com/briansteffens/basm
    - cd basm && cabal build && cd ..
    - sudo ln -s $PWD/basm/dist/build/basm/basm /usr/local/bin/basm

    # btest
    - git clone https://github.com/briansteffens/btest
    - cd btest && make && sudo make install && cd ..

    # run the tests
    - btest