How to unit test a command line interface

Question

I've written a command line tool that I want to test (I'm not looking to run unit tests from command line). I want to map a specific set of input options to a specific output. I haven't been able to find any existing tools for this. The application is just a binary and could be written in any language but it accepts POSIX options and writes to standard output.

Something along the lines of:

• For each known set of input options:
  1. Launch application with specified input.
  2. Pipe output to a file.
  3. Diff output to stored (desired) output.
  4. If diff is not empty, record error.

(Btw, is this what you call an integration test rather than a unit test?)

Edit: I know how I would go about writing my own tool for this, I don't need help with the code. What I want to learn is if this has already been done.

Answer 1

DejaGnu is a mature and somewhat standard framework for writing test suites for CLI programs.

Here is a sample test taken from this tutorial:

# send a string to the running program being tested:
send "echo Hello world!\n"

# inspect the output and determine whether the test passes or fails:
expect {
    -re "Hello world.*$prompt $" {
        pass "Echo test"
    }
    -re "$prompt $" {
        fail "Echo test"
    }
    timeout {
        fail "(timeout) Echo test"
    }
}

Using a well-established framework like this is probably going to be better in the long run than anything you can come up with yourself, unless your needs are very simple.

Answer 2

You are looking for BATS (Bash Automated Testing System): https://github.com/bats-core/bats-core

From the docs:

example.bats contains
#!/usr/bin/env bats

@test "addition using bc" {
  result="$(echo 2+2 | bc)"
  [ "$result" -eq 4 ]
}  

@test "addition using dc" {
  result="$(echo 2 2+p | dc)"
  [ "$result" -eq 4 ]
}


$ bats example.bats

 ✓ addition using bc
 ✓ addition using dc

2 tests, 0 failures

bats-core

Answer 3

Well, I think every language should have a way of execute an external process.

In C#, you could do something like:

var p = new Process();
p.StartInfo = new ProcessStartInfo(@"C:\file-to-execute.exe");
... //You can set parameters here, etc.
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.RedirectStandardInput = true;
p.StartInfo.UseShellExecute = false;
p.Start();

//To read the standard output:
var output = p.StandardOutput.ReadToEnd();

I have never had to write to the standard input, but I believe it can be done by accessing to p.StandardInput as well. The idea is to treat both inputs as Stream objects, because that's what they are.

In Python there is the subprocess module. According to its documentation:

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.

I had to do the same when writing unit tests for the code generation part of a compiler I write some months ago: Writing unit tests in my compiler (which generates IL)

Answer 4

We wrote should, a single-file Python program to test any CLI tool. The default usage is to check that a line of the output contains some pattern. From the docs:

# A .should file launches any command it encounters.
echo "hello, world"

# Lines containing a `:` are test lines.
# The `test expression` is what is found at the right of the `:`.
# Here 'world' should be found on stdout, at least in one line.
:world

# What is at the left of the `:` are modifiers.
# One can specify the exact number of lines where the test expression has to appear.
# 'moon' should not be found on stdout.
0:moon

Should can check occurrences counts, look for regular expressions, use variables, filter tests, parse json data, and check exit codes.

Answer 5

Sure, it's been done literally thousands of times. But writing a tool to run simple shell scripts or batch files like what you propose is a trivial task, hardly worth trying to turn into a generic tool.