How to stop a process from within the tests, when testing a never-ending process?

Question

I am developing a long-running program in Ruby. I am writing some integration tests for this. These tests need to kill or stop the program after starting it; otherwise the tests hang.

For example, with a file bin/runner

#!/usr/bin/env ruby
while true do
  puts "Hello World"
  sleep 10
end

The (integration) test would be:

class RunReflectorTest < TestCase
  test "it prints a welcome message over and over" do
    out, err = capture_subprocess_io do
      system "bin/runner"
    end
    assert_empty err
    assert_includes out, "Hello World"
  end
end

Only, obviously, this will not work; the test starts and never stops, because the system call never ends.

How should I tackle this? Is the problem in system itself, and would Kernel#spawn provide a solution? If so, how? Somehow the following keeps the out empty:

class RunReflectorTest < TestCase
  test "it prints a welcome message over and over" do
    out, err = capture_subprocess_io do
      pid = spawn "bin/runner"
      sleep 2
      Process.kill pid
    end
    assert_empty err
    assert_includes out, "Hello World"
  end
end

. This direction also seems like it will cause a lot of timing-issues (and slow tests). Ideally, a reader would follow the stream of STDOUT and let the test pass as soon as the string is encountered and then immediately kill the subprocess. I cannot find how to do this with Process.

Not sure whether the async approach you mentioned is feasible. But maybe it would work to do it from shell via a [timeout command](https://stackoverflow.com/questions/5161193/bash-script-that-kills-a-child-process-after-a-given-timeout). Or maybe even Ruby's Timeout.timeout method — max pleaner, Jun 16 '17 at 17:14
"let the test pass as soon as the string is encountered" is a job for IO#expect. https://ruby-doc.org/stdlib-2.4.1/libdoc/expect/rdoc/IO.html — Todd A. Jacobs, Jun 17 '17 at 22:55

score 1 · Answer 1 · answered Jun 17 '17 at 22:47

Test Behavior, Not Language Features

First, what you're doing is a TDD anti-pattern. Tests should focus on behaviors of methods or objects, not on language features like loops. If you must test a loop, construct a test that checks for a useful behavior like "entering an invalid response results in a re-prompt." There's almost no utility in checking that a loop loops forever.

However, you might decide to test a long-running process by checking to see:

If it's still running after t time.
If it's performed at least i iterations.
If a loop exits properly given certain input or upon reaching a boundary condition.

Use Timeouts or Signals to End Testing

Second, if you decide to do it anyway, you can just escape the block with Timeout::timeout. For example:

require 'timeout'

# Terminates block
Timeout::timeout(3) { `sleep 300` }

This is quick and easy. However, note that using timeout doesn't actually signal the process. If you run this a few times, you'll notice that sleep is still running multiple times as a system process.

It's better is to signal the process when you want to exit with Process::kill, ensuring that you clean up after yourself. For example:

pid = spawn 'sleep 300'
Process::kill 'TERM', pid
sleep 3
Process::wait pid

Aside from resource issues, this is a better approach when you're spawning something stateful and don't want to pollute the independence of your tests. You should almost always kill long-running (or infinite) processes in your test teardown whenever you can.

My simplified example was a bit too condensed: I am not testing "that there is a loop", merely that the loop is there, my tests don't care about it, but "a kind of loop" prohibits the process from stopping itself. — berkes, Jun 20 '17 at 08:38

score 1 · Answer 2 · answered Jun 18 '17 at 10:20

Ideally, a reader would follow the stream of STDOUT and let the test pass as soon as the string is encountered and then immediately kill the subprocess. I cannot find how to do this with Process.

You can redirect stdout of spawned process to any file descriptor by specifying out option

pid = spawn(command, :out=>"/dev/null") # write mode

Documentation

Example of redirection

score 0 · Accepted Answer · answered Jun 20 '17 at 15:12

With the answer from CodeGnome on how to use Timeout::timeout and the answer from andyconhin on how to redirect Process::spawn IO, I came up with two Minitest helpers that can be used as follows:

it "runs a deamon" do
  wait_for(timeout: 2) do
    wait_for_spawned_io(regexp: /Hello World/, command: ["bin/runner"])
  end
end

The helpers are:

def wait_for(timeout: 1, &block)
  Timeout::timeout(timeout) do
    yield block
  end
rescue Timeout::Error
  flunk "Test did not pass within #{timeout} seconds"
end

def wait_for_spawned_io(regexp: //, command: [])
  buffer = ""

  begin
    read_pipe, write_pipe = IO.pipe
    pid = Process.spawn(command.shelljoin, out: write_pipe, err: write_pipe)

    loop do
      buffer << read_pipe.readpartial(1000)
      break if regexp =~ buffer
    end
  ensure
    read_pipe.close
    write_pipe.close
    Process.kill("INT", pid)
  end

  buffer
end

These can be used in a test which allows me to start a subprocess, capture the STDOUT and as soon as it matches the test Regular Expression, it passes, else it will wait 'till timeout and flunk (fail the test).

The loop will capture output and pass the test once it sees matching output. It uses a IO.pipe because that is most transparant for subprocesses (and their children) to write to.

I doubt this will work on Windows. And it needs some cleaning up of the wait_for_spawned_io which is doing slightly too much IMO. Antoher problem is that the Process.kill('INT') might not reach the children which are orphaned but still running after this test has ran. I need to find a way to ensure the entire subtree of processes is killed.

How to stop a process from within the tests, when testing a never-ending process?

3 Answers3

Test Behavior, Not Language Features

Use Timeouts or Signals to End Testing