21

There are apparently two ways to integrate testthat with R CMD check. I can't get either to work.

Approach #1: (perhaps deprecated)

According to the devtools wiki:

When developing a package, put your tests in inst/tests and then create a file tests/run-all.R (note that it must be a capital R), which contains the following code:

library(testthat) 
library(mypackage)
test_package("mypackage")

This will evaluate your tests in the package namespace (so you can test non-exported functions), and it will throw an error if there are any test failures. This means you'll see the full report of test failures and R CMD check won't pass unless all tests pass.

The whole package is here. In it are the two files: 

## minimalbugexample/inst/tests/run-all.R
library(testthat)
library(minimalbugexample)
test_package('minimalbugexample')

and

## minimalbugexample/inst/tests/test-use-Matrix-package.R
context("Intentional break")
  expect_that( TRUE, equals(FALSE))

my DESCRIPTION is 

Package: minimalbugexample
Title: 
Description: 
Version: 0.1.1
Author: Nathan VanHoudnos <nathanvan@letterafterFmail.com>
Maintainer: Nathan VanHoudnos <nathanvan@letterafterFmail.com>
Depends:
    R (>= 3.0.1),
    Matrix (>= 1.0)
Suggests:
    testthat
License: GPL
LazyData: true
Collate:
    'minimalbugexample-package.r'
    'use-Matrix-package.R'

After installing the package, I can run the tests just fine (they fail, as expected). 

> test_package('minimalbugexample')
Intentional break : 1


1. Failure:  -------------------------------------------------------------------
TRUE not equal to FALSE
1 element mismatch
Error: Test failures
>

But an R CMD check doesn't run the tests.

$ R CMD check minimalbugexample_0.1.1.tar.gz 
... snip ...
* checking PDF version of manual ... WARNING
WARNING: There was 1 warning.
See
  ‘/home/nathanvan/software/minimalbugexample.Rcheck/00check.log’
for details.

I don't think that the PDF warning has anything to do with this, but I can provide more details if requested.

Approach #2: (bleeding edge)

According to the README file of the testthat repository:

Now, recommend practice is to put your tests in tests/testthat, and ensure R CMD check runs then by putting the following code in tests/test-all.R:

library(testthat)
test_check(yourpackage)

So I made sure I had the most recent version of testthat installed: 

> install_github("testthat")

And then changed the package. You can get this version here. I modified the two files to be 

## minimalbugexample/inst/tests/test-all.R
library(testthat)
test_check(minimalbugexample)

and

## minimalbugexample/inst/tests/testthat/test-use-Matrix-package.R
context("Intentional break")
  expect_that( TRUE, equals(FALSE))

Then updating the package version to 0.1.2 in the DESCRIPTION file, I can build it, install it, and use testthat to check it and get the same output as before. So it seems that as far as testthat is concerned, its working.

However, R CMD check still doesn't run the tests: 

$ R CMD check minimalbugexample_0.1.2.tar.gz 
... snip ...
* checking PDF version of manual ... WARNING
LaTeX errors when creating PDF version.
This typically indicates Rd problems.
WARNING: There was 1 warning.
See
  ‘/home/nathanvan/software/minimalbugexample.Rcheck/00check.log’
for details.

So the question:

What am I doing wrong? My preference is for a solution for Approach 2, but I'll take either!

Nathan VanHoudnos
  • 12,923
  • 2
  • 23
  • 29
  • 2
    Just a note that if you're using approach #1 you [need to add](http://stackoverflow.com/questions/21380288/loading-the-testthat-package-only-for-tests-but-not-requiring-in-dependencies) testthat to `Suggests:` in `DESCRIPTION`. – Ari B. Friedman Jan 27 '14 at 12:28

1 Answers1

14

You don't have a tests directory. test-all.R should be located at minimalbugexample/tests/test-all.R.

Then your actual tests go in minimalbugexample/inst/tests for approach #1 or minimalbugexample/tests/testthat/ for approach #2.

For approach #2, the test-all.R file should use test_check(yourpackage) instead of test_package(yourpackage) and the library(yourpackage) call is no longer required.

GSee
  • 48,880
  • 13
  • 125
  • 145
  • I guess the issue is that I can't read! Thanks! (Works for Approach #2!) – Nathan VanHoudnos Jul 11 '13 at 16:47
  • Actually, for approach #2, the `test-all.R` file should be located at `minimalbugexample/tests/test-all.R` as you say, but the actual tests go in `minimalbugexample/tests/testthat/`. It took some more bug testing for me to work it out. If you incorporate that into your answer I'll re-accept it! [Example on gitHub](https://github.com/nathanvan/minimalbugexample/tree/e6d62a6da7127fc165332935db04aa88b4575ae6). – Nathan VanHoudnos Jul 11 '13 at 17:52
  • Thanks. Editing is a new priv for me, so I was unsure of the community norms. – Nathan VanHoudnos Jul 11 '13 at 19:41
  • 2
    Also, to avoid a warning about unspecified dependencies, you should add `Suggests: testthat` to your DESCRIPTION file – Juancentro Apr 21 '14 at 01:09
  • Running the tests is amongst the last actions of `R CMD check `, so any errors that occur earlier could prevent them from running. Such as the lack of a `NAMESPACE` file. – Steve Pitchers Jul 09 '15 at 15:08
  • Also note that the *uppercase* extension to `test-all.R` is essential. Otherwise `R CMD check` won't run the tests. – Michael Schubert Dec 07 '16 at 11:56