2

I have some unit tests for code that is doing some very minor image manipulation (combining several small images into a larger image). When I was running the tests, I noticed that three out of four of them failed on the line where they are reading an image from a directory (fails with an index out of bounds error).

However, if I run it again, they all pass. When I was writing the code as well, I noticed that whenever I set a breakpoint in my code, I'd have to run the unit tests twice because (after the very first time) it would run through the tests without hitting any breakpoints.

I have my repo organized like this:

src/
    /* source code .m files are in here */

unit_tests/
    images/
        squares/
            - img1.png
            - img2.png
            ...
            - imgn.png
    - unit_tests.m

And I have a line in my setup (inside unit_tests.m) to generate and add paths for all of the code:

function tests = unit_tests()
    addpath(genpath('..'));    
    tests = functiontests(localfunctions);
end

The unit tests all have this format:

function testCompositeImage_2x3(testCase)
    squares = dir('images/squares/*.png');
    num_images = length(squares);
    img = imread([squares(1).folder filesep squares(1).name]); % all same size squares
    rows = 2;
    cols = 3;
    buffer = 2;

    for idx = 1:num_images - (rows*cols)
        imarray = cell(1,(rows*cols));
        n = 1;
        for ii = idx:idx +(rows*cols) -1
            imarray{n} = imread([squares(ii).folder filesep squares(ii).name]);
            n = n + 1;
        end

        newimg = createCompositeImage(rows,cols,imarray, buffer);

        expCols = cols*size(img,1) + (cols+1)*2*buffer;
        expRows = rows*size(img,2) + (rows+1)*2*buffer;
        assert(checksize(newimg, expRows, expCols, 3) == true);
    end
end

("checksize" is just a helper I wrote that returns a boolean b/c assert doesn't compare matrices)

When I launch a fresh matlab session and run the unit tests (using the "Run Tests" button in the editor tab), they pass with this output:

>> runtests('unit_tests\unit_tests.m')
Running unit_tests
.......
Done unit_tests
__________


ans = 

  1×7 TestResult array with properties:

    Name
    Passed
    Failed
    Incomplete
    Duration
    Details

Totals:
   7 Passed, 0 Failed, 0 Incomplete.
   0.49467 seconds testing time.

Running it a second time (again, by pressing the button):

>> runtests('unit_tests')
Running unit_tests
..
================================================================================
Error occurred in unit_tests/testCompositeImage_2x2 and it did not run to completion.

    ---------
    Error ID:
    ---------
    'MATLAB:badsubscript'

    --------------
    Error Details:
    --------------
    Index exceeds array bounds.

    Error in unit_tests>testCompositeImage_2x2 (line 47)
        img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
/*similar error info for the other two failing tests...*/
...
Done unit_tests
__________

Failure Summary:

     Name                               Failed  Incomplete  Reason(s)
    ==================================================================
     unit_tests/testCompositeImage_2x2    X         X       Errored.
    ------------------------------------------------------------------
     unit_tests/testCompositeImage_2x3    X         X       Errored.
    ------------------------------------------------------------------
     unit_tests/testCompositeImage_3x2    X         X       Errored.


ans = 

  1×7 TestResult array with properties:

    Name
    Passed
    Failed
    Incomplete
    Duration
    Details

Totals:
   4 Passed, 3 Failed (rerun), 3 Incomplete.
   0.0072287 seconds testing time.

The fact that it's failing on basically the first line because it's not reading anything from the folder leads me to suspect that even when the other 4 tests supposedly pass, they are in fact not running at all. And yet, if I run the tests again, they all pass. Run it a 4th time, and they fail once again.

At first, I thought that perhaps the unit tests were executing too quickly (only on even-numbered runs somehow?) and it was running the unit tests before the addpath/genpath functions in the setup had finished, so I added a pause statement and re-ran the tests, but I had the same issue only this time it would wait for the requisite number of seconds before going ahead and failing. If I run it again, no problem - all my tests pass.

I am completely at a loss as to why this is happening; I am using vanilla matlab (R2018a) running on a Win10 machine and don't have anything fancy going on. I feel like you should be able to run your unit tests as many times as you like and expect the same result! Is there something I've just somehow overlooked? Or is this some bizarre feature?

asdf
  • 109
  • 7
  • It looks like the command you use to run the tests is different the two times. – Cris Luengo Apr 23 '20 at 01:06
  • I have the feeling that the `addpath` is to blame. I would compute an absolute path with `mfilename` and `fileparts` and `fullfile`, rather than using `..`. – Cris Luengo Apr 23 '20 at 01:08
  • 1
    @CrisLuengo - I can't believe I overlooked it, but you are totally correct - they are being run with two different commands! I was using the "Run Tests" button and incorrectly assumed that it was using the same command. Interestingly, if I run `runtests('unit_tests')` twice from the commandline, it gives me different numbers of passing failing tests (2 instead of 3), while `runtests('unit_tests\unit_tests.m')` always gives the correct output. – asdf Apr 23 '20 at 02:07
  • I tried changing the addpath to `filepath = regexp(fileparts(which(mfilename)), '\unit_tests', 'split');` and `addpath(genpath(filepath{1}));`, however this has actually made it worse for some unfathomable reason. Now it only runs successfully one time (clicking the button only uses `runtests('unit_tests\unit_tests.m')` once) Do you have any suggestions on the correct method? – asdf Apr 23 '20 at 02:28
  • 1
    Something like `fullfile(fileparts(fileparts(mfilename('fullpath'))),'src')` should build the right path, if I’m not mistaken. – Cris Luengo Apr 23 '20 at 02:40
  • @CrisLuengo thanks, that worked. I wrote up my fix as an answer in case someone ever has the same issue. If you've got a more elegant solution, feel free to edit/etc – asdf Apr 23 '20 at 07:03

1 Answers1

1

Adding my fix just in case someone else runs into the same issue.

As pointed out by Cris, something about the line 

addpath(genpath('..'));

causes the GUI to go into a weird state where pressing the "Run Tests" button alternates between calling runtests('unit_tests\unit_tests.m') and runtests('unit_tests') which in turn causes the tests to alternately pass and fail. It does not seem to be an issue with the path variable itself (as it always contains - at a minimum - the necessary directories), but rather something intrinsic to matlab itself. The closest I could get to the root of the issue was the call to the (compiled) dir function inside the genpath function.

The "correct" solution was to remove the line from the unit_tests function entirely and add it to a setupOnce function:

function tests = unit_tests()
    tests = functiontests(localfunctions);
end

function setupOnce(testCase)
    addpath(genpath('..'));
end

A hack (not recommended) which doesn't require a setupOnce function is the following:

function tests = unit_tests()
    pth = fullfile(fileparts(fileparts(mfilename('fullpath'))),'src');
    paths = regexp(genpath(pth), ';', 'split');
    for idx = 1:length(paths) - 1 % last element is empty
        addpath(paths{idx});
    end
end

I needed to relaunch matlab for the changes to take effect. This worked for my setup using r2018a running on Win10.

asdf
  • 109
  • 7
  • You can `Accept` your own answer. This may seem perverse at first glance but it does effectively make the answer, more `findable`, to others with a similar issue. – Rolf of Saxony May 21 '20 at 17:30