4

I'm working on setting up my team's new unit test and integration test infrastructure and want to make sure I'm starting off by selecting the correct test frameworks. I'm an embedded developer testing code running on a VxWorks operating system with a C/C++ production codebase.

We need a framework capable of directly testing C/C++ for unit testing, so for our unit tests I chose Googletest as our framework.

However, for integration tests we've generally tested using Python scripts (with no test framework). The Python scripts connect to the embedded system over a network and test cases via sending commands and receiving telemetry.

Would using pytest as a test framework be beneficial to the way we're currently using Python for integration testing an embedded system? Most of the examples I've seen use pytest in a more unit test fashion by creating assertions for single functions in a Python production codebase.

EDIT: Per hoefling's comment, i'll provide a (very simplified) example of one of our existing Python integration test cases, and also what I believe its corresponding Pytest implementation would be.

#Current example
def test_command_counter():
    preTestCmdCount = getCmdCountFromSystem()
    sendCommandToSystem()
    postTestCmdCount = getCmdCountFromSystem()

    if (postTestCmdCount != (preTestCmdCount + 1)):
        print("FAIL: Command count did not increment!")
    else:
        print("PASS")


#Using Pytest?
def test_command_counter():
    preTestCmdCount = getCmdCountFromSystem()
    sendCommandToSystem()
    postTestCmdCount = getCmdCountFromSystem()

    assert postTestCmdCount == (preTestCmdCount + 1)

So, correct me if I'm wrong, but it appears that the advantages of using Pytest over plain Python for this simplified case would be:

  1. Being able to make use of pytest's automated test case discovery, so that I can easily run all of my test functions instead of having to create custom code to do so.

  2. Being able to make use of the 'assert' syntax which will automatically generate pass/fail statements for each test instead of having to manually implement pass/fail print statements for each test case

tyler124
  • 613
  • 1
  • 8
  • 20
  • 2
    `pytest` is highly customizable and can be well used for integration tests. As for the question, it is too broad IMO and depends on your requirements; maybe reformulate? Or give an example of the test scripts you are currently writing and ask how it would look like when rewritten using `pytest` etc. – hoefling Jul 02 '19 at 21:10
  • Thanks for your response @hoefling. Please see edited question. – tyler124 Jul 02 '19 at 21:49
  • Nice, upvoted! Adding my 2 cents - key features IMO are reusing code for resources setup/teardown via fixtures, test parametrization without loops so e.g. running same test against multiple devices is a one-liner and parallelizing test execution (including partitioning tests on multiple machines). – hoefling Jul 02 '19 at 22:29
  • The rest are small but neat features like versatile test selection for execution (e.g. run only tests that have failed in the last run, run only "new" tests, print found tests without running them etc), controlled execution (e.g. stopping on first failure or after max. `n` failures, skipping tests, marking tests as "expected failure" that won't fail the test suite), customizable reporting etc. Also, a wide range of plugins. – hoefling Jul 02 '19 at 22:32

2 Answers2

3

I've been in a similar situation, and from what I gathered unit testing frameworks are NOT appropriate for integration testing on embedded systems. A similar question was asked here: Test framework for testing embedded systems in Python

We personally use Google's OpenHTF to automate both the integration testing (as in your case) and the production verification testing, which includes bringup, calibration and overall verification of assembly.

Check it out: https://github.com/google/openhtf

We automate advanced test equipment such as RF switches and Spectrum Analysers all in Python in order to test and calibrate our RF units, which operate in the >500 MHz range.

Using OpenHTF, you can create complete tests with measurements very quickly. It provides you with a builtin web GUI and allows you to write your custom export 'callbacks'.

We're currently building a complete test solution for hardware testing. I'd be glad to help with OpenHTF if needed, as we're basing our flagship implementation on it.

William Laroche
  • 131
  • 8
1

This thread's old, but these suggestions might help someone...

For unit testing embedded C on a host PC and on target we use Unity and CMock. http://www.throwtheswitch.org/

For hardware-in-the-loop testing we use pytest.

Both work well and are part of our Jenkins release workflow.

Mark Grubb
  • 21
  • 2