7

I'm creating a library of components in Modelica, and would appreciate some input on techniques for unit testing the package.

So far I have a test package, consisting of a set of models, one per component. Each test model instantiates a component, and connects it to some very simple helper classes that provide the necessary inputs and outputs.

This works fine when using it interactively in the OMEditor, but I'm looking for a more automated solution with pass/fail criteria etc.

Should I start writing .mos scripts, or is there another/better way ?

Thanks.

gwr
  • 465
  • 6
  • 17
Jack Welch
  • 115
  • 6
  • I wonder if the way in which [jModelica](http://www.jmodelica.org/api-docs/compliance_diagnostics/1.17/check/index.html) checks its versions represents a good way of performing the Unit Testing... but how do they do this...? Perhaps via python script and checking FMUs? – Scott G Mar 23 '16 at 18:32
  • Here is a nice presentation that was given by Dassault at the Modelica 2017 conference: https://modelica.org/events/modelica2017/documents/3ds_TestingTutorialSlides.pdf – matth Jun 13 '17 at 09:05

4 Answers4

7

I like how Openmodelica testing results look, see

No idea how they are doing it, though. Obviously some kind of regression testing is done, with previous results stored, but no idea if that is from some testing library or self-made.

In general, I find it kinda sad/suboptimal, that there isn't "the one" testing solution everybody can/should use (cf. e.g. nose or pytest in the python ecosystem), instead everybody seems to cook up their own solutions (or tries to), and all you find is some Modelica conference papers (often without a trace of implementation) or unmaintained library of unknown status.

Off the top of my head, I found/know of (some already linked in other answers here)

This seems like a pathological instance of https://xkcd.com/927/. It's kinda impossible for a (non-dev) user to know which of those to choose, which are actually good/usable/available/...

(Not real testing, but also relevant: parsing and semantic analysis using ANTLR: modelica.org/events/Conference2003/papers/h31_parser_Tiller.‌​pdf)

gwr
  • 465
  • 6
  • 17
Christoph
  • 5,480
  • 6
  • 36
  • 61
  • I've made an in house unit testing platform for our libraries. It looks quite like the openModelica hudson. I used a text template to generate modelica script and parse test results to a JUNIT xml format. In Dymola, you can run .mos scripts directly from batch or direct API calls from python scripts and javascripts. If you are interested in further details shoot me an email, I believe some further discussions will make my stuff a better platform. – Hang Yu Mar 24 '16 at 15:11
  • 2
    Very detailed list - thanks ! I agree that it's a shame there isn't a common solution (or class of solutions) that address this. I do a lot of work in the .Net/C# world, where there are many testing solutions that all follow the same pattern: xunit, nunit etc etc, each with a VS integrated test runner AND automated build support. There are so many things to like about Modelica, but the lack of good devops tooling stands out as a big minus to me. – Jack Welch Mar 24 '16 at 16:38
  • 1
    Another example of unit testing and overall project implementation for Modelica that seems to have been done well is the [Buildings Library](https://github.com/lbl-srg) from Lawrence Berkeley National lab. It'd probably good to add it to the list. – Scott G Apr 01 '16 at 13:12
  • I added some more details about the different packages that I dug up. So far it's looking bleak. :-/ – Christoph Jul 13 '16 at 11:14
  • There is yet another effort to get test and verification standardised: `Modelica_Requirements` http://www.ep.liu.se/ecp/118/067/ecp15118625.pdf – Dietmar Winkler Aug 04 '16 at 12:08
  • Dietmar, do you know where the library is available from? This looks good, but the paper happily talks about "open source" and "implemented" and "can be used", but does not offer a URL, and searching via Google and Impact I came up empty.. :-/ – Christoph Aug 04 '16 at 12:28
  • Not real testing, but also relevant here: parsing and semantic analysis using ANTLR: https://www.modelica.org/events/Conference2003/papers/h31_parser_Tiller.pdf – matth May 09 '17 at 09:17
  • BuildingsPy should now work with Python 3, see https://github.com/lbl-srg/BuildingsPy/issues/94 – matth May 09 '17 at 12:51
  • 1
    @Christoph You already know all this, but you might still like this presentation: https://docs.google.com/presentation/d/1V0Kov6ACsGKjnfDI3QLxAMgEtJAb2jzbk19xmK_23uQ/pub?start=false&loop=false&delayms=3000 – matth May 18 '17 at 11:16
  • nice compilation of the current state by mthorade. Seems to be for the 2017 conference, hopefully can get an actually downloadable version from there later. Also, maybe it gets something moving w.r.t. a standard solution.. – Christoph May 18 '17 at 11:45
  • 1
    A detailed summary of the available testing solutions can also be found in https://modelica.org/events/modelica2017/documents/3ds_TestingTutorialSlides.pdf. Brief guides for installation and how to setup tests are included for the tool which are freely available. – marco Oct 12 '18 at 14:41
3

Writing a .mos script would be one way but there is also a small proof-of-concept library by Michael Tiller: XogenyTest which you could use as a basis.

Dietmar Winkler
  • 942
  • 5
  • 11
  • 1
    Thanks - anything by Michael Tiller is worth looking at. – Jack Welch Mar 24 '16 at 16:30
  • yeah, but the fact that this library only has very few commits and hasn't been updated in 3 years makes me think it didn't really work out well (otherwise people would have adopted it). :-/ – Christoph Mar 25 '16 at 19:52
  • Christoph, it was simply done as a proof of concept and probably not meant to be a full project. Still it can serve as useful. – Dietmar Winkler Mar 27 '16 at 00:38
  • Yes, I gathered that, but the question that occurs to me: if it could indeed serve as useful, why was it not developed further/maintained/developed into a full package? In any case, it's on my list of things to evaluate some time... – Christoph Mar 29 '16 at 06:19
3

I prefer using the .mos script, it works pretty well when you further integrate your test framework into a continuous integration tool. BuildingPy is a good example of this, though it's not implemented in CI tools, it's still a good tool.

Here's a reference of a good framework design: UnitTesting: A Library for Modelica Unit Testing

Hang Yu
  • 476
  • 2
  • 14
3

If you have Mathematica and SystemModeler you can run the simulation from Mathematica and use the VerificationTest "function" to test: VerificationTest[Abs[WSMSimulate["HelloWorld"]["x", .1] - .90] < .01].

Multiple tests can then be simulated in a TestReport[].

gwr
  • 465
  • 6
  • 17
jrhodin
  • 659
  • 5
  • 14