I'm working with a fairly big MPI code. I started to include unit tests into the existing code base. However, as soon as the unit under test uses an MPI routine the test executable crashes with the error message "calling an MPI routine before MPI_Init"
Yes, it is possible.
As explained in https://github.com/catchorg/Catch2/issues/566, you will have to supply a custom main function.
#define CATCH_CONFIG_RUNNER
#include "catch.hpp"
#include <mpi.h>
int main( int argc, char* argv[] ) {
MPI_Init(&argc, &argv);
int result = Catch::Session().run( argc, argv );
MPI_Finalize();
return result;
}
To amplify your experience using Catch2 in combination with MPI, you might want to avoid redundant console output. That requires injecting some code into ConsoleReporter::testRunEnded of catch.hpp.
#include <mpi.h>
void ConsoleReporter::testRunEnded(TestRunStats const& _testRunStats) {
int rank id = -1;
MPI Comm rank(MPI COMM WORLD,&rank id);
if(rank id != 0 && testRunStats.totals.testCases.allPassed())
return;
printTotalsDivider(_testRunStats.totals);
printTotals(_testRunStats.totals);
stream << std::endl;
StreamingReporterBase::testRunEnded(_testRunStats);
}
Finally you might also want to execute your test cases with a different number of MPI ranks. I found the following to be an easy and well working solution:
SCENARIO("Sequential Testing", "[1rank]") {
// Perform sequential tests here
}
SCENARIO("Parallel Testing", "[2ranks]") {
// Perform parallel tests here
}
Then you can call the tagges scenarios individually with
mpiexec -1 ./application [1rank]
mpiexec -2 ./application [2rank]
For anyone looking to remove all duplicated console output when running Catch2 distributed, here's a solution.
Find the definition of ConsoleReporter in catch.hpp (in v2.10.0, it's at line 15896). It will look something like:
ConsoleReporter::ConsoleReporter(ReporterConfig const& config)
: StreamingReporterBase(config),
m_tablePrinter(new TablePrinter(config.stream(),
[&config]() -> std::vector<ColumnInfo> {
if (config.fullConfig()->benchmarkNoAnalysis())
{
return{
{ "benchmark name", CATCH_CONFIG_CONSOLE_WIDTH - 43, ColumnInfo::Left },
{ " samples", 14, ColumnInfo::Right },
{ " iterations", 14, ColumnInfo::Right },
{ " mean", 14, ColumnInfo::Right }
};
}
else
{
return{
{ "benchmark name", CATCH_CONFIG_CONSOLE_WIDTH - 32, ColumnInfo::Left },
{ "samples mean std dev", 14, ColumnInfo::Right },
{ "iterations low mean low std dev", 14, ColumnInfo::Right },
{ "estimated high mean high std dev", 14, ColumnInfo::Right }
};
}
}())) {}
ConsoleReporter::~ConsoleReporter() = default;
Though not shown here, the base-class StreamingReporterBase provides a stream attribute which we'll disable, by the failbit trick shown here.
Inside the final {} above (an empty constructor definition), insert:
// I solemnly swear that I am up to no good
int rank;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
// silence non-root nodes
if (rank != 0)
stream.setstate(std::ios_base::failbit);
You can see an example on this repo.
When all tests passed, I only need to hear about it once (e.g. by the master rank). But I found that if a test fails, it's still beneficial to know on which rank it failed and how. My solution does not require manipulating the catch.hpp file, instead it only requires the custom main to look something like this:
#define CATCH_CONFIG_RUNNER
#include "catch.hpp"
#include "mpiHelpers.hpp"
#include <sstream>
int main( int argc, char* argv[] ) {
MPI_Init_H autoInit { argc, argv }; // calls MPI_Finalize() on destruction
std::stringstream ss;
/* save old buffer and redirect output to string stream */
auto cout_buf = std::cout.rdbuf( ss.rdbuf() );
int result = Catch::Session().run( argc, argv );
/* reset buffer */
std::cout.rdbuf( cout_buf );
MPI_Comm_H world { MPI_COMM_WORLD };
std::stringstream printRank;
printRank << "Rank ";
printRank.width(2);
printRank << std::right << world.rank() << ":\n";
for ( int i{1}; i<world.size(); ++i ){
MPI_Barrier(world);
if ( i == world.rank() ){
/* if all tests are passed, it's enough if we hear that from
* the master. Otherwise, print results */
if ( ss.str().rfind("All tests passed") == std::string::npos )
std::cout << printRank.str() + ss.str();
}
}
/* have master print last, because it's the one with the most assertions */
MPI_Barrier(world);
if ( world.isMaster() )
std::cout << printRank.str() + ss.str();
return result;
}
So I'm simply redirecting the output to a string stream buffer. Then I can decide later whether or not to print them.
The two helper classes MPI_Init_H and MPI_Comm_H are not really necessary, you can do it with the standard MPI_Init and MPI_Comm, but for completeness, here they are:
#ifndef MPI_HELPERS
#define MPI_HELPERS
#include <iostream>
#include <mpi.h>
class MPI_Init_H {
public:
/* constructor initialises MPI */
MPI_Init_H( int argc, char* argv[] ){
MPI_Init(&argc, &argv);
}
/* destructor finalises MPI */
~MPI_Init_H(){ MPI_Finalize(); }
};
/* content of mpiH_Comm.hpp */
class MPI_Comm_H
{
private:
MPI_Comm m_comm;
int m_size;
int m_rank;
public:
MPI_Comm_H( MPI_Comm comm = MPI_COMM_NULL ) :
m_comm {comm}
{
if ( m_comm != MPI_COMM_NULL ){
MPI_Comm_size(m_comm, &m_size);
MPI_Comm_rank(m_comm, &m_rank);
}
else {
m_size = 0;
m_rank = -1;
}
}
/* contextual conversion to bool, which returns true if m_comm is a valid
* communicator and false if it is MPI_COMM_NULL */
operator bool() const { return m_comm != MPI_COMM_NULL; }
const MPI_Comm& comm() const {
#ifndef NDEBUG
if ( !(*this) )
std::cerr
<< "WARNING: You called comm() on a null communicator!\n";
#endif
return m_comm;
}
int rank() const {
return m_rank;
}
int size() const {
assert( *this && "You called size() on a null communicator!");
return m_size;
}
int master() const {
assert( *this && "You called master() on a null communicator!");
return 0;
}
bool isMaster() const { return rank() == 0; }
/* allow conversion to MPI_Comm */
operator const MPI_Comm&() const { return m_comm; }
};
#endif
Apart from supplying a custom main function, the oft-encountered issue of duplicate output from multiple MPI processes can be circumvented by forcing each MPI process to dump their test reports into separate files, e.g. process p dumps it's test report to report_p.xml.
A quick and dirty approach would be to extend the vector of command line arguments argv with extra entries for the process dependent filename paired with a --out.
Source:
// mytests.cpp
#define CATCH_CONFIG_RUNNER
#include "catch.hpp"
#include <mpi.h>
#include <string>
#include <vector>
int
main(int argc, char* argv[]) {
MPI_Init(&argc, &argv);
int mpi_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &mpi_rank);
// make space for two extra arguments
std::vector<const char*> new_argv(argc + 2);
for (int i = 0; i < argc; i++) {
new_argv[i] = argv[i];
}
// set "--out report_p.xml" as last two arguments
auto filename = "report_" + std::to_string(mpi_rank) + ".xml";
new_argv[argc] = "--out";
new_argv[argc+1] = filename.data();
int result = Catch::Session().run(new_argv.size(), new_argv.data());
MPI_Finalize();
return result;
}
Compile and Run:
mpic++ mytests.cpp -o mytests
mpirun -np 4 ./mytests --reporter junit
Expected Output Files
report_0.xml
report_1.xml
report_2.xml
report_3.xml
MPI processes writing to the same file using std::ostream run the risk of breaking XML formatting, crashing JUnit XML parsers and failing CI pipelines even if all the tests pass. Dumping the test report from each process into their own separate file avoids this issue. The separate files can be concatenated later on, if needed.
Even though the extra arguments in the above example have been placed at the end of the argument list, they do not override any previous --out key-value from the command line. Instead, the built-in CLI parser treats it as a duplication of argument keyword and throws an error. So a more inclusive implementation of the above approach would not extend the argument list but replace the argv pointer corresponding to the value of --out with the appended filename.
PS: pwaul and Anti Earth must have done some serious digging into the Catch source code to figure out the smart code injections, salutations to them!
