Introduction to Parallel Programming with MPI: Automated Testing

Automated Testing

Automated testing is very helpful when developing your application. It allows you to quickly and easily check the correctness of your application while making changes to it. It’s easy to make mistakes, and debugging an entire application takes a significant amount of time. As a bonus, when the code is well covered by automated tests, it will always work correctly, at least in the ways you have defined in the tests.

In the ideal scenario, you start from a correctly working serial application that already includes a full suite of tests for each function. If this is not the case, it is best to start by writing tests for each function in the files you wish to parallellise. If there are long functions, it may be a good idea to split them into smaller ones and test these separately.

Unit Tests and Integration Tests

What we are doing here is called unit testing, testing individual small functions, or units. This is useful for developing from an already correct implementation of an algorithm. A failed test will immediately point out the location of a bug, and the smaller the units, the easier it is to find the problem. Smaller units also allow each test to run faster.

You might also wish to test the correctness of the whole algorithm. This can be done in essentially the same way, but running the entire program or a large section of it in the test. These integration tests are often separated from the unit tests and can take significantly longer to run.

Especially in scientific applications, the correct output is often not known. In these cases you can test for known properties of the output, or test the results compared to a run of the serial version.

Unit testing

In the following examples and challenges we use the Cmocka unit testing framework. Testing frameworks in general use similar concepts and any lessons learned here can be applied to any framework.

Make sure you have the Cmocka library installed. Take a look at the setup page for details.

Here’s an simple example of a test suite with a single test:

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmocka.h>

/* A test that checks if 1 is true */
static void test_1_true(void **state) {
   assert_true(1);
}

/* In the main function create the list of the tests */
int main(void) {
   const struct CMUnitTest tests[] = {
      cmocka_unit_test(test_1_true),
   };

   // Call a library function that will run the tests
   return cmocka_run_group_tests(tests, NULL, NULL);
}

Try compiling this using

gcc file_name.c -lcmocka

and running the resulting binary.

In the following examples and challenges we use the FRUIT unit testing framework. Testing frameworks in general use similar concepts and any lessons learned here can be applied to any framework.

You will need one file from the FRUIT ZIP archive. If you haven’t downloaded it yet, take a look at the setup for details.

Copy fruit.f90 from the src folder in the ZIP archive to your working directory. Compile it with

gfortran -c fruit.f90

This creates two modules, fruit.mod and fruit_util.mod, and an object file, fruit.o.

Tests are defined in test modules. Here is a very simple example for making sure that 1 is 1.

module example_test
  use fruit
  implicit none

contains
  subroutine test_true
    integer:: result

    call assert_true( 1==1, "1 is 1")
  end subroutine test_true
end module example_test

We also need to define a program to run, called a driver. In addition to some FRUIT related calls, this will contain calls to all your tests.

program fruit_driver
  use fruit
  use example_test
  call init_fruit
  call test_true
  call fruit_summary
  call fruit_finalize
end program fruit_driver

Compile this and the test module, here in example_test.F08, and the fruit.o object file

gfortran example_driver.F08 example_test.F08 fruit.o

and run the resulting binary.

The program will run our little test and produce some statistics about it.

Test suites automate and simplify testing

A test suite is a program that runs your functions and checks that the output is correct. You check the output using a set of assert functions. You assert that certain statements are true or that certain numbers are equal. The test suite will report a failure if one or more of the assert statements fail.

It is good practice to write to name the test functions according to the function it is testing, and to give more information if there is more than one test for the function. Don’t be afraid of using long function names. When the test suite reports a failure, it will report the name of the test function. This way you know exactly where in your code the problem is.

Tests

The following example code tests the find_sum function from a previous example.

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmocka.h>
#include <stdio.h>

// Calculate the sum of numbers in a vector
double find_sum( double * vector, int N ){
  double sum = 0;
  for( int i=0; i<N; i++){
     sum += vector[i];
  }
  return sum;
}

// Find the maximum of numbers in a vector
double find_maximum( double * vector, int N ){
  double max = 0;
  for( int i=0; i<N; i++){
     if( vector[i] > max ){
        max = vector[i];
     }
  }
  return max;
}


/* Test the find_sum function */
static void test_find_sum(void **state) {
  int n_numbers = 1024;
  double vector[n_numbers];
  double sum;

  // Generate a vector
  for( int i=0; i<n_numbers; i++){
     vector[i] = i;
  }

  //Find the sum and check it's correct
  sum = find_sum( vector, n_numbers );
  assert_true( sum == 523776 );
}


int main(void) {
  const struct CMUnitTest tests[] = {
     cmocka_unit_test(test_find_sum),
  };
  return cmocka_run_group_tests(tests, NULL, NULL);
} 

Test module:

module example_test
 use fruit
 implicit none

 contains

   subroutine find_sum( vector, N, sum )
     real, intent(in) :: vector(:)
     real, intent(inout) :: sum
     integer, intent(in) :: N
     integer i
     
     sum = 0
     do i = 1, N
       sum = sum + vector(i)
     end do

   end subroutine find_sum


   subroutine find_max( vector, N, max )
     real, intent(in) :: vector(:)
     real, intent(inout) :: max
     integer, intent(in) :: N
     integer i
     
     max = 0
     do i = 1, N
       if (max < vector(i)) then
         max = vector(i)
       end if
     end do

   end subroutine find_max


   subroutine test_sum
     integer, parameter :: N=10
     integer i
     real vector(N)
     real sum

     do i = 1, N
       vector(i) = i
     end do
 
     call find_sum( vector, N, sum )

     call assert_true( sum == 55, "Test Sum")
   end subroutine test_sum

end module example_test

Driver Program:

program fruit_driver
 use fruit
 use example_test
 call init_fruit
 call test_sum
 call fruit_summary
 call fruit_finalize
end program fruit_driver

Write a test for the find_maximum function as well and run both tests.

Solution

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmocka.h>
#include <stdio.h>

// Calculate the sum of numbers in a vector
double find_sum( double * vector, int N ){
  double sum = 0;
  for( int i=0; i<N; i++){
     sum += vector[i];
  }
  return sum;
}

// Find the maximum of numbers in a vector
double find_maximum( double * vector, int N ){
  double max = 0;
  for( int i=0; i<N; i++){
     if( vector[i] > max ){
        max = vector[i];
     }
  }
  return max;
}


/* Test the sum function */
static void test_find_sum(void **state) {
  int n_numbers = 1024;
  double vector[n_numbers];
  double sum;

  // Generate a vector
  for( int i=0; i<n_numbers; i++){
     vector[i] = i;
  }

  //Find the sum and check it's correct
  sum = find_sum( vector, n_numbers );
  assert_true( sum == 523776 );
}

/* Test find_maximum */
static void test_find_maximum(void **state) {
  int n_numbers = 1024;
  double vector[n_numbers];
  double max;

  // Generate a vector
  for( int i=0; i<n_numbers; i++){
     vector[i] = i;
  }

  //Find the sum and check it's correct
  max = find_maximum( vector, n_numbers );
  assert_true( max == 1023 );
}


int main(void) {
  const struct CMUnitTest tests[] = {
     cmocka_unit_test(test_find_sum),
     cmocka_unit_test(test_find_maximum),
  };
  return cmocka_run_group_tests(tests, NULL, NULL);
}

Solution

Add a second test subroutine to the test module

   subroutine test_max
     integer, parameter :: N=10
     integer i
     real vector(N)
     real max

     do i = 1, N
       vector(i) = i
     end do
 
     call find_max( vector, N, max )

     call assert_true( max == 10, "Test Max")
   end subroutine test_max

And add the subroutine call to the driver program

program fruit_driver
 use fruit
 use example_test
 call init_fruit
 call test_sum
 call test_max
 call fruit_summary
 call fruit_finalize
end program fruit_driver

Setup and teardown

Above we had to create a vector separately for both tests. In a larger program, writing a test can require a large amount of setup. The environment and the parameters need to mimic the way the function is called in the code.

Doing these separately in each test results in a lot of repeated code. This can be avoided using setup and teardown routines, run before and after each test.

The groups in Cmocka exist simplify running multiple tests with similar inputs or environments. In solution to the exercise we added both tests to the same group and used cmocka_run_group_tests to run both tests.

Test groups also share setup and teardown functions.

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmocka.h>
#include <stdio.h>
#include <stdlib.h>

// Calculate the sum of numbers in a vector
double find_sum( double * vector, int N ){
 double sum = 0;
 for( int i=0; i<N; i++){
    sum += vector[i];
 }
 return sum;
}

// Find the maximum of numbers in a vector
double find_maximum( double * vector, int N ){
 double max = 0;
 for( int i=0; i<N; i++){
    if( vector[i] > max ){
       max = vector[i];
    }
 }
 return max;
}


/* Test the sum function */
static void test_find_sum(void **state) {
 double * state_vector = *state;
 double * vector = state_vector +1;
 int n_numbers = state_vector[0];
 double sum;

 //Find the sum and check it's correct
 sum = find_sum( vector+1, n_numbers );
 assert_true( sum == 523776 );
}

/* Test find_maximum */
static void test_find_maximum(void **state) {
 double * state_vector = *state;
 double * vector = state_vector +1;
 int n_numbers = state_vector[0];
 double max;

 //Find the sum and check it's correct
 max = find_maximum( vector+1, n_numbers );
 assert_true( max == 1023 );
}


static int setup(void **state)
{
 int n_numbers = 1024;
 double * state_vector = malloc( (n_numbers+1)*sizeof(double) );
 state[0] = (void *) state_vector;
 double * vector = state_vector + 1;

 // vector[0] will contain n_numbers
 state_vector[0] = n_numbers;
 
 // Generate a vector
 for( int i=0; i<n_numbers; i++){
    vector[i] = i;
 }

 return 0;
}

static int teardown(void **state)
{
 free( state[0] );
 return 0;
}

int main(void) {
 const struct CMUnitTest tests[] = {
    cmocka_unit_test(test_find_sum),
    cmocka_unit_test(test_find_maximum),
 };
 return cmocka_run_group_tests(tests, setup, teardown);
}

We probably didn’t save any lines of code by doing this. Actually we may have added some. But in a longer set of tests, this can save a lot of coding and allows you to change all of the tests at once.

Setup and Teardown in FRUIT

Since FRUIT allows us to call the test functions directly, calling a setup and a teardown function is simple. First, let’s add them to the test module and update the tests accordingly

   subroutine setup( vector, N )
     real, intent(out) :: vector(:)
     integer, intent(in) :: N
     integer i

     ! Set the vector
     do i = 1, N
       vector(i) = i
     end do
   end subroutine setup

   subroutine teardown()
     ! Nothing to do here
   end subroutine teardown


   subroutine test_sum(vector, N)
     real, intent(in) :: vector(:)
     integer, intent(in) :: N
     integer i
     real sum

     call find_sum( vector, N, sum )

     call assert_true( sum == 55, "Test Sum")
   end subroutine test_sum


   subroutine test_max(vector, N)
     real, intent(in) :: vector(:)
     integer, intent(in) :: N
     integer i
     real max

     call find_max( vector, N, max )

     call assert_true( max == 10, "Test Max")
   end subroutine test_max

We can then call them in the driver

program fruit_driver
 use fruit
 use example_test
 implicit none

 integer, parameter :: N=10
 real vector(N)
 call init_fruit

 call setup(vector, N)
 call test_sum(vector, N)
 call teardown()

 call setup(vector, N)
 call test_max(vector, N)
 call teardown()

 call fruit_summary
 call fruit_finalize
end program fruit_driver

In this case the tests do not change the vector, so there is no real need to rerun setup, but we might want to change this later, so it’s good to be careful. In general, we don’t want previous test to be able affect the current one.

FRUIT also contains methods for automatically creating test groups, called baskets, and generating the driver to run them. These features use ruby, which you need to install separately. It’s well worth looking into the features of this tool if you plan to develop in Fortran.

You will notice that most test consist of three parts. First they set up the inputs and the environment to mock how the function might be called in the actual program. They then run the function and finally do a set of checks on the output and the environment.

Testing an MPI Code

You can test a function that uses MPI in the same way. The main differences are that you need to set up the MPI environment and mock the data and the environment for each rank.

It’s not good practice to include MPI_Init or MPI_Finalize in the test function itself. If an assert is triggered, MPI_Finalize may not be run, which could in turn ruin the rest of the tests. Instead call them either in the setup and teardown functions or in the main program.

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmocka.h>
#include <stdio.h>
#include <mpi.h>
#include <stdlib.h>

// Calculate the sum of numbers in a vector
double find_sum( double * vector, int N ){
   double sum = 0;
   double global_sum;

   // Calculate the sum on this rank as befor
   for( int i=0; i<N; i++){
      sum += vector[i];
   }

   // Call MPI_Allreduce to find the full sum
   MPI_Allreduce( &sum, &global_sum, 1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD );

   return global_sum;
}

// Find the maximum of numbers in a vector
double find_maximum( double * vector, int N ){
   double max = 0;
   double global_max;

   // Calculate the maximum on this rank as before
   for( int i=0; i<N; i++){
      if( vector[i] > max ){
         max = vector[i];
      }
   }

   // Call MPI_Allreduce to find the maximum over all the ranks
   MPI_Allreduce( &max, &global_max, 1, MPI_DOUBLE, MPI_MAX, MPI_COMM_WORLD );

   return global_max;
}


/* Test the find_maximum function */
static void test_find_maximum(void **state) {
   double * state_vector, * vector;
   state_vector = state[0];
   vector = state_vector+1;
   int n_numbers = state_vector[0];
   double max;
   int n_ranks;

   //Find the sum and check it's correct
   max = find_maximum( vector+1, n_numbers );

   MPI_Comm_size(MPI_COMM_WORLD, &n_ranks);

   assert_true( max == n_ranks*n_numbers - 1 );
}


static int setup(void **state)
{
   int n_numbers = 1024;
   int rank, my_first_number;
   double * state_vector, * vector;
   
   // Start by calling MPI_Init with mocked input
   char *argv[] = {"mpi_test"};
   int argc = 1;
   MPI_Init(&argc, (char ***) &argv);
   
   state_vector = malloc( (n_numbers+1)*sizeof(double) );
   state[0] = (void *) state_vector;
   vector = state_vector + 1;

   // vector[0] will contain n_numbers
   state_vector[0] = n_numbers;

   // Each rank will have n_numbers numbers,
   // starting from where the previous left
   MPI_Comm_rank(MPI_COMM_WORLD, &rank);
   my_first_number = n_numbers*rank;

   // Generate a vector
   for( int i=0; i<n_numbers; i++){
      vector[i] = my_first_number + i;
   }

   return 0;
}

static int teardown(void **state)
{
   free( state[0] );

   // End by calling MPI_Finalize
   MPI_Finalize();

   return 0;
}


int main(int argc, char** argv) {
   int cmocka_return_code;

   const struct CMUnitTest tests[] = {
      cmocka_unit_test(test_find_maximum),
   };
   
   cmocka_return_code = cmocka_run_group_tests(tests, setup, teardown);

   return cmocka_return_code;
}

Test module:

module example_test
 use fruit
 implicit none
 include "mpif.h" 

contains

 subroutine find_sum( vector, N, global_sum )
   real, intent(in) :: vector(:)
   real, intent(inout) :: global_sum
   real sum
   integer, intent(in) :: N
   integer i, ierr
    
   sum = 0
   do i = 1, N
     sum = sum + vector(i)
   end do

   ! Call MPI_Allreduce to find the full sum
   call MPI_Allreduce( sum, global_sum, 1, MPI_REAL, MPI_SUM, MPI_COMM_WORLD, ierr )

 end subroutine find_sum

 ! Find the maximum of numbers in a vector
 subroutine find_max( vector, N, global_max )
   real, intent(in) :: vector(:)
   real, intent(inout) :: global_max
   real max
   integer, intent(in) :: N
   integer i, ierr
    
   max = 0
   do i = 1, N
      if (max < vector(i)) then
         max = vector(i)
      end if
   end do

   ! Call MPI_Allreduce to find the full sum
   call MPI_Allreduce( max, global_max, 1, MPI_REAL, MPI_MAX, MPI_COMM_WORLD, ierr )

 end subroutine find_max


 ! Setup routine: call MPI_Init and create the vector
 subroutine setup( vector, n_numbers )
   real, intent(inout) :: vector(:)
   integer, intent(in) :: n_numbers
  
   integer rank, ierr
   real my_first_number
   integer i

   ! Get my rank
   call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)

   ! Each rank will have n_numbers numbers,
   ! starting from where the previous left off
   my_first_number = n_numbers*rank;

   ! Set the vector
   do i = 1, n_numbers
     vector(i) = my_first_number + i
   end do
 end subroutine setup

 subroutine teardown()
   ! Nothing to do here
 end subroutine teardown


 subroutine test_max(vector, N)
   real, intent(in) :: vector(:)
   integer, intent(in) :: N
   integer i, n_ranks, ierr
   real max

   call find_max( vector, N, max )

   ! Find the correct value using a simple serial method
   call MPI_Comm_size(MPI_COMM_WORLD, n_ranks, ierr)

   call assert_true( max == n_ranks*N, "Test Max")
 end subroutine test_max

end module example_test

Driver Program:

program fruit_driver
 use fruit
 use example_test
 implicit none

 integer, parameter :: N=10
 integer ierr
 real vector(N)

 call init_fruit

 ! Start with MPI_Init
 call MPI_Init(ierr)

 call setup(vector, N)
 call test_max(vector, N)
 call teardown()

 ! Call MPI_Finalize at the end
 call MPI_Finalize(ierr)

 call fruit_summary
 call fruit_finalize
end program fruit_driver

Testing MPI

Implement a test of the find_sum function in the above example

Solution

The test function:

/* Test the find_sum_ function */
static void test_find_sum(void **state) {
  double * state_vector, * vector;
  state_vector = state[0];
  vector = state_vector+1;
  int n_numbers = state_vector[0];
  double sum, correct_sum;
  int n_ranks;

  //Find the sum
  sum = find_sum( vector, n_numbers );
  
  //Find the correct value using a simple serial method
  MPI_Comm_size(MPI_COMM_WORLD, &n_ranks);
  correct_sum = 0;
  for( int i=0; i<n_ranks*n_numbers; i++){
    correct_sum += i;
  }
  
  assert_true( sum == correct_sum );
}

Add the test to the list in the main function

int main(int argc, char** argv) {
  int cmocka_return_code;

  const struct CMUnitTest tests[] = {
     cmocka_unit_test(test_find_maximum),
     cmocka_unit_test(test_find_sum),
  };
  
  cmocka_return_code = cmocka_run_group_tests(tests, setup, teardown);

  return cmocka_return_code;
}

Solution

Add the test function to the test module

 subroutine test_max(vector, N)
   real, intent(in) :: vector(:)
   integer, intent(in) :: N
   integer i
   real max

   call find_max( vector, N, max )

   write(6,*) "Maximum = ", max

   call assert_true( max == 20, "Test Max")
 end subroutine test_max

Driver Program:

program fruit_driver
 use fruit
 use example_test
 implicit none

 integer, parameter :: N=10
 integer ierr
 real vector(N)

 call init_fruit

 ! Start with MPI_Init
 call MPI_Init(ierr)

 call setup(vector, N)
 call test_max(vector, N)
 call teardown()

 call setup(vector, N)
 call test_sum(vector, N)
 call teardown()

 ! Call MPI_Finalize at the end
 call MPI_Finalize(ierr)

 call fruit_summary
 call fruit_finalize
end program fruit_driver

Since all processes are running the tests, you will see multiple copies of the each of the lines printed in the terminal. The test suite is not aware of MPI—it’s just running multiple copies of the tests. This is not necessarily a problem. A test can fail on one rank and succeed on a different one. In general you want to test all the ranks.

The --tag-output option for mpirun is useful here. It will append rank numbers to every line printed by the application. This helps in keeping track of the ranks with errors.

Some tests might make sense only on a specific rank. You can use if statements to run asserts only on that ranks. The code in a test is normal general purpose code. It’s good to keep them simple, though.

Another important caveat is the order of floating point operations. Since the order of operations ofter depends on the number of ranks, and floating point operations are not associative, different rounding errors can accumulate with different rank counts. In most cases it’s sufficient to allow some rounding error. In very chaotic algorithms, it may not be possible to compare the results. This does not necessarily mean that the implementation is incorrect.

Floating Point Ordering

Set the vector lenght to 2014 in the above test and increase the number of ranks. At some point the test of the find_sum function will start to fail.

• Why does it fail? What is the difference between the serial and the MPI versions?
• Is this a problem? How would you fix it?
• Let’s say we don’t care about the small difference. Change the test to only report differences larger than 0.0001.