Debugging MPI codes with GDB

Simple example of debugging a single process of a multi-process C++ MPI program on Kay is as follows.

Create a file named hello_mpi.cpp with contents

#include <mpi.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
    MPI_Init( &argc, &argv );
    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    printf("Hello from %d, received %d args\n", rank, argc);

    MPI_Finalize();
    return 0;
}

Launch an interactive session on a compute node using

srun -A <account-name> --time=1:00:00 -p DevQ --pty bash

Load the Intel module with

module load intel/2018u4

set the I_MPI_CXX environment variable with

export I_MPI_CXX=icpc

Build the code with -g flag to include symbols and -O0 flag to disable compiler optimizations

mpicxx -g -O0 -o hello_mpi hello_mpi.cpp

When launching the MPI with gdb, will need to set a breakpoint beyond the call to MPI_Init and tell the debugger to run to this point. This can be done by creating a command file and passing it to gdb using the -x flag. For example, create a file named gdb_cmds.txt with

break hello_mpi.cpp:9
run

To run the MPI code and attach the debugger to a single process, use the --multi-prog switch for srun (see "MULTIPLE PROGRAM CONFIGURATION" of the srun manual page for more details). Create a multi-program configuration file named multi_prog.conf with contents

0 hello_mpi 1 2
1 gdb -x gdb_cmds.txt hello_mpi 34
2-3 hello_mpi 2 3

which will launch the hello_mpi program on processes 0, 2 and 3 (with arguments 1 2 on process 0 and arguments 2 3 on processes 2 and 3) and on process 1 a gdb session looking at the hello_mpi program which should run until it gets to line 9 of the code.

From here, regular gdb commands can be used although there appear to be some issues with the formatting in the console. Exiting this can be done using the quit command and it might be necessary to hit Ctrl-C a few times to get back to the console.

Debugging MPI codes with GDB

Supported By