The mpinode module is a novel runtime mechanism of OMPi that treats nodes in a cluster as separate offloading devices. Essentially, the user may employ the OpenMP device interface (OpenMP versions ≥ 4.0) to offload portions of the application code to remote nodes, transparently (see the related publication).

In what follows, it is assumed that OMPi will be installed on a homogeneous cluster.

Requirements

In order to use the mpinode module, an MPI installation with multiple threads support is required. The module was initially developed and tested with OpenMPI 3.0. It was also successfully tested with MPICH 3.4.2 and 4.0.2.

» MPI must be installed on the node that will act as the “host” (which will execute the main part of an application) and on every other node that will be used as a “device” (where code will be offloaded to). Please note that a correct MPI installation is assumed; it is very easy to encounter problems, especially when multiple MPI implementations are installed on the same machine.

» Support for multiple threads can be verified as follows:

• If OpenMPI is installed, execute the following and make sure that the output includes “MPI_THREAD_MULTIPLE: yes“.

ompi_info | grep -i thread

• In any other case, compile and execute a simple program like the one given below:
Simple program to check for multithread support

#include <stdio.h>
#include <mpi.h>

int main(void)
{
    int thr_provided_level;
    MPI_Init_thread(NULL, NULL, MPI_THREAD_MULTIPLE, &thr_provided_level);

    if (thr_provided_level == MPI_THREAD_MULTIPLE) 
        printf("Multithread IS supported\n");    
    else 
        printf("Multithread NOT supported\n"); 

    MPI_Finalize();
    return 0;
}

Installation

To build mpinode, MPI support needs to be enabled when configuring OMPi:

./configure --prefix=‹install-dir› --enable-mpi

Then OMPi can be compiled and installed as usual:

make 
make install

Usage

Specifying cluster nodes

After installing OMPi, you need to provide a simple text configuration file which contains the names or IP addresses of the participating nodes which will act as OpenMP “devices”. In fact, an empty file named .ompi_mpi_nodes is created on your home directory, when configuring OMPi with MPI enabled. Edit this file and add the names or the IP addresses of the cluster nodes you would like to utilize, one node per line.

» Avoid using localhost as the name of the host node as it has been found to cause problems in some MPI installations.

» To check correctness, just examine that the output of the following command agrees with the contents of the ~/.ompi_mpi_nodes file:

ompicc --devvifo

» The above command also reveals the numeric device IDs that should be utilized in device() clauses of target regions.

1 configured device module(s): mpinode

MODULE [mpinode]:
------
OMPi mpinode device module.
  To configure devices for this module, edit ~/.ompi_mpi_nodes.
  Put node hostnames or IP addresses in separate lines.
Available devices : 0
------

Total number of available devices: 0

1 configured device module(s): mpinode

MODULE [mpinode]:
------
OMPi mpinode device module.
  To configure devices for this module, edit ~/.ompi_mpi_nodes.
  Put node hostnames or IP addresses in separate lines.
Available devices : 3

  device id < 1 > { 
    node: node0
  }
  device id < 2 > { 
    node: node1
  }
  device id < 3 > { 
    node: node2
  }
------

Total number of available devices: 3

Compiling and executing applications

In order to compile an OpenMP program you simply go:

ompicc program.c

The best way to run the executable is through the mpiexec command:

mpiexec -n 1 ./a.out

OpenMPI may allow you to execute ./a.out directly, but the execution through mpiexec works universally.

#include <stdio.h>
#include <omp.h>

int main(void)
{
    /* 
     * Expected result (assuming that device with ID 1 is an mpinode device):
     * Running on mpinode device 
     */
    #pragma omp target device(1)
    {
        if (omp_is_initial_device()) 
            printf("Running on host\n");    
        else 
            printf("Running on mpinode device\n"); 
    }
    return 0;
}