The mpinode module is a novel runtime mechanism of OMPi that treats nodes in a cluster as separate offloading devices. Essentially, the user may employ the OpenMP device interface (OpenMP versions ≥ 4.0) to offload portions of the application code to remote nodes, transparently (see the related publication).
In what follows, it is assumed that OMPi will be installed on a homogeneous cluster.
Requirements
In order to use the mpinode module, an MPI installation with multiple threads support is required. The module was initially developed and tested with OpenMPI 3.0. It was also successfully tested with MPICH 3.4.2 and 4.0.2.
» MPI must be installed on the node that will act as the “host” (which will execute the main part of an application) and on every other node that will be used as a “device” (where code will be offloaded to). Please note that a correct MPI installation is assumed; it is very easy to encounter problems, especially when multiple MPI implementations are installed on the same machine.
» Support for multiple threads can be verified as follows:
- If OpenMPI is installed, execute the following and make sure that the output includes “
MPI_THREAD_MULTIPLE: yes
“.
ompi_info | grep -i thread
Installation
To build mpinode, MPI support needs to be enabled when configuring OMPi:
./configure --prefix=‹install-dir› --enable-mpi
Then OMPi can be compiled and installed as usual:
make make install
Usage
Specifying cluster nodes
After installing OMPi, you need to provide a simple text configuration file which contains the names or IP addresses of the participating nodes which will act as OpenMP “devices”. In fact, an empty file named .ompi_mpi_nodes
is created on your home directory, when configuring OMPi with MPI enabled. Edit this file and add the names or the IP addresses of the cluster nodes you would like to utilize, one node per line.
» Avoid using localhost
as the name of the host node as it has been found to cause problems in some MPI installations.
» To check correctness, just examine that the output of the following command agrees with the contents of the ~/.ompi_mpi_nodes
file:
ompicc --devvifo
» The above command also reveals the numeric device IDs that should be utilized in device()
clauses of target
regions.
Compiling and executing applications
In order to compile an OpenMP program you simply go:
ompicc program.c
The best way to run the executable is through the mpiexec
command:
mpiexec -n 1 ./a.out
OpenMPI may allow you to execute ./a.out
directly, but the execution through mpiexec
works universally.