{"id":1204,"date":"2024-05-21T15:04:55","date_gmt":"2024-05-21T12:04:55","guid":{"rendered":"https:\/\/paragroup.cse.uoi.gr\/wpsite\/?page_id=1204"},"modified":"2025-12-15T16:38:44","modified_gmt":"2025-12-15T14:38:44","slug":"ompi-remote-offloading","status":"publish","type":"page","link":"https:\/\/paragroup.cse.uoi.gr\/wpsite\/software\/ompi\/documentation\/ompi-remote-offloading\/","title":{"rendered":"OMPi: Remote offloading"},"content":{"rendered":"

In cluster environments, OMPi implements a novel runtime mechanism which can exploit the computational resources of any cluster node by treating them as OpenMP devices<\/em>. In particular, both CPUs and accelerators (such as GPUs) present on remote nodes are virtualized and appear as if they\u00a0were devices local to the node that executes the OpenMP application. As such, the developer can employ the well-known OpenMP device interface (OpenMP versions \u2265 4.0) and offload portions of the application code to them; the actual kernel execution is carried on the corresponding remote devices, transparently (see the related publication<\/a>).<\/p>\n

Requirements<\/h2>\n

In order to use the remote offloading mechanism of OMPi, an MPI installation is required<\/strong>. While not necessary, the installation of MPI should preferably\u00a0support\u00a0multiple threads<\/em>. The mechanism was initially developed and tested with OpenMPI 4.1.1. It was also successfully tested with MPICH 4.0.2 and Intel MPI 2018.<\/p>\n

\u00bb MPI must be installed on the node that will act as the “host” (we call this the primary node<\/em>, and it executes the main part of an application), as well as on every other node that contains devices you\u00a0want to offload to (these are the remote or worker nodes<\/em>). Please note that a correct<\/em> MPI installation is assumed; it is very easy to encounter problems, especially when multiple MPI implementations are installed on the same machine.<\/p>\n

\u00bb Support for multiple threads can be verified as follows:<\/p>\n

    \n
  • If OpenMPI is installed, execute the following and make sure that the output includes “MPI_THREAD_MULTIPLE: yes<\/code>“:\n
    ompi_info | grep -i thread<\/pre>\n<\/li>\n<\/ul>\n
      \n
    • In any other case, compile and execute a simple program like the one given below:\n
      <\/span>Simple program to check for multithread support<\/div>
      \n
      #include<\/strong> <stdio.h>\r\n#include<\/strong> <mpi.h>\r\n\r\nint<\/strong> main(void<\/strong>)\r\n{\r\n  int<\/strong> thr_provided_level;\r\n  MPI_Init_thread(NULL, NULL, MPI_THREAD_MULTIPLE, &thr_provided_level);\r\n\r\n  if<\/strong> (thr_provided_level == MPI_THREAD_MULTIPLE) \r\n    printf<\/strong>(\"Multithread IS supported\\n\"); \r\n  else<\/strong> \r\n    printf<\/strong>(\"Multithread is NOT supported\\n\");\r\n\r\n  MPI_Finalize();\r\n  return<\/strong> 0;\r\n}<\/pre>\n<\/div><\/div>\n<\/li>\n<\/ul>\n
      Setting up OMPi<\/h2>\n
      Configuration file: specifying remote devices<\/h3>\n
      Before installing OMPi, you need to provide a simple text configuration file in your home directory (~\/.ompi_remote_devices<\/code>) which contains the names or IP addresses of the participating cluster nodes, along with their devices. The reason for this is that (most probably) OMPi must be installed across all worker nodes, in order to have complete support for all types of available devices.<\/p>\n
      To make it easier, a script that can generate a sample configuration file is located in the utilities<\/code> directory of the OMPi sources. First, you need to execute this script as follows (assuming your current directory is the OMPi sources root):<\/p>\n
      .\/utilities\/remote_offload_create_config.sh<\/pre>\n
      Then, edit\u00a0the generated ~\/.ompi_remote_devices<\/code>\u00a0and add the names or the IP addresses of the cluster nodes along with the devices you would like to utilize.<\/p>\n
      <\/span>Sample .ompi_remote_devices<\/div>
      \n
      \n
      # This is a comment, till the end of the line\r\n# The node syntax is as follows:\r\n# <node ID> { <device type> : <count> [,] <device type> : <count> [,] ... }\r\n\r\n# A node with 6 devices\r\nnode1.mycluster.org {\r\n\u00a0 \u00a0\u00a0cpu: 1,   # remote CPUs are devices, too, and must be listed first\r\n\u00a0 \u00a0\u00a0cuda: 2   # 2 CUDA GPUs\r\n},\r\n\r\nnode2.mycluster.org {\r\n\u00a0 \u00a0\u00a0cpu: 1    # commas are not necessary\r\n\u00a0 \u00a0\u00a0cuda: 1\r\n},\r\n\r\n# IPs can be used instead of hostnames\r\n195.251.100.230 { \r\n\u00a0 \u00a0 cpu: 1\r\n}<\/pre>\n<\/div>\n<\/div><\/div>\n
      Important notes:<\/strong><\/p>\n
      \u00bb The primary node should not<\/strong> be included in the configuration file.<\/p>\n
      \u00bb Make sure to include all the available devices of the remote nodes, including their CPUs as the very first device of every worker node<\/em>.<\/p>\n
      \u00bb Regarding CPUs, there is flexibility in the device count one can specify. For example, if a node has N processors, each with C cores, one can treat the whole node as 1 CPU device; alternatively, each processor can be treated as a separate device, so the CPU device count should be N; finally, if each core is treated as a separate device, NxC CPU devices should be declared. In the current version of OMPi, 1 CPU device per worker node provides the best performance.<\/em><\/p>\n
      Building and installing OMPi<\/h3>\n
      To facilitate the process, we provide a dedicated script that automates building and installing OMPi across the nodes listed in the configuration file. The script is located in the root directory of the OMPi sources and can be executed as:<\/p>\n
      .\/remote_offload_setup.sh --prefix=<installdir> [options]<\/pre>\n
      Apart from all the\u00a0available options provided in the\u00a0configure<\/code> script for a typical OMPi installation, you can also specify any of the following options:<\/p>\n
        \n
      • --cpus-only<\/code> – Use only the worker CPUs as devices<\/li>\n
      • --static-procs<\/code> – Avoid using MPI_Comm_spawn()<\/code> and create single-threaded MPI processes statically<\/li>\n<\/ul>\n
        Usage<\/h2>\n
        After installing OMPi, you can check the correctness of the configuration file by checking that the output of the following command\u00a0agrees with the contents of the ~\/.ompi_remote_devices<\/code> configuration file:<\/p>\n
        ompiconf --devvinfo<\/pre>\n
        \u00bb The above command also reveals the numeric device IDs<\/em> that should be utilized in device()<\/code> clauses of target<\/code> regions.<\/p>\n
        \u00bb If you modify the configuration file, you have to rebuild and re-install OMPi<\/em> in order for changes to take effect<\/p>\n
        Compiling and executing applications<\/h3>\n
        In order to compile an OpenMP program you simply go:<\/p>\n
        ompicc program.c<\/pre>\n
        The best way to run the executable, assuming\u00a0MPI_THREAD_MULTIPLE<\/code> is supported, is through the mpiexec<\/code> command:<\/p>\n
        mpiexec -n 1 .\/a.out<\/pre>\n
        OpenMPI may allow you to execute .\/a.out<\/code> directly, but the execution through mpiexec<\/code> works universally.<\/p>\n
        In case of static, single-threaded MPI, a helper script is generated for you, named ompi-mpirun.sh<\/code> and located in the same directory as a.out<\/code>, which can be run as:<\/p>\n
        .\/ompi-mpirun.sh .\/a.out<\/pre>\n
        <\/span>Simple program to test remote offloading<\/div>
        \n
        The following sample application can be used to verify the\u00a0successful deployment of the\u00a0remote offloading:<\/p>\n
        #include<\/strong> <stdio.h>\r\n#include<\/strong> <omp.h>\r\n\r\nint<\/strong> main(void<\/strong>)\r\n{\r\n    \/* \r\n     * Expected result (assuming that device with ID 0 is a remote device):\r\n     * Running on remote device \r\n     *\/\r\n    #pragma omp target device(0)<\/span><\/strong>\r\n    {<\/span>\r\n        if<\/span> <\/strong>(<\/span>omp_is_initial_device<\/span>())<\/span> \r\n            printf<\/span>(<\/span>\"Running on host<\/span>\\n<\/span>\"<\/span>);<\/span>    \r\n        else<\/span> <\/strong>\r\n            printf<\/span>(<\/span>\"Running on remote device\\n\"); <\/span>\r\n    }\r\n    return<\/strong> 0;\r\n}\r\n<\/span><\/pre>\n<\/div><\/div>\n
        Utilizing the user-level remote offloading API<\/h3>\n
        According to OpenMP specifications, a target<\/code> code region is offloaded for execution to the default device or to any desired device specified by its numerical ID in a device()<\/code> clause. To effectively use the available resources, one would need to remember all the available devices and their IDs. To ease this burden, we provide an API that simplifies managing the available devices. The API currently offers four calls:<\/p>\n
        int<\/strong>   ompx_get_module_num_devices(char<\/strong> *<\/strong>);\r\nint<\/strong>   ompx_get_module_device(char<\/strong> *<\/strong>, int<\/strong>);\r\nchar *<\/strong>ompx_get_device_module_name(int<\/strong>);\r\nint<\/strong>   ompx_get_module_node_info(char<\/strong> *<\/strong>, char<\/strong> *<\/strong>, int<\/strong> *<\/strong>);\r\n<\/pre>\n
        The first two functions focus on helping with work-sharing between devices. When the user wants to target devices of a specific module type (e.g. CUDA GPUs), he\/she can determine their count with ompx_get_module_num_devices()<\/code>. The function ompx_get_module_device()<\/code> accepts as a parameter the module name and the device index among all devices of the specified module across all nodes. The return value is the global device ID that can be used in a device()<\/code> clause.<\/p>\n
        The function ompx_get_device_module_name()<\/code> is mainly provided for testing and verifying the module type that a device ID belongs to.<\/p>\n
        Finally, the function ompx_get_module_node_info()<\/code> provides information regarding a specific module on a specific node. The user passes as parameters the names of the node and the module, as well as a pointer to an integer that upon return will contain the global device ID of the first usable device of that module in the specified node. The function returns the total number of devices of that module in the given node.<\/p>\n
        Using Remote Offloading with SLURM<\/h3>\n
        OMPi has also been tested on a system that is managed by the SLURM workload manager. Each SLURM installation might behave differently and modifications might be required in your case, but this section aims to provide some guidance in case you attempt to use our mechanism through SLURM. Note that the system we used did not allow the salloc<\/code> command.<\/p>\n
        \u00bb Worker nodes become known only after<\/em> executing a program. Building and installing the required modules beforehand, through the provided installation script, is not possible<\/em>. The solution is to install OMPi on a single node, which contains all the required tools and libraries for the required modules. We were able to load the toolkits required on the login node and install OMPi through the script using the option --cpus-only<\/code>, which will not attempt remote connecting to any node.<\/p>\n
        \u00bb The CUDA module contains a verification process that tests the execution of a small program. As the login nodes did not contain a GPU, we also had to manually disable this check.<\/p>\n
        \u00bb When compiling a program with OMPi and remote offloading enabled, a snapshot of the configuration file gets included in the resulting binary. However, during compilation it is impossible to know which nodes will be allocated by SLURM when we submit a job for execution. We provide an option to disable inclusion of the snapshot of the configuration file during installation:<\/p>\n
        .\/remote_offload_setup.sh --prefix=<installdir> --ignore-snapshot [options]<\/pre>\n
        This way, the configuration file will be parsed at runtime instead of compile time. Whenever we submitted jobs and SLURM allocated some nodes, we used a script to modify the configuration file and use the available nodes and their devices. We can provide that script or other assistance with setting up OMPi correctly upon request.<\/p>\n","protected":false},"excerpt":{"rendered":"
        In cluster environments, OMPi implements a novel runtime mechanism which can exploit the computational resources of any cluster node by treating them as OpenMP devices. In particular, both CPUs and accelerators (such as GPUs) present on remote nodes are virtualized and appear as if they\u00a0were devices local to the node that executes the OpenMP application. […]<\/p>\n","protected":false},"author":4,"featured_media":0,"parent":120,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-1204","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/paragroup.cse.uoi.gr\/wpsite\/wp-json\/wp\/v2\/pages\/1204","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/paragroup.cse.uoi.gr\/wpsite\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/paragroup.cse.uoi.gr\/wpsite\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/paragroup.cse.uoi.gr\/wpsite\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/paragroup.cse.uoi.gr\/wpsite\/wp-json\/wp\/v2\/comments?post=1204"}],"version-history":[{"count":46,"href":"https:\/\/paragroup.cse.uoi.gr\/wpsite\/wp-json\/wp\/v2\/pages\/1204\/revisions"}],"predecessor-version":[{"id":1442,"href":"https:\/\/paragroup.cse.uoi.gr\/wpsite\/wp-json\/wp\/v2\/pages\/1204\/revisions\/1442"}],"up":[{"embeddable":true,"href":"https:\/\/paragroup.cse.uoi.gr\/wpsite\/wp-json\/wp\/v2\/pages\/120"}],"wp:attachment":[{"href":"https:\/\/paragroup.cse.uoi.gr\/wpsite\/wp-json\/wp\/v2\/media?parent=1204"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}