9. One-Sided Communication

One-sided communication is a communication paradigm that allows processes to directly access and update the memory of other processes without explicit coordination. It is also called Remote Memory Access (RMA).

9.1. Window Allocation

The target processes specifies a window of its memory, and the origin processes can perform operations like putting data into that window, getting data from that window, or accumulating data in that window. A process can be target and origin at the same time within a window.

To enable one-sided communication, an MPI_Win needs to be created. Typically, this is accomplished using the MPI_Win_create or MPI_Win_allocate functions.

MPI_Win_create:

Creating a window of shared memory, associated with the communicator comm.

int MPI_Win_create(void *base, MPI_Aint size, int disp_unit, MPI_Info info, \
                   MPI_Comm comm, MPI_Win *win);

  • base: It is the starting address of the window in the calling process’s address space. This is where the window begins.

  • size: It represents the size of the window. It determines the extent of the memory associated with the window.

  • disp_unit: This argument specifies the unit of displacement for addressing within the window. It is typically the size of the basic data type on which the displacement is based.

  • info: An info object specifying window creation hints. This argument allows you to provide additional information or hints to control the behavior of window creation.

  • comm: It is the communicator associated with the window. The window is created in the context of this communicator.

  • win: This is the resulting window object. It is a handle to the created window, and you will use this handle to refer to the window in subsequent MPI operations.

MPI_Win_Free:

After the window has been used, it should be freed using MPI_Win_free:

MPI_Win_free(win);

9.2. Remote Memory Access

One-sided communication in MPI is facilitated by operations like MPI_Put, MPI_Get, and MPI_Accumulate which are also called RMA operations. The target process specifies a window of its memory, and the origin process can perform operations like putting data into that window MPI_Put, getting data from that window MPI_Get, or accumulating data in that window MPI_Accumulate.

MPI_Put

Copies data from a local memory buffer to a remote memory window.

int MPI_Put(const void *origin_addr, int origin_count, \
            MPI_Datatype origin_datatype, int target_rank, \
            MPI_Aint target_disp, int target_count, \
            MPI_Datatype target_datatype, MPI_Win win);

  • origin_addr: A pointer to the starting address of the data in the memory of the origin process. This is the data to be copied to the target process.

  • origin_count: The number of data elements to be sent from the origin process. It specifies how many elements of the specified origin_datatype are to be sent.

  • origin_datatype: The datatype of each element in the origin_addr buffer. It defines the interpretation of the data in the origin buffer.

  • target_rank: The rank of the target process in the communicator associated with the window win. This specifies the process whose memory will receive the data.

  • target_disp: The displacement (in bytes) within the target process’s memory window win where the data will be copied. It’s the offset from the beginning of the window.

  • target_count: The number of data elements to be received by the target process. It specifies how many elements of the specified target_datatype are expected to be received.

  • target_datatype: The datatype of each element in the target buffer. It defines the interpretation of the data in the target buffer.

  • win: The MPI window object that represents a region of memory. It specifies the memory that can be accessed using one-sided communication operations.

MPI_Get

Copies data from a remote memory window to a local memory buffer.

int MPI_Get(void *origin_addr, int origin_count, \
            MPI_Datatype origin_datatype, int target_rank, \
            MPI_Aint target_disp, int target_count, \
            MPI_Datatype target_datatype, MPI_Win win);

MPI_Accumulate

Combines data from a local buffer with the data in a remote memory window using an associative and commutative operation.

int MPI_Accumulate(const void *origin_addr, int origin_count, \
                   MPI_Datatype origin_datatype, int target_rank, \
                   MPI_Aint target_disp, int target_count, \
                   MPI_Datatype target_datatype, MPI_Op op, MPI_Win win);

In the one-sided communication paracdigm, three different synchronization mechanisms (p.588) are used. one of them is active target synchronization and the following MPI function is used for this kind of synchronization:

MPI_Win_fence:

This function completes all preceding RMA operations on the specified window, ensuring availability of all local and remote operations.

MPI_Win_fence(int assert, MPI_Win win);
mpi_tree_broadcast

Fig. 9.2.1 One-sided communication between two processes

Task

Consider Fig. 9.2.1, write three different executable C/C++ programs for each of the three items (a, b, c) with the appropriate MPI communication. Utilize active target synchronization with fences and consider the transferred data is just an integer value.

9.3. Synchronization Mechanisms

Let’s say we have a buffer called buf. The buffer’s size in the process with rank 0 is equal to the number of processes. On all other processes buf may hold just one value. Initially we set the buffers’ values to rank * 10. The objective is to use MPI_Put so that rank 0 gathers the values of all other ranks. For instance, if we have 4 processes, the final buf on rank 0 should look like: [0.0, 10.0, 20.0, 30.0].

Task

Implement the described procedure using:

  1. General active target synchronization.

  2. Passive target synchronization.