8. Collective Communication

Collective operations involve communication among a group of processes rather than just between pairs of processes. These operations are essential for parallel algorithms and can significantly improve the efficiency of parallel programs by allowing multiple processes to work together.

8.1. Common Operations

  1. MPI_Bcast (Broadcast):

This operation sends a message from one process (the root) to all other processes in the communicator.

MPI_Bcast(void* buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm);

  1. MPI_Scatter (Scatter):

This operation divides data on the root process and distributes the obtained chunks to the other processes in the communicator.

MPI_Scatter(void* sendbuf, int sendcount, MPI_Datatype sendtype,
            void* recvbuf, int recvcount, MPI_Datatype recvtype,
            int root, MPI_Comm comm);

  1. MPI_Gather (Gather):

The inverse of MPI_Scatter, this operation collects data chunks from all processes and sends it to the root process.

MPI_Gather(void* sendbuf, int sendcount, MPI_Datatype sendtype,
           void* recvbuf, int recvcount, MPI_Datatype recvtype,
           int root, MPI_Comm comm);

  1. MPI_Reduce (Reduce):

This operation performs a reduction operation (e.g., sum, product, maximum) on data ditributed across all processes. The result stored on a specified root process.

MPI_Reduce(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype,
           MPI_Op op, int root, MPI_Comm comm);

  1. MPI_Allreduce (Allreduce):

Similar to MPI_Reduce, but the result is broadcasted to all processes.

MPI_Allreduce(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype,
              MPI_Op op, MPI_Comm comm);

  1. MPI_Barrier (Barrier):

This operation synchronizes all processes in the communicator.

MPI_Barrier(MPI_Comm comm);
mpi_tree_broadcast

Fig 7.1.1 Broadcast with a tree approach.

Task

  1. Write a C/C++ function named “my_broadcast” in which the root process broadcasts the values of an array of doubles to all other processes.

  2. Initialize the array of double values with a user-defined size, N, specified through the Terminal.

  3. Use only Point-to-Point communication. Your approach could look like a tree (Fig. 7.1.1).

  4. Benchmark your “my_broadcast” function for N=100000 and 10 processes on the short or standard partition of the draco cluster. Compare the runtime with the standard “MPI_Bcast” function. Use the “MPI_Wtime()” function described in Section 6.3 for your measurements.

  5. Include the slurm script, the job output, the implementation itself and all necessary documentation in your submission.

Bonus:

  • We compare the runtime of your implementation with the implementations of the other teams

  • The winner receives 2 extra points

  • submit an anonymous name for your team