Additional MPI Notes

You are not allowed to alias input and output arguments in MPI communications. So, for example, in MPI_Reduce you cannot use the same argument for both sendbuf and recvbuf. It might work, but don't count on it.

The MPI gather function has the calling sequence

MPI_Gather(void *sendbuf,         /* starting address of send buffer   */
           int sendcount,         /* number of elements in send buffer */
           MPI_Datatype sendtype, /* data type of send buffer elements */ 
           void *recvbuf,         /* address of receive buffer         */
           int recvcount,         /* number of elements for any single
                                     receive                           */
           MPI_Datatype recvtype, /* data type of recv buffer elements */
           int root,              /* rank of reciving process          */
           MPI_Comm comm)         /* communicator group                */

Almost always recvtype and sendtype are the same, and sendcount and recvcount will be equal. The last two specify the number of elements from each process, not the total amount of data received. This means, e.g., if sendcount = recvcount = 1, then p data items will be accumulated in recvbuf. They will also be accumulated in rank order.

MPI_Allgather() does the similar thing, except that the data is accumulated into recvbuf on every process in the communicator. There is a shortcoming with MPI_Allgather(); the sendcount must be identical on every process. For the CG algorithm with the order n of the problem not a multiple of the number of processes, that means you would need to "pad" one or more segments of the gathered vector so that they all have the same size. To handle this, there is a vector variant:

MPI_Gatherv(void *sendbuf,
            int sendcount,
            MPI_Datatype sendtype,
            void *recvbuf,       
            int *recvcounts,    
            int *displs,
            MPI_Datatype recvtype,
            int root,            
            MPI_Comm comm)

Here recvcount has been replaced by a vector recvcounts, such that recvcounts[i] contains the number of entries sent by process i. It also allows you to place the data almost wherever in the recvbuf you desire, using the displs[] array. The data sent from process i is placed in the i^th portion of the receive buffer recvbuf on process root. The i^th part of recvbuf begins at offset displs[i] elements into recvbuf. The receive buffer is ignored for all non-root processes, but should be provided.

WARNING: This operation must not cause any overlapping of writes in the recvbuf on root. That means you cannot allow the specification of counts, types, and displacements to let two different processes to have part of their send data writing to the same location in recvbuf.

The "All" version of MPI_Gatherv is straightforward, given the above ideas.

Type signatures

Note the above weasel words "sendtype and recvtype almost always match". When you use vector variants of the MPI functions, or for heterogenous data use MPI derived datatypes, or MPI Pack/Unpack functions, the idea of a type signature is important. The data part of a message is specified by a sequence of pairs { (T₀, d₀), (T₁, d₁), (T₂, d₂), ... (T_n-1, d_n-1)} where each T_i is an MPI datatype and d_i is a displacement into the message. The sequence of types alone, {T₀, T₁, T₂, ..., T_n-1} is called the type signature of the message. The rule is that the type signatures specified by sender and receiver must be compatible. So if a send operation sends a message with signature
{T₀, T₁, T₂, ..., T_n-1}
and the matching receive specifies a type signature
{U₀, U₁, U₂, ..., U_m-1},
then n must not be larger than m, and T_i = U_i for i = 0, 1, ..., m-1.

When you specify only a single datatype, as with MPI_Send, but a sendcount larger than 1, then MPI creates the type signature as the concatenation of sendcount copies of the MPI datatype specified.

For collective communications, the type signatures of all participating processes must be identical.

Last Modified: Thu 08 Feb 2018, 07:18 AM