Non Blocking Communication in MPI-1 and Modes

Nonblocking communications allow the overlap of computation and communication - typically, large numbers of operations could go on during the time it takes to send a message ... or at least, that is likely what you will have found out from Exercise 2. A non-blocking point-to-point communication typically has the form

Post the operation and immediately return.
Do some other (local) work, such as computations.
Probe for completion of the communication.

This allows overlap of communication and computation, particularly on systems where a PE consists of two processors, one responsible for computation and the other handling communications.

When using nonblocking send/receives, MPI uses a handle of type MPI_Request to give something that you can later on test against. So the invocation

MPI_Isend(void *msg, int count, 
          MPI_Datatype datatype,
          int dest,	
          int tag,
          MPI_Comm comm,
          MPI_Request *request);

will return the handle in the variable "request". You should not mess with "request" except through MPI function calls. After posting the nonblocking send, you can later on wait for completion of the send by posting

MPI_Wait(MPI_Request *request, MPI_Status *status)

A similar procedure is used for MPI_Irecv().

You can also just check for completion, rather than having to wait for completion, by:

MPI_Test(MPI_Request *request,
         int *flag,
          MPI_Status *status);

Here flag = true if the operation indicated by "request" is complete, false otherwise. You would typically use a test instead of a wait when there are multiple messages pending, and you want to process them as soon as they come in - rather than in a somewhat arbitrary order typically imposed by a loop.

Other sorta useful variations on this are

MPI_Waitany(): gives an array of handles, and blocks until any one of them completes. On return, tells you which one them triggered the return.
MPI_Waitall(): blocks until all operations in the request block complete.
MPI_Testall(), MPI_Testany(): these are the testing versions of the previous two.

There is a zoo of MPI functions that can be used: MPI_Waitsome, MPI_Testsome, etc. However, it is best to learn the basic ones and understand how they work, than to try to learn all of the operations. Furthermore, clearly you could implement a (probably inefficient) version of MPI_Waitany() using MPI_Test() - and that is just what many vendors have done.

MPI Modes

As we have seen, blocking and non-blocking do not necessarily correspond to what naive thinking might indicate. This is because in MPI, the implementation has been carefully separated from the logical view of the library. So where you may think of particular implementations when you think of blocking/nonblocking, there is no such guaranteed mapping.

Communication Modes in MPI

Much closer to the implementation is the mode used. Until now we have used standard mode, but there are three others. Here are the essentials:

Standard mode is the default. The vendor's implementation decides if outgoing messages are buffered by the system. If buffered, blocking send can complete before the corresponding receive is posted. If not, blocking send cannot complete until the corresponding receive is posted and the entire message is moved to the destination. This means that MPI code that works alright on one machine may deadlock on another, if you have not carefully arranged the send/receive pairs.
Buffered mode has the user allocating buffer space for messages, taking the place of a possible system buffer. Here a blocking send can complete before the corresponding receive is posted, since once the message is copied to the buffer the user is safe to overwrite the message space. The function MPI_BUFFER_ATTACH() attaches a buffer for MPI to use (which only allows one at a time).
Synchronous mode allows a blocking send to start before the corresponding receive is posted, but can complete only if the receive operation has started. If the receive is also synchronous, this forces both processes to sychronize at the communication point.
Ready mode allows the send to start only if the matching receive has already been posted. It completes when the send buffer is empty. This allows the system to possibly do a direct copy of the outgoing message to the receiving message buffer, without doing any system buffering. Ready mode is dangerous; if the corresponding receive has not been posted, then the behaviour is undefined: the message may be delivered later, discarded silently, or a core dump results at the choice of the MPI implementor.

The last three are for the send functions, and consist of prepending a B,S,or R to the corresponding send function. They all use the same corresponding receive function as for standard mode. All four modes have blocking and nonblocking versions.

Last Modified: Thu 08 Feb 2018, 07:17 AM