Introduction to Parallel Programming with MPI: Glossary

Many problems can be distributed across several processors and solved faster.

mpirun runs copies of the program.

The copies are separated MPI rank.

Algorithms can have parallelisable and non-parallelisable sections.

A highly parallel algorithm may be slower on a single processor.

The theoretical maximum speed is determined by the serial sections.

The other main restriction is communication speed between the processes.

Use MPI_Send to send messages.

And MPI_Recv to receive them.

MPI_Recv will block the program until the message is received.

Two major paradigms, message passing and data parallel.

MPI implements the Message Passing paradigm.

Several standard patterns: Trivial, Queue, Master / Worker, Domain Decomposition, All-to-All.

Non-blocking functions allows interleaving communication and computation.

Use MPI_Barrier for global synchronisation.

All-to-All, One-to-All and All-to-One communications have efficient implementation in the library.

There are functions for global reductions. Don’t write your own.

Start from a working serial code.

Write a parallel implementation for each function or parallel region.

Connect the parallel regions with a minimal amount of communication.

Continuously compare with the working serial code.

Use a profiler to find the most important functions and concentrate on those.

The profiler needs to understand MPI. Your cluster probably has one.

If a lot of time is spent in communication, maybe it can be rearranged.

Start from a working serial code.

Write a parallel implementation for each function or parallel region.

Connect the parallel regions with a minimal amount of communication.

Continuously compare with the working serial code.