ECE506 Spring 2014 new problems A Comparing Shared Memory And Message Passing Models
Introduction
For solving very large problems (e.g. modeling of ocean currents) fast, we need parallel computers. On a single processor although instruction level parallelism can be achieved, it will not give the same performance as using multiple processors. Superscalar architecture and out of order execution technique improved the performance, but still scientific problems like simulation of single protein folding can take years on the most advanced processor, but it only takes days on a large and powerful multi processors. In addition to that the multiprocessor system is more attractive in terms of the power adjusted performance.
Types of Parallel Programming models
Communication between the multiprocessors is very important. In parallel programming, there are two widely used architecture models based on the way the multiprocessors communicate with each other.
Both of these models focus on how parallel tasks access common data. There is a third model called data parallel model which is composed of a set of identical tasks which operate on different subsets of data.
Communication: In shared memory model, the parallel tasks that are unit of computation that can be executed independent of each other, can access any location of the memory through implicit loads and stores to a global address space. Reads/writes on shared address space are visible to all other threads immediately.
Synchronization: In the shared memory model, explicit synchronization is needed in order to access data among threads. Three types of synchronization primitives are used widely,
- Point to point synchronization using post and wait
- Synchronization using lock and unlock which is useful to form a critical section.
- Synchronization using Barrier.
Hardware support: In shared memory model, hardware support is necessary, as it is needed to give the illusion that all processors are using a single addressable memory, this is a bit difficult to achieve at lower cost.
Development efforts: In shared memory, the data communication between processors happen through implicit loads and stores, it is not important where the data is located and how is it laid out. Also the typical structure of program code is somewhat same as its sequential version. Hence it is easier to program, has graceful migration path, and hence it is an attractive model.
Tuning Efforts:For a very large multiprocessor systems<ref>http://en.wikipedia.org/wiki/Multiprocessing</ref>, it may take considerably large time to access the remote memory than the single memory, hence it becomes important that how the data is laid out and where it is located hence Shared memory model has high tuning efforts than tuning efforts required by message passing model. Shared memory model works well on moderate-scale tightly-coupled systems. Drawback of shared memory model are Race conditions, synchronization hazards, fault intolerance.
Various mechanisms such as locks / semaphores may be used to control access to the shared memory.<ref>http://behelmy.wordpress.com/parrallel-computing/shared-memory-vs-message-passing-programming-model/</ref>
- An advantage of Shared memory model from the programmer’s point of view : There is no need to specify explicitly the communication of data between tasks. Program development can often be simplified.
- An important disadvantage in terms of performance is that it becomes more difficult to understand and manage data locality.
*Keeping data local to the processor that works on it conserves memory accesses, cache refreshes and bus traffic that occurs when multiple processors use the same data.
*Unfortunately, controlling data locality is hard to understand and beyond the control of the average user.
Implementations:
- On shared memory platforms, the native compilers translate user program variables into actual memory addresses, which are global.
Message Passing Model
Communication:In message passing, parallel tasks cannot access another task's memory, they have their own local memory. Message passing employs an explicit communication model in which explicit messages are exchanged among processors. SEND and RECEIVE are the primitives used for message passing. Message passing requires “cooperating” processes.
Synchronization: In message passing model, since communication happens through the explicit messages, it implicitly act as a synchronization that control the relative ordering of data access by different threads.
Hardware Support: In message passing model, since each processor is using its own local memory and inter processor communication is through explicit messages, such abstraction is not needed and hence hardware support is not required.
Development efforts: Writing a message passing program requires the programmers to immediately think about how to partition the data among the processors and how to communicate the data among them through explicit message passing. Hence it is difficult to program, especially for irregular, dynamic programs.
Tuning efforts: Less tuning efforts are involved in Message passing model.
Implementations of message passing model
- From a programming perspective, message passing implementations commonly comprise a library of subroutines that are embedded in source code. The programmer is responsible for determining all parallelism.
References
<references />
- Solihin Yan, Fundamentals of Parallel Computer Architecture 2008-2009