Expertiza_Wiki - User contributions [en]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-04T03:29:22Z

Wjfisher: /* Summary */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. Generally, parallelism is easily found by investigating the blocks of code that take the longest to execute. Parallelism can take several different forms but is most often found in loop structures. The focus of this article is to discuss the implementation of parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. Each of these will be discussed in detail below with references to outside material for more comprehensive information, if desired.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [http://www.threadingbuildingblocks.org/documentation.php Intel®'s open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 10 and 11.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a; // make local copies of constructor arguments
float *b = my_b;
public:
// operator function describes parallel action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a); // set value of local vars to arguments
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
// calls SimpleLoop.operator() on range 0 to n-1
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 25 - 29.

//first pipeline operation
class Filter1: public filter {
float* my_a; // make local copies of constructor arguments
public:
// operator function describes pipelined action taken by object
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
{}
};

//second pipeline operation
class Filter2: public filter {
float* my_a; // make local copies of constructor arguments
float* my_b;
public:
// operator function describes pipelined action taken by object
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
//first pipeline action
Filter1 filter1(a);
pipeline.add_filter(filter1);

//second pipeline action
Filter2 filter2(a,b);
pipeline.add_filter(filter2);

//execute pipelined operations n-1 times
pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code. Reduction can be performed for addition and multiplication along with logical operators such as AND and OR. In TBB, reduction can be specified using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 18-20.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a; // make local copies of constructor arguments
public:
float *my_sum;
// operator function describes pipelined action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

// join performs the final summation of the individual summations
void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0) // set value of local vars to arguments
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
//perform sum by reduction for n-1 iterations
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads. Section 3.7 on page 60 of the Solihin text has an overview of OpenMP 2.0 that should be used to supplement the information covered here.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives. See the "Did you know?" box in Chapter 3 of the Solihin text on page 63 for more information about the DOACROSS and DOPIPE parallelism.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= Summary =

A brief comparison of the code libraries discussed in this article will be made here in the form of a table. As you can see, none of the libraries contain support for all of the types of parallelism discussed.

{| align="center cellpadding="4"

|-
!Type of Parallelism
!Posix Threads
!Intel® TBB
!OpenMP 2.0
!OpenMp 3.0

|-
!DOALL
|align="center"|Yes
|align="center"|Yes
|align="center"|Yes
|align="center"|Yes

|-
!DOACROSS
|align="center"|Yes
|align="center"|No
|align="center"|No
|align="center"|No

|-
!DOPIPE
|align="center"|Yes
|align="center"|Yes
|align="center"|No
|align="center"|No

|-
! Reduction
|align="center"|No
|align="center"|Yes
|align="center"|No
|align="center"|No

|-
! Functional Parallelism
|align="center"|No
|align="center"|No
|align="center"|Yes
|align="center"|Yes

|}

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix Threads", Human Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-04T03:27:39Z

Wjfisher: /* Summary */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. Generally, parallelism is easily found by investigating the blocks of code that take the longest to execute. Parallelism can take several different forms but is most often found in loop structures. The focus of this article is to discuss the implementation of parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. Each of these will be discussed in detail below with references to outside material for more comprehensive information, if desired.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [http://www.threadingbuildingblocks.org/documentation.php Intel®'s open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 10 and 11.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a; // make local copies of constructor arguments
float *b = my_b;
public:
// operator function describes parallel action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a); // set value of local vars to arguments
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
// calls SimpleLoop.operator() on range 0 to n-1
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 25 - 29.

//first pipeline operation
class Filter1: public filter {
float* my_a; // make local copies of constructor arguments
public:
// operator function describes pipelined action taken by object
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
{}
};

//second pipeline operation
class Filter2: public filter {
float* my_a; // make local copies of constructor arguments
float* my_b;
public:
// operator function describes pipelined action taken by object
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
//first pipeline action
Filter1 filter1(a);
pipeline.add_filter(filter1);

//second pipeline action
Filter2 filter2(a,b);
pipeline.add_filter(filter2);

//execute pipelined operations n-1 times
pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code. Reduction can be performed for addition and multiplication along with logical operators such as AND and OR. In TBB, reduction can be specified using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 18-20.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a; // make local copies of constructor arguments
public:
float *my_sum;
// operator function describes pipelined action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

// join performs the final summation of the individual summations
void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0) // set value of local vars to arguments
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
//perform sum by reduction for n-1 iterations
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads. Section 3.7 on page 60 of the Solihin text has an overview of OpenMP 2.0 that should be used to supplement the information covered here.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives. See the "Did you know?" box in Chapter 3 of the Solihin text on page 63 for more information about the DOACROSS and DOPIPE parallelism.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= Summary =

A brief comparison of the code libraries discussed in this article will be made here in the form of a table. As you can see, none of the libraries contain support for all of the types of parallelism discussed.

{| align="center cellpadding="4"

|-
! Type of Parallelism
! Posix Threads
! Intel® TBB
! OpenMP 2.0
! OpenMp 3.0

|-
! DOALL
|align="center"|Yes
|align="center"| Yes
|align="center"| Yes
|align="center"| Yes

|-
! DOACROSS
| Yes
| No
| No
| No
| align="center"

|-
! DOPIPE
| Yes
| Yes
| No
| No
| align="center"

|-
! Reduction
| No
| Yes
| No
| No
| align="center"

|-
! Functional Parallelism
| No
| No
| Yes
| Yes
| align="center"

|}

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix Threads", Human Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-04T03:26:59Z

Wjfisher: /* Summary */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. Generally, parallelism is easily found by investigating the blocks of code that take the longest to execute. Parallelism can take several different forms but is most often found in loop structures. The focus of this article is to discuss the implementation of parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. Each of these will be discussed in detail below with references to outside material for more comprehensive information, if desired.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [http://www.threadingbuildingblocks.org/documentation.php Intel®'s open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 10 and 11.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a; // make local copies of constructor arguments
float *b = my_b;
public:
// operator function describes parallel action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a); // set value of local vars to arguments
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
// calls SimpleLoop.operator() on range 0 to n-1
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 25 - 29.

//first pipeline operation
class Filter1: public filter {
float* my_a; // make local copies of constructor arguments
public:
// operator function describes pipelined action taken by object
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
{}
};

//second pipeline operation
class Filter2: public filter {
float* my_a; // make local copies of constructor arguments
float* my_b;
public:
// operator function describes pipelined action taken by object
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
//first pipeline action
Filter1 filter1(a);
pipeline.add_filter(filter1);

//second pipeline action
Filter2 filter2(a,b);
pipeline.add_filter(filter2);

//execute pipelined operations n-1 times
pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code. Reduction can be performed for addition and multiplication along with logical operators such as AND and OR. In TBB, reduction can be specified using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 18-20.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a; // make local copies of constructor arguments
public:
float *my_sum;
// operator function describes pipelined action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

// join performs the final summation of the individual summations
void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0) // set value of local vars to arguments
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
//perform sum by reduction for n-1 iterations
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads. Section 3.7 on page 60 of the Solihin text has an overview of OpenMP 2.0 that should be used to supplement the information covered here.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives. See the "Did you know?" box in Chapter 3 of the Solihin text on page 63 for more information about the DOACROSS and DOPIPE parallelism.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= Summary =

A brief comparison of the code libraries discussed in this article will be made here in the form of a table. As you can see, none of the libraries contain support for all of the types of parallelism discussed.

{| align="center cellpadding="4"

|-
! Type of Parallelism
! Posix Threads
! Intel® TBB
! OpenMP 2.0
! OpenMp 3.0
|align="center"

|-
! DOALL
| Yes
| Yes
| Yes
| Yes
|align="center"

|-
! DOACROSS
| Yes
| No
| No
| No
| align="center"

|-
! DOPIPE
| Yes
| Yes
| No
| No
| align="center"

|-
! Reduction
| No
| Yes
| No
| No
| align="center"

|-
! Functional Parallelism
| No
| No
| Yes
| Yes
| align="center"

|}

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix Threads", Human Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-04T03:25:07Z

Wjfisher: /* Summary */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. Generally, parallelism is easily found by investigating the blocks of code that take the longest to execute. Parallelism can take several different forms but is most often found in loop structures. The focus of this article is to discuss the implementation of parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. Each of these will be discussed in detail below with references to outside material for more comprehensive information, if desired.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [http://www.threadingbuildingblocks.org/documentation.php Intel®'s open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 10 and 11.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a; // make local copies of constructor arguments
float *b = my_b;
public:
// operator function describes parallel action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a); // set value of local vars to arguments
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
// calls SimpleLoop.operator() on range 0 to n-1
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 25 - 29.

//first pipeline operation
class Filter1: public filter {
float* my_a; // make local copies of constructor arguments
public:
// operator function describes pipelined action taken by object
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
{}
};

//second pipeline operation
class Filter2: public filter {
float* my_a; // make local copies of constructor arguments
float* my_b;
public:
// operator function describes pipelined action taken by object
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
//first pipeline action
Filter1 filter1(a);
pipeline.add_filter(filter1);

//second pipeline action
Filter2 filter2(a,b);
pipeline.add_filter(filter2);

//execute pipelined operations n-1 times
pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code. Reduction can be performed for addition and multiplication along with logical operators such as AND and OR. In TBB, reduction can be specified using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 18-20.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a; // make local copies of constructor arguments
public:
float *my_sum;
// operator function describes pipelined action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

// join performs the final summation of the individual summations
void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0) // set value of local vars to arguments
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
//perform sum by reduction for n-1 iterations
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads. Section 3.7 on page 60 of the Solihin text has an overview of OpenMP 2.0 that should be used to supplement the information covered here.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives. See the "Did you know?" box in Chapter 3 of the Solihin text on page 63 for more information about the DOACROSS and DOPIPE parallelism.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= Summary =

A brief comparison of the code libraries discussed in this article will be made here in the form of a table. As you can see, none of the libraries contain support for all of the types of parallelism discussed.

{|

|-
! Type of Parallelism
! Posix Threads
! Intel® TBB
! OpenMP 2.0
! OpenMp 3.0
|align="center"

|-
! DOALL
| Yes
| Yes
| Yes
| Yes
|align="center"

|-
! DOACROSS
| Yes
| No
| No
| No
| align="center"

|-
! DOPIPE
| Yes
| Yes
| No
| No
| align="center"

|-
! Reduction
| No
| Yes
| No
| No
| align="center"

|-
! Functional Parallelism
| No
| No
| Yes
| Yes
| align="center"

|}

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix Threads", Human Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-04T03:24:49Z

Wjfisher: /* Summary */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. Generally, parallelism is easily found by investigating the blocks of code that take the longest to execute. Parallelism can take several different forms but is most often found in loop structures. The focus of this article is to discuss the implementation of parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. Each of these will be discussed in detail below with references to outside material for more comprehensive information, if desired.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [http://www.threadingbuildingblocks.org/documentation.php Intel®'s open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 10 and 11.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a; // make local copies of constructor arguments
float *b = my_b;
public:
// operator function describes parallel action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a); // set value of local vars to arguments
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
// calls SimpleLoop.operator() on range 0 to n-1
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 25 - 29.

//first pipeline operation
class Filter1: public filter {
float* my_a; // make local copies of constructor arguments
public:
// operator function describes pipelined action taken by object
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
{}
};

//second pipeline operation
class Filter2: public filter {
float* my_a; // make local copies of constructor arguments
float* my_b;
public:
// operator function describes pipelined action taken by object
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
//first pipeline action
Filter1 filter1(a);
pipeline.add_filter(filter1);

//second pipeline action
Filter2 filter2(a,b);
pipeline.add_filter(filter2);

//execute pipelined operations n-1 times
pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code. Reduction can be performed for addition and multiplication along with logical operators such as AND and OR. In TBB, reduction can be specified using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 18-20.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a; // make local copies of constructor arguments
public:
float *my_sum;
// operator function describes pipelined action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

// join performs the final summation of the individual summations
void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0) // set value of local vars to arguments
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
//perform sum by reduction for n-1 iterations
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads. Section 3.7 on page 60 of the Solihin text has an overview of OpenMP 2.0 that should be used to supplement the information covered here.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives. See the "Did you know?" box in Chapter 3 of the Solihin text on page 63 for more information about the DOACROSS and DOPIPE parallelism.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= Summary =

A brief comparison of the code libraries discussed in this article will be made here in the form of a table. As you can see, none of the libraries contain support for all of the types of parallelism discussed.

{|

|-
! Type of Parallelism
! Posix Threads
! Intel® TBB
! OpenMP 2.0
! OpenMp 3.0
| align="center"

|-
! DOALL
| Yes
| Yes
| Yes
| Yes
| align="center"

|-
! DOACROSS
| Yes
| No
| No
| No
| align="center"

|-
! DOPIPE
| Yes
| Yes
| No
| No
| align="center"

|-
! Reduction
| No
| Yes
| No
| No
| align="center"

|-
! Functional Parallelism
| No
| No
| Yes
| Yes
| align="center"

|}

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix Threads", Human Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-04T03:23:23Z

Wjfisher: /* Summary */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. Generally, parallelism is easily found by investigating the blocks of code that take the longest to execute. Parallelism can take several different forms but is most often found in loop structures. The focus of this article is to discuss the implementation of parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. Each of these will be discussed in detail below with references to outside material for more comprehensive information, if desired.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [http://www.threadingbuildingblocks.org/documentation.php Intel®'s open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 10 and 11.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a; // make local copies of constructor arguments
float *b = my_b;
public:
// operator function describes parallel action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a); // set value of local vars to arguments
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
// calls SimpleLoop.operator() on range 0 to n-1
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 25 - 29.

//first pipeline operation
class Filter1: public filter {
float* my_a; // make local copies of constructor arguments
public:
// operator function describes pipelined action taken by object
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
{}
};

//second pipeline operation
class Filter2: public filter {
float* my_a; // make local copies of constructor arguments
float* my_b;
public:
// operator function describes pipelined action taken by object
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
//first pipeline action
Filter1 filter1(a);
pipeline.add_filter(filter1);

//second pipeline action
Filter2 filter2(a,b);
pipeline.add_filter(filter2);

//execute pipelined operations n-1 times
pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code. Reduction can be performed for addition and multiplication along with logical operators such as AND and OR. In TBB, reduction can be specified using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 18-20.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a; // make local copies of constructor arguments
public:
float *my_sum;
// operator function describes pipelined action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

// join performs the final summation of the individual summations
void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0) // set value of local vars to arguments
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
//perform sum by reduction for n-1 iterations
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads. Section 3.7 on page 60 of the Solihin text has an overview of OpenMP 2.0 that should be used to supplement the information covered here.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives. See the "Did you know?" box in Chapter 3 of the Solihin text on page 63 for more information about the DOACROSS and DOPIPE parallelism.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= Summary =

A brief comparison of the code libraries discussed in this article will be made here in the form of a table. As you can see, none of the libraries contain support for all of the types of parallelism discussed.

{|

|-
! Type of Parallelism
! Posix Threads
! Intel® TBB
! OpenMP 2.0
! OpenMp 3.0

|-
! DOALL
| Yes
| Yes
| Yes
| Yes

|-
! DOACROSS
| Yes
| No
| No
| No

|-
! DOPIPE
| Yes
| Yes
| No
| No

|-
! Reduction
| No
| Yes
| No
| No

|-
! Functional Parallelism
| No
| No
| Yes
| Yes

|}

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix Threads", Human Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-04T03:22:48Z

Wjfisher:

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. Generally, parallelism is easily found by investigating the blocks of code that take the longest to execute. Parallelism can take several different forms but is most often found in loop structures. The focus of this article is to discuss the implementation of parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. Each of these will be discussed in detail below with references to outside material for more comprehensive information, if desired.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [http://www.threadingbuildingblocks.org/documentation.php Intel®'s open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 10 and 11.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a; // make local copies of constructor arguments
float *b = my_b;
public:
// operator function describes parallel action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a); // set value of local vars to arguments
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
// calls SimpleLoop.operator() on range 0 to n-1
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 25 - 29.

//first pipeline operation
class Filter1: public filter {
float* my_a; // make local copies of constructor arguments
public:
// operator function describes pipelined action taken by object
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
{}
};

//second pipeline operation
class Filter2: public filter {
float* my_a; // make local copies of constructor arguments
float* my_b;
public:
// operator function describes pipelined action taken by object
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
//first pipeline action
Filter1 filter1(a);
pipeline.add_filter(filter1);

//second pipeline action
Filter2 filter2(a,b);
pipeline.add_filter(filter2);

//execute pipelined operations n-1 times
pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code. Reduction can be performed for addition and multiplication along with logical operators such as AND and OR. In TBB, reduction can be specified using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 18-20.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a; // make local copies of constructor arguments
public:
float *my_sum;
// operator function describes pipelined action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

// join performs the final summation of the individual summations
void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0) // set value of local vars to arguments
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
//perform sum by reduction for n-1 iterations
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads. Section 3.7 on page 60 of the Solihin text has an overview of OpenMP 2.0 that should be used to supplement the information covered here.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives. See the "Did you know?" box in Chapter 3 of the Solihin text on page 63 for more information about the DOACROSS and DOPIPE parallelism.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= Summary =

A brief comparison of the code libraries discussed in this article will be made here in the form of a table. As you can see, none of the libraries contain support for all of the types of parallelism discussed.

{|

|-
! Type of Parallelism
! Posix Threads
! Intel® TBB
! OpenMP 2.0
! OpenMp 3.0

|-
! DOALL
| Yes
| Yes
| Yes
| Yes

|-
! DOACROSS
| Yes
| No
| No
| No

|-
! DOPIPE
| Yes
| Yes
| No
| No

|-
! Reduction
| No
| Yes
| No
| No

|-
! Functional Parallelism
| No
| No
| Yes
| Yes
}

|}

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix Threads", Human Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-04T03:12:31Z

Wjfisher:

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. Generally, parallelism is easily found by investigating the blocks of code that take the longest to execute. Parallelism can take several different forms but is most often found in loop structures. The focus of this article is to discuss the implementation of parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. Each of these will be discussed in detail below with references to outside material for more comprehensive information, if desired.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [http://www.threadingbuildingblocks.org/documentation.php Intel®'s open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 10 and 11.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a; // make local copies of constructor arguments
float *b = my_b;
public:
// operator function describes parallel action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a); // set value of local vars to arguments
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
// calls SimpleLoop.operator() on range 0 to n-1
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 25 - 29.

//first pipeline operation
class Filter1: public filter {
float* my_a; // make local copies of constructor arguments
public:
// operator function describes pipelined action taken by object
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
{}
};

//second pipeline operation
class Filter2: public filter {
float* my_a; // make local copies of constructor arguments
float* my_b;
public:
// operator function describes pipelined action taken by object
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
//first pipeline action
Filter1 filter1(a);
pipeline.add_filter(filter1);

//second pipeline action
Filter2 filter2(a,b);
pipeline.add_filter(filter2);

//execute pipelined operations n-1 times
pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code. Reduction can be performed for addition and multiplication along with logical operators such as AND and OR. In TBB, reduction can be specified using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 18-20.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a; // make local copies of constructor arguments
public:
float *my_sum;
// operator function describes pipelined action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

// join performs the final summation of the individual summations
void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0) // set value of local vars to arguments
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
//perform sum by reduction for n-1 iterations
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads. Section 3.7 on page 60 of the Solihin text has an overview of OpenMP 2.0 that should be used to supplement the information covered here.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives. See the "Did you know?" box in Chapter 3 of the Solihin text on page 63 for more information about the DOACROSS and DOPIPE parallelism.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix Threads", Human Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-04T03:03:56Z

Wjfisher: /* Intel Threading Building Blocks */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [http://www.threadingbuildingblocks.org/documentation.php Intel®'s open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 10 and 11.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a; // make local copies of constructor arguments
float *b = my_b;
public:
// operator function describes parallel action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a); // set value of local vars to arguments
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
// calls SimpleLoop.operator() on range 0 to n-1
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 25 - 29.

//first pipeline operation
class Filter1: public filter {
float* my_a; // make local copies of constructor arguments
public:
// operator function describes pipelined action taken by object
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
{}
};

//second pipeline operation
class Filter2: public filter {
float* my_a; // make local copies of constructor arguments
float* my_b;
public:
// operator function describes pipelined action taken by object
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order); //execute the actions of this filter in order
my_a(a); // set value of local vars to arguments
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
//first pipeline action
Filter1 filter1(a);
pipeline.add_filter(filter1);

//second pipeline action
Filter2 filter2(a,b);
pipeline.add_filter(filter2);

//execute pipelined operations n-1 times
pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code. Reduction can be performed for addition and multiplication along with logical operators such as AND and OR. In TBB, reduction can be specified using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 18-20.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a; // make local copies of constructor arguments
public:
float *my_sum;
// operator function describes pipelined action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

// join performs the final summation of the individual summations
void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0) // set value of local vars to arguments
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
//perform sum by reduction for n-1 iterations
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix Threads", Human Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-04T02:49:23Z

Wjfisher: /* DOALL Parallelism */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [http://www.threadingbuildingblocks.org/documentation.php Intel®'s open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 10 and 11.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a; // make local copies of constructor arguments
float *b = my_b;
public:
// operator function describes parallel action taken by object
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a); // set value of local vars to arguments
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
// calls SimpleLoop.operator() on range 0 to n-1
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 25 - 29.

class Filter1: public filter {
float* my_a;
public:
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order);
my_a(a);
{}
};

class Filter2: public filter {
float* my_a;
float* my_b;
public:
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order);
my_a(a);
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
Filter1 filter1(a);
pipeline.add_filter(filter1);

Filter2 filter2(a,b);
pipeline.add_filter(filter2);

pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 18-20.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix Threads", Human Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-04T02:44:52Z

Wjfisher: /* Intel Threading Building Blocks */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [http://www.threadingbuildingblocks.org/documentation.php Intel®'s open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 10 and 11.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 25 - 29.

class Filter1: public filter {
float* my_a;
public:
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order);
my_a(a);
{}
};

class Filter2: public filter {
float* my_a;
float* my_b;
public:
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order);
my_a(a);
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
Filter1 filter1(a);
pipeline.add_filter(filter1);

Filter2 filter2(a,b);
pipeline.add_filter(filter2);

pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] on pages 18-20.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix Threads", Human Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-03T23:04:23Z

Wjfisher: /* Intel Threading Building Blocks */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [http://www.threadingbuildingblocks.org/documentation.php Intel®'s open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] found in their documentation.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] found in their documentation.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] found in their documentation.

class Filter1: public filter {
float* my_a;
public:
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order);
my_a(a);
{}
};

class Filter2: public filter {
float* my_a;
float* my_b;
public:
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order);
my_a(a);
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
Filter1 filter1(a);
pipeline.add_filter(filter1);

Filter2 filter2(a,b);
pipeline.add_filter(filter2);

pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end. This example is adapted from the [http://software.intel.com/sites/products/documentation/hpc/tbb/tutorial.pdf Intel® TBB Tutorial] found in their documentation.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix Threads", Human Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-03T22:59:37Z

Wjfisher: /* References */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [Intel's open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples.

class Filter1: public filter {
float* my_a;
public:
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order);
my_a(a);
{}
};

class Filter2: public filter {
float* my_a;
float* my_b;
public:
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order);
my_a(a);
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
Filter1 filter1(a);
pipeline.add_filter(filter1);

Filter2 filter2(a,b);
pipeline.add_filter(filter2);

pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix Threads", Human Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-03-03T22:58:29Z

Wjfisher: /* Intel Threading Building Blocks */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
[https://computing.llnl.gov/tutorials/pthreads/#Pthread POSIX thread], also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads pthread_create()] function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads thread] argument is used to provide a unique identifier for the thread you are creating. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads attr] argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads start_routine] argument is the program subroutine that will be executed by the thread being created. The [https://computing.llnl.gov/tutorials/pthreads/#CreatingThreads arg] argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A [https://computing.llnl.gov/tutorials/pthreads/#MutexOverview mutex variable] is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically [https://computing.llnl.gov/tutorials/pthreads/#MutexCreation create a mutex] variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_lock()] and [https://computing.llnl.gov/tutorials/pthreads/#MutexLocking pthread_mutex_unlock()] functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

[https://computing.llnl.gov/tutorials/pthreads/#ConVarOverview Conditional variables] allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_wait()] and [https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal pthread_cond_signal()]. The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts threads to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library. For more comprehensive information than is covered in this discussion, please see [Intel's open source site].

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples.

class Filter1: public filter {
float* my_a;
public:
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order);
my_a(a);
{}
};

class Filter2: public filter {
float* my_a;
float* my_b;
public:
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order);
my_a(a);
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
Filter1 filter1(a);
pipeline.add_filter(filter1);

Filter2 filter2(a,b);
pipeline.add_filter(filter2);

pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is [https://computing.llnl.gov/tutorials/openMP/#Introduction OpenMP]? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: [https://computing.llnl.gov/tutorials/openMP/#ParallelRegion #pragma omp parallel] and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: [https://computing.llnl.gov/tutorials/openMP/#SECTIONS #pragma omp section] (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/6 DD

2010-02-27T04:40:45Z

Wjfisher: /* '''Inclusion Property''' */

=='''MULTICORE Architecture'''==

The term '''multicore''' means two or more independent cores. The term '''architecture''' means the design/layout of a particular idea. Hence the combined term multicore architecture simply means the integration of cores and its layout. For e.g. '''dual core''' means each processor contains 2 independent cores.

=='''NUMA architecture'''==
The term '''NUMA''' is an acronym for Non Uniform Memory Access. It is one way of organizing the '''MIMD''' (Multiple Instruction Multiple Data: independent processors are connected together to form a multiprocessor system.) architecture. NUMA is also known as distributed shared memory. Each processor has an independent cache, memory and they are connected to a common interconnection network like a crossbar, mesh, hypercube, etc.

A '''crossbar''' is a switch which can connect processors to memory in any order. However, it becomes expensive to use, hence as a compromise between crossbar and shared bus sytems, multistage interconnection networks are used (e.g. network on chip interconnects)

=='''CACHE'''==
'''Cache'' is a storage device having small memory. Why were caches introduced? [1] 
The CPU works at clock frequencies of 3 GHZ and most common RAM speeds are about 400 MHz. Hence, there needs to be a transfer of data to and fro from RAM and CPU. This will increase the latency and CPU will remain idle most times. The main reason of introducing caches was to reduce this latency to access the data from main memory. These small buffers (caches) operate at higher speeds than normal RAM and hence data can be accessed quickly from the cache.

<center>[[Image: memchart.jpg|120p]]</center> 
<center>Memory Hierarchy[2]</center>
The cache which is closest to CPU is called Level 1 Cache or L1 cache. It is also known as primary cache. The size is between 8 KB – 128 KB. The next level is L2 cache (secondary cache) whose size is between 256 KB –1024 KB. If the data that CPU is looking for is not found in registers, it seeks it in L1 cache, if not found, then in L2 cache,then to the main memory, then finally to an external storage device.

This leads us to the introduction of two new terms: cache hit and cache miss.
The data searched for by registers, if found in cache, results in cache hit, and if not found in cache, results in cache miss.
With the introduction of multiple cache levels, it can be individually stated as L1 cache hit/miss, L2 cache hit/miss and so on.

Most modern computers use the following memory hierarchy. The figure below shows an uni processor memory hierarchy & a recent dual core memory hierarchy

[[Image:Picture4.png]] [[Image: new.png]] 
<center>Uniprocessor memory hierarchy (L) [3] & Dual core memory hierarchy (R) [4]</center>

Previously the processors had an unified L1 cache i.e. data and instruction cache were unified in one. If the Instruction fetch unit wants to fetch data from the cache and the memory unit wants to load an address simultaneously, then the memory operation is given priority and the fetch unit is stalled. Hence, L1 cache was split into L1 data cache (L1 – D) and L1 instruction cache (L1 -I) so that it helps in concurrency. One more advantage is fewer number of ports will be required.
L2 cache is still unified. As the latency difference between main memory and the fastest cache has become larger, some processors have begun to utilize L3 cache, some don’t. Hence the above figure shows 2 boxes, one which includes L1 and L2, and the other which combines L3 with them. The L3 cache is roughly about 4 MB – 32 MB in size.

In addition to these 3 levels of cache, some multicore processors use: Inclusion property, Victim cache, Translation LookAside Buffer, Trace Cache, Smart Cache (Intel)

=== '''Inclusion Property''' ===
Here, the larger lower level cache (L2) includes all the data of upper level (L1) cache. The advantage of this property is that if a value is not found in the lower level cache, it is guaranteed it is not present in the upper level cache, which reduces the amount of checking. The disadvantage is if L2 cache is smaller in size, most of it contains only L1 cache data and hardly any new data, defeating the purpose of another level of cache altogether.

=== '''Victim Cache''' ===
A '''victim cache''' is a cache used to hold blocks evicted from a CPU cache upon replacement. When main cache evicts a block, the victim cache will take the evicted block. This block is called the victim block. When the main cache misses, it searches the victim cache for recently evicted blocks. So, a hit in victim cache means the main cache doesn’t have to go to the next level of memory.

=== '''Translation LookAside Buffer''' ===
The '''TLB''' is a small cache holding recently used virtual-to-physical address translations. TLB is a hardware cache that sits alongside the L1 instruction and data caches.

=== '''Trace Cache''' ===
The '''trace cache''' is located between the decoder unit and the execution unit. Thus the fetch unit grabs data directly from L2 memory cache.

=== '''Smart Cache''' ===
Intel Advanced '''Smart Cache'''[5] is a multi-core optimized cache that improves performance by increasing the probability that each execution core of a dual-core processor can access data from a higher-performance, more-efficient cache subsystem. To accomplish this, Intel Core micro architecture shares the Level 2 (L2) cache between the cores. If one of the cores is inactive, the other core will have access to the full cache. It provides a peak transfer rate of 96 GB/sec (at 3 GHz frequency).

=== '''Parameters of a cache:''' ===
1. Size: cache data storage. 
2. Associativity: Number of blocks in a set 
3. Block Size: Total number of bytes in a single block. 
A set is a fancy term used for the row of a cache.

=='''PLACEMENT POLICY'''==
The placement policy is the first step in managing a cache. This decides where a block in memory can be placed in the cache. The placement policies are:

=== '''Direct mapped''' ===
If each block has only one place where it can appear in a cache, it is called direct- mapping. It is also known as one way set associative.[6] 
The main memory address (32 bits) is broken down as: (Assumed 8 blocks and 8 lines in cache.)

[[Image:table3.jpg]] 
Number of offset bits = log 2 (Block Size) 
Number of index bits = log 2 (Number of sets) 
Number of Tag bits = 32- offset bits- index bits 

Each line has its own tag associated with it. The index tells which row of cache to look at. To search for a word in the cache, compare the tag bits of the address with the tag of the line. If it matches, the block is in the cache. It is a cache hit. Then select the corresponding word from that line.

=== '''Fully Associative''' ===
If any line can store the contents of any memory location, it is known as fully associative cache. Hence, no index bits are required. 
The main memory address (32 bits) is broken down as:

[[Image:table2.jpg]]

It is also known as Content Addressable Memory. To search for a word in the cache, compare the tag bits of the address with the tag of all lines in the cache simultaneously. If any one matches, the word is in the cache. Then select the corresponding word from that line.

=== '''Set Associative''' ===

[[Image:table4.jpg]]

Hence, a compromise between the fully associative cache and the direct mapping technique is made, known as set associative mapping. Almost 90% of the current processors use this mapping.
Here the cache is divided into ‘s’ sets, where s is a power of 2. The associativity can be found by using the following relation:
Associativity = Cache size/ (Block Size * Number of sets). For an associativity of ‘n’, the cache is ‘n- way’ set associative.

<center>[[Image: Picture15.png]]</center>
<center>2 way set associative mapping [3]</center>

Here the cache size is the same, but the index field is 1 bit shorter. Based on the index bits, select a particular set. To search for a word in the cache, compare the tag bits of the address with the tag bits of the line in each set (here 2). If it matches with any one of the tags, it is a cache hit. Else, it is a cache miss and the word has to be fetched from main memory.

Conceptually, the direct mapped and fully associative caches are just "special cases"[6] of the N-way set associative cache. If you make n=1, it becomes a "1-way" set associative cache. i.e. there is only one line per set, which is the same as a direct mapped cache. On the other hand, if you make ‘n’ to be equal to the number of lines in the cache, then you only have one set, containing all of the cache lines, and each and every memory location points to that set. This gives us a fully associative cache.

=='''REPLACEMENT POLICY'''==
This decides which block needs to be evicted to make space available for a new block entry. The replacement policies used are: LRU, Psuedo LRU, FIFO, OPT, random, etc. 
The most commonly used is LRU (Least recently used replacement policy). It works as follows: [3]

1. Make small counters per block in a set and the number of bits in each counter =
log2 (Associativity) 
2. If there is a cache hit, make the current block’s counter value as 0, which indicates that it is the most recently used one. Increment the counters of the other blocks whose counter values are less than the referenced block’s old counter value 
3.If there is a cache miss, make the newly allocated block’s counter value as 0 and then increment the counters of all other blocks. 

One eg: trace ABCEF. (0-2 indicate counter numbers)
<center>
{| class="wikitable" border="1"
|-
| 0
| 1
| 2
|-
| 2
| 0 B
| 1 A
|-
| 0 C
| 1 B
| 2 A
|-
| 1 C
| 2 B
| 0 E
|-
| 2 C
| 0 F
| 1 E
|}</center>

When E came in, the block with the highest count here 2, block A was evicted since it was least recently used and the value of other counters were incremented accordingly. Similarly, when F came in, block B was evicted.

FIFO policy simply removes the cache block which was brought in earliest in the cache, irrespective of it being least, most recently used.

=='''WRITE POLICY'''==

=== '''Write Through''' ===
Whenever a value is written in the cache, it is immediately written in the next level of memory hierarchy. The main advantage is that both levels have the recent updated value at all times. No dirty bit is required. The disadvantage is that since it writes every time, it consumes a lot of bandwidth. A bit stored in the cache can flip its value when struck by alpha particles resulting in soft errors[4]. The effect of this is that the value in the cache is lost. Since, the lower level of cache contains the same updated value, it is safe to discard the block, as it can be easily re fetched. So, only error detection is enough.

=== '''Write Back''' ===
The next level of memory hierarchy is not updated as soon as a value is written in the cache, but is updated only when a block is evicted from the cache. To cope up with the issue of having the most recent value, a dirty bit is associated with the cache block. When there is a write to the cache, the dirty bit is set 1. So, if this block were to be evicted, indication of dirty bit being 1, will make the block to be written back to the main memory. When a new block is written in cache, the dirty bit is made 0. Since, lower level of cache does not contain the same updated value, error correction is necessary.

Thus in most processors, L1 cache uses a WT policy and L2 cache uses a WB policy.

=== '''Write Allocate policy'''===
On a write miss, the block is brought back into the cache before it is written. A write back policy generally uses write allocate.

=== '''Write No Allocate Policy''':===
On a write miss, the write is propagated to the lower level of memory hierarchy but it is not brought back into the cache. A write through policy generally uses write no allocate.

=='''RECENT MULTI-CORE ARCHITECTURES'''==
Cores in multi-core systems may implement architectures such as superscalar, VLIW, vector processing, SIMD, or multithreading.

==='''Super scalar'''===
'''Superscalar''' architecture implements Instruction level parallelism within a single processor. A superscalar processor executes more than one instruction during a clock cycle. 

==='''Very long instruction word (VLIW)'''===
'''VLIW'''[7] refers to a CPU architecture designed to take advantage of instruction level parallelism (ILP). It includes out of order execution. It is a type of MIMD. VLIW CPUs offer significant computational power with less hardware complexity (but greater compiler complexity) than is associated with most superscalar CPUs. One VLIW instruction encodes multiple operations; specifically, one instruction encodes at least one operation for each execution unit of the device. 

=== ''' Simultaneous Multithreading''' ===
'''SMT''' is a technique for improving the overall efficiency of superscalar CPUs with hardware multithreading. SMT permits many independent threads of execution to make better utilization of the resources provided by modern processor architectures. 

The following table shows the type of cache structure used by recent processors.[8]

[[Image:table31.jpg]] [[Image:table32.jpg]] 
The above tables are not directly cited from anywhere, information has been found and framed.

== '''RECENT NUMA ARCHITECTURES''' ==

1. 4 '''Opteron 875''' dual core processors: 1 L1- D cache per core (2 per socket), size: 65 KB
,1 L1- I cache per core, size: 65 KB
,1 L2 cache per core (2 per socket), size: 1MB

2.AMD '''Hammer''': L1: 128 KB, 2 way set assoc & L2: 1 MB.
It also uses a large TLB and a victim cache.

3. Altix: The '''Altix UV''' supercomputer architecture combines a development of the NUMAlink interconnect used in the Altix 4000 (NUMAlink 5) with 4/6/8-core "Nehalem-EX" Intel Xeon processors. Altix UV systems run either SuSE Linux Enterprise Server or Red Hat Enterprise Linux, and scale from 32 to 2,048 cores with support for up to 16 TB of shared memory in a single system. It came in the end of 2009.

Other examples are: IBM p690, SGI Origin. Here are two different structures.

Tilera TILE 64 consists of a mesh network of 64 tiles. It is based on VLIW instruction set. Each of the cores (tile) has its own L1 and L2 cache plus an overall virtual L3 cache, having mutual inclusion property. It was developed in 2007. 
Tilera, TILE- Gx is a future multi core processor consisting of a mesh network of 100 cores. It is a 64 bit core (3 issue)
L1- I: 32 KB/ core, L1- D: 32 KB/ core, L2: 256 KB/ core, L3: 26 MB/ chip.

== '''WRITE POLICIES used in recent multi core architectures''' ==

1. '''Intel Pentium Processors''': [10] Mainly 2 bits control the cache. CD (cache disable) and NW (not write through)
<center>CD= 1: cache disabled, CD=0: enables the cache</center>
<center>NW= 0: Write Through, NW=1: Write Back</center>
A WB [11] line is always fetched into the cache if a cache miss occurs. 
A WT [11]line is not fetched into the cache on a write miss. For the Pentium Pro processor, a WT hit to the L1 cache updates the L1 cache. A WT hit to L2 cache invalidates the L2 cache. 
A WP [11] ('''Write Protected''') line is cacheable, but a write to it cannot modify the cache line. A WP line is not fetched into the cache on a write miss. For the Pentium Pro processor, a WP hit to the L2 cache invalidates the line in the L2 cache. 
An UC [11] ('''Uncacheable''') line is not put into the cache. A UC hit to the L1 or L2 cache invalidates the entry.
Cache consistency: MESI (Modified, Exclusive, Shared, Invalid) policy is used.

2.'''Intel IA 32 IA64 architecture''': [12] In IA 32, the number of processor sharing the cache is one plus value represented by CPUID.4.EAX [25:14]

'''Write combining''': Successive writes to the same cache line are combined. 
'''Write collapsing''': Successive writes to the same bytes result in only the last write being visible. 
'''Weakly ordered''': No ordering is preserved between WC stores or between other loads or stores. 
'''Uncacheable & Write No Allocate''': Stored data is written around the cache and it uses Write No Allocate. 
'''Non-temporal''': Data which is referenced once and not reused in the immediate future

If the programmer specifies a non-temporal store to UC or WP memory types, then the store behaves like an uncacheable store. There is no conflict if a non temporal store to WC is specified. If the programmer specifies a non-temporal store to WB or WT memory types, and if the data is in the cache, will ensure consistency. However, if it is a miss, the transaction is subject to all WC memory semantics, which will not write allocate.

'''3'''.Intel's '''Celeron''' and '''Xeon''' based on '''Netburst microarchitecture''' (i.e. based on Pentium 4) will have the same specs as Pentium 4 but with different L2 cache size, while Celeron and Xeon based on '''Core microarchitecture''' (i.e. based on Core 2 Duo) will have the same specs as Core 2 Duo but with different L2 cache size. 
Intel Xeon Clowertown: L1: 8 way associative & L2: 16 way associative. 
Intel Celeron using Intel Core 2 architecture: L2: 16 way associative & L1- D: 8 way associative & L1- I: 8 way associative. It also has 16 entry DTLB. 

4. Intel '''RAID''': These use Write Back policy. 
5. Intel Core microarchitecture uses FIFO replacement policy. 
6. AMD uses cache exclusion unlike Intel’s cache inclusion 
7. Sun's Niagara and SPARC use L1 caches as WT, with allocate on load and noallocate on stores. 
8. In general, WT invalidate protocols gives better performance than WB- MESI protocols. But most modern processors use WB policy.

=='''REFERENCES'''==

[1]: http://www.karbosguide.com/books/pcarchitecture/chapter10.htm 
[2]: http://www.real-knowledge.com/memory.htm 
[3]: Computer Design & Technology- Lectures slides by Prof.Eric Rotenberg 
[4]: Fundamentals of Parallel Computer Architecture by Prof.Yan Solihin 
[5] : http://download.intel.com/technology/architecture/sma.pdf 
[6]: http://www.pcguide.com/ref/mbsys/cache/funcMapping-c.html 
[7]:http://en.wikipedia.org/wiki/Very_Long_Instruction_Word 
[8]: http://en.wikipedia.org/wiki/Multi-core_processor 
[9]: http://en.wikipedia.org/wiki/Power_Architecture 
[10]: http://www.intel.com/design/intarch/papers/cache6.pdf 
[11]: http://download.intel.com/design/archives/processors/pro/docs/24269001.pdf 
[12]: http://www.intel.com/Assets/PDF/manual/248966.pdf 
[13]: http://www.intel.com/pressroom/kits/quickreffam.htm 
[14]: http://www.hardwaresecrets.com

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T15:45:34Z

Wjfisher: /* DOPIPE Parallelism */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples.

class Filter1: public filter {
float* my_a;
public:
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order);
my_a(a);
{}
};

class Filter2: public filter {
float* my_a;
float* my_b;
public:
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order);
my_a(a);
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
Filter1 filter1(a);
pipeline.add_filter(filter1);

Filter2 filter2(a,b);
pipeline.add_filter(filter2);

pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T15:45:05Z

Wjfisher: /* DOPIPE Parallelism */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Exploiting DOPIPE parallelism is more complicated than DOALL parallelism in TBB. The '''pipeline''' and '''filter''' classes are used to implement the pipeline pattern.

// sequential loop amenable to DOPIPE parallelism
for(size_t i = 0; i < n; i++) {
a[i] = a[i-1] + 5.3;
b[i] = a[i] + 2.7;
}

In the DOPIPE code below, each section of code that can be separated into a chunk should be defined in a different '''filter''' class instance. For the sequential code example, there are two separate statements that can be executed in parallel; therefore there are two '''filter''' instances created. In order to run them in parallel, a '''pipeline''' instance needs to be created and given passed the instances of the created '''filter''' instantiations. Now, simply run the pipeline over '''n''' iterations and you have pipelined parallelism in TBB. Notice that the '''operator()''' function is called to actually perform the code execution as for all the previous TBB examples.

class Filter1: public filter {
float* my_a;
public:
void operator() {
my_a[i] = my_a[i-1] + 5.3;
}

Filter1(float *a) :
filter(serial_in_order);
my_a(a);
{}
};

class Filter2: public filter {
float* my_a;
float* my_b;
public:
void operator() {
my_b[i] = my_a[i] + 2.7;
}

Filter2(float *a, float *b) :
filter(serial_in_order);
my_a(a);
my_b(b);
{}
};

// pipeline creation and execution
void main(void) {
size_t n;
float a[], b[];
...
Filter1 filter1(a);
pipeline.add_filter(filter1);

Filter2 filter2(a,b);
pipeline.add_filter(filter2);

pipeline.run(n);

pipeline.clear();
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T15:03:30Z

Wjfisher: /* Intel Threading Building Blocks */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Text

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T04:02:48Z

Wjfisher: /* Intel Threading Building Blocks */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP, where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Text

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

== Flexibility ==

TBB also boasts of the ability to combine its package with other threading packages, such as OpenMP and Posix threads. This allows the programmer more flexibility to mix and match packages to obtain, for example, the functional parallelism benefits of OpenMP 2.0 and still retain the ability to use reduction in TBB.

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T03:50:40Z

Wjfisher: /* Intel Threading Building Blocks */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP, where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism are not available for explicit usage using the library.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== DOPIPE Parallelism ==

Text

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T03:46:58Z

Wjfisher: /* DOALL Parallelism */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP, where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism is not available for explicit usage using the library.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This function takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T03:46:44Z

Wjfisher: /* Reduction */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP, where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism is not available for explicit usage using the library.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. This function takes two parameters just like we have seen from '''parallel_for()''' above. The first is the range of indices of the loop that have an operation to reduce in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T03:44:45Z

Wjfisher: /* DOALL Parallelism */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP, where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism is not available for explicit usage using the library.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float a[], float b[];
int n;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T03:44:03Z

Wjfisher: /* Reduction */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP, where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism is not available for explicit usage using the library.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float *a, float *b;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== Reduction ==

Reduction is a way of removing conflicts in parallel code and can be specified in TBB using the '''parallel_reduce()''' construct. The code example described in this section involves calculating a sum of variables in an array. The sequential code version is shown below.

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Instead of using a single '''sum''' variable, in TBB partial sums are calculated in parallel and then summed together to form a total sum at the end.

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}

SimpleSum(SimpleSum& x, split): my_a(x.my_a), my_sum(0) {}

void join(const SimpleSum& y) {
my_sum += y.my_sum;
}

SimpleSum(float a[]) :
my_a(a), my_sum(0)
{}
};

void main(void) {
float a[], sum;
int n;
...
...
SimpleSum ss(a);
parallel_reduce(blocked_range<size_t>(0,n), ss);
sum = ss.my_sum;
}

In the version of the code using '''parallel_reduce()''', there a few changes to the format used in the '''parallel_for''' example above that are slightly more complex. A class must be created to perform the parallel operations as before but this time the splitting and combining operations of reduction must be specified. The splitting is found in the constructor and the joining is found in the '''join()''' function as a member of the class.

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T03:20:42Z

Wjfisher: /* DOALL Parallelism */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP, where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism is not available for explicit usage using the library.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float *a, float *b;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== Reduction ==

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Explanation

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}
SimpleSum()

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T03:18:48Z

Wjfisher: /* DOALL Parallelism */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP, where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism is not available for explicit usage using the library.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

<source lang=cpp>// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float *a, float *b;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}</source>

== Reduction ==

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Explanation

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}
SimpleSum()

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T03:16:38Z

Wjfisher: /* DOALL Parallelism */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP, where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism is not available for explicit usage using the library.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. A simple sequential code loop is showed below followed by the same code being executed using parallel iterations with DOALL parallelism.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + 5;
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism in TBB. First of all, a class object must be created with a public function called '''operator()'''. This function should have a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created for any arguments are passed for '''parallel_for''' to operate properly. This is further explained by the example parallel loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
'''for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;'''
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float *a, float *b;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== Reduction ==

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Explanation

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}
SimpleSum()

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T03:13:06Z

Wjfisher: /* Intel Threading Building Blocks */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP, where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Instead they learn how to use the API for the TBB library with new or existing C++ code. Also, any compiler that supports ISO C++ can compile Intel® TBB code with a simple extra command. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS parallelism is not available for explicit usage using the library.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. For a DOALL loop, this second parameter should include all possible loop indices.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + c[i];
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism. First of all, a class object must be created with a public function called '''operator()''' with a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created, if any arguments are passed, for '''parallel_for''' to operate properly. This is further explained by the example DOALL loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float *a, float *b;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== Reduction ==

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Explanation

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}
SimpleSum()

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T03:10:28Z

Wjfisher: /* Intel Threading Building Blocks */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According to the [http://software.intel.com/en-us/intel-tbb/ Intel® Software site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP, where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Also, any compiler that supports ISO C++ can compile Intel® TBB code. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS and DOPIPE parallelism are not available for explicit usage here.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. For a DOALL loop, this second parameter should include all possible loop indices.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + c[i];
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism. First of all, a class object must be created with a public function called '''operator()''' with a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created, if any arguments are passed, for '''parallel_for''' to operate properly. This is further explained by the example DOALL loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float *a, float *b;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== Reduction ==

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Explanation

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}
SimpleSum()

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T03:09:21Z

Wjfisher: /* References */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According the to Intel® Software [http://software.intel.com/en-us/intel-tbb/ site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP, where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Also, any compiler that supports ISO C++ can compile Intel® TBB code. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS and DOPIPE parallelism are not available for explicit usage here.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. For a DOALL loop, this second parameter should include all possible loop indices.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + c[i];
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism. First of all, a class object must be created with a public function called '''operator()''' with a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created, if any arguments are passed, for '''parallel_for''' to operate properly. This is further explained by the example DOALL loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float *a, float *b;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== Reduction ==

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Explanation

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}
SimpleSum()

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* Blaise Barney, "POSIX Threads Programming", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/pthreads/ https://computing.llnl.gov/tutorials/pthreads/], January 2010.

* S. Bydiec, "Programming Posix ThreadsHuman Factor website[http://www.humanfactor.com/pthreads/ http://www.humanfactor.com/pthreads/]

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* OpenMP API website, [http://openmp.org/wp/ http://openmp.org/wp/]

* Blaise Barney, "OpenMP", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/], January 2009.

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-24T03:01:11Z

Wjfisher: /* DOALL Parallelism */

Parallel programming reduces execution time versus sequential programming by taking advantage of code structure. In practice, there are various C/C++ code libraries that offer parallel programming support without needing to learn a new language or programming model. These include Posix threads, Intel® Threading Building Blocks, and OpenMP. The focus of this article is to discuss the implementation of these libraries for parallel programming models related to loop structure, specifically DOALL, DOACROSS and DOPIPE parallelism, reduction, and functional parallelism.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 5

void *Foo(void *threadid)
{
long tid;
tid = (long)threadid;
....
....
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)t); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code above shows a very simple example of how to create threads using pthreads. Notice the arguments passed to the pthread_create() function, and note their syntax. For completeness, the example code above shows a simple function being run by every thread.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 50
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
tid = (long) data->threadid;
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->a[tid] = (double) data->a[tid - 1] + (double) data->b[tid];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

for(t=0; t<NUM_THREADS; t++){
printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[t], NULL, Foo, (void *)data); // create pthread
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
}
pthread_exit(NULL);
}

The code block above shows roughly how use conditional variable in order to exploit DOACROSS parallelism.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS 2
#define NUM_ELEMENTS 50

typedef struct
{
double a[NUM_ELEMENTS];
double b[NUM_ELEMENTS];
double c[NUM_ELEMENTS];
long threadid;
} DATA;

pthread_mutex_t mutexvar; //mutex variable required for conditional wait and signal functions
pthread_cond_t condvar; //conditional variable required for conditional wait and signal functions

void *Foo(void *data)
{
long tid;
int i;
tid = (long) data->threadid;
for(i = 0; i < NUM_ELEMENTS; i++)
{
data->a[i] = (double) data->a[i - 1] + (double) data->b[i];
pthread_cond_signal(&condvar); //signal that its safe to continue
pthread_mutex_unlock(&mutexvar);
}
pthread_exit(NULL);
}

void *Bar(void *data)
{
int i;
for(i = 0; i < NUM_ELEMENTS; i++)
{
pthread_mutex_lock(&mutexvar);
pthread_cond_wait(&condvar, &mutexvar); //wait until safe to continue
data->c[i] = (double) data->a[i];
}
}

int main (int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int rc;
long t;
DATA data;

//ASSUME data struct is initialized

//Initialized Mutex and Conditional Variable
pthread_mutex_init(&mutexvar, NULL);
pthread_cond_init (&condvar, NULL);

printf("In main: creating thread %ld\n", t);
rc = pthread_create(&threads[0], NULL, Foo, (void *)data); // create pthread that runs Foo
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
rc = pthread_create(&threads[1], NULL, Bar, (void *)data); // create pthread that runs Bar
}
pthread_exit(NULL);
}

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exists, all that is necessary is that the threads have to be created and have each thread execute some data set for a single function. If you you refer back up to the pthread creation example, that code block exploits DOALL parallelism.

= Intel Threading Building Blocks =

According the to Intel® Software [http://software.intel.com/en-us/intel-tbb/ site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

A goal of the library is similar to that of OpenMP, where the programmer is not required to have an extensive knowledge of thread programming or learn an entirely new language. Also, any compiler that supports ISO C++ can compile Intel® TBB code. However, as a drawback of its simplicity, certain types of parallelism such as DOACROSS and DOPIPE parallelism are not available for explicit usage here.

== Overhead ==

In order to use TBB, you must always insert the following code at the beginning of every file to include the TBB library and make its functions and variables available for use. In an attempt to simplify the examples below, this overhead will be omitted.

#include "tbb/tbb.h"

using namespace tbb;

== DOALL Parallelism ==

A DOALL parallel loop can be specified using the '''parallel_for()''' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is a set of operations that can be processed as a unit and are safe to run concurrently. For a DOALL loop, this second parameter should include all possible loop indices.

// sequential loop
for(size_t i = 0; i < n; i++) {
a[i] = b[i] + c[i];
}

A certain amount of overhead is necessary when creating a loop using DOALL parallelism. First of all, a class object must be created with a public function called '''operator()''' with a parameter of the form "const blocked_range<size_t>&" and a constructor must also be created, if any arguments are passed, for '''parallel_for''' to operate properly. This is further explained by the example DOALL loop below. This is a parallel version of the sequential loop above.

// parallel loop using Intel® TBB
class SimpleLoop {
float *a = my_a;
float *b = my_b;
public:
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *b = my_b;
for(size_t i = r.begin(); i <= r.end(); i++)
a[i] = b[i] + 5;
}
SimpleLoop(float *a, float *b):
my_a(a);
my_b(b);
};

void main(void) {
float *a, float *b;
...
...
parallel_for(blocked_range<size_t>(0,n), SimpleLoop(a, b));
}

== Reduction ==

// sequential loop summation
float sum = 0;
for(size_t i = 0; i < n; i++)
sum += a[i];

Explanation

// parallel reduction using Intel® TBB
class SimpleSum {
float *my_a;
public:
float *my_sum;
void operator(const blocked_range<size_t>& r) {
float *a = my_a;
float *sum = my_sum;
size_t end = r.end();
for(size_t i = r.begin(); i < r.end(); i++)
sum += a[i];
my_sum = sum;
}
SimpleSum()

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

#include <omp.h>

main () {

int nthreads, tid;

/* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
{

/* Obtain and print thread id */
tid = omp_get_thread_num();
printf("Hello World from thread = %d\n", tid);

/* Only master thread does this */
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("Number of threads = %d\n", nthreads);
}

} /* All threads join master thread and terminate */

}

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

#include<omp.h>
....
#pragma omp parallel
{
...
#pragma omp parallel for default(shared) // DOALL: all iterations are done in parallel
for(i=0; i<n; i++)
A[i] = B[i]+C[i];
}

The simple code block above shows how OpenMP 2.0 creates a parallel region that exploit DOALL parallelism. Notice how much this resemble sequential programming. If the compiler directives were taken out, this code could be run on a sequential machine.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

#include<omp.h>
...
#pragma omp parallel
{
...
#pragma omp section
for(i=0;i<n;i++)
A[i] = A[i] + B[i];
#pragma omp section
for(i=0;i<n;i++)
C[i]=C[i-1] + 5;
}

The code block above shows how to exploit function parallelism. Each section compiler directive tell the following code to be executed on a single thread. The code above is broken into 2 sections, which mean the two for loops are handled by 2 threads.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* https://computing.llnl.gov/tutorials/pthreads/

* http://www.humanfactor.com/pthreads/

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* http://openmp.org/wp/

* https://computing.llnl.gov/tutorials/openMP/

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-23T22:59:00Z

Wjfisher: /* Intel Threading Building Blocks */

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-23T22:33:17Z

Wjfisher: /* Intel Threading Building Blocks */

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-23T22:26:48Z

Wjfisher: /* DOALL Parallelism */

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-23T22:19:16Z

Wjfisher: /* Intel Threading Building Blocks */

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-23T22:02:15Z

Wjfisher: /* Intel Threading Building Blocks */

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-23T21:54:59Z

Wjfisher: /* Intel Threading Building Blocks */

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-23T21:23:37Z

Wjfisher:

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-23T21:12:04Z

Wjfisher: /* OpenMP 3.0 */

Supplement to Chapter 3: Support for parallel-programming models. Discuss how DOACROSS, DOPIPE, DOALL, etc. are implemented in packages such as Posix threads, Intel Thread Building Blocks, OpenMP 2.0 and 3.0.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exits. All that is necessary is that the threads have to be created.

= Intel Threading Building Blocks =

According the to Intel® Software [http://software.intel.com/en-us/intel-tbb/ site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

== Parallel Loops ==

A DOALL parallel construct can be specified using the ''parallel_for()'' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is the a solid chunk of operations that can be processed as a units that are safe to run concurrently. For a DOALL loop, this second parameter should include all possible loop indices. Also, an optional third parameter can be specified to define the chunk size of the loop and information about cache affinity.

= OpenMP 2.0 =
What is OpenMP? OpenMP or Open Multi-processing is a multi-platform API used for shared addressed space programming. There are versions of OpenMP for both C/C++ and Fortran. The OpenMP libraries provide a list of compiler directives that easily allow for one to write shared memory parallel programs. Before explaining how to exploit the different types of parallelism supported by OpenMP 2.0 it is necessary to understand how to create threads.

==Parallel Region==

In OpenMP 2.0 in order to create threads, the C/C++ compiler directive used is: #pragma omp parallel and is enclosed by curly brackets (See reference for Fortran directive). This directive means the start of a parallel region, and at the start of a parallel threads are created. In order to specify the number of threads that are created in a parallel region the function omp_set_num_threads(int n) is called.

Inside a parallel region different compiler directive can be used to exploit different types of parallelism. The two types of parallelism discussed below are DOALL and function parallelism. With OpenMP 2.0 DOACROSS and DOPIPE parallelism cannot be expressed through the use of compiler directives.

==DOALL parallelism==

OpenMP exploits DOALL parallelism through the use of a simple directive that tells the compiler to execute a section of code on multiple threads. The C/C++ directive is: #pragma omp parallel for. (See reference for Fortran Directive). This directive if placed before a normal sequential for loop in C/C++ will execute all iterations of the for loop in parallel.

==Function parallelism==

OpenMP 2.0 can also exploit function parallelism with compiler directives. The C/C++ directive is: #pragma omp section (See reference for Fortran Directive). This directive is placed before a section of code that is to be executed by a single thread. For function parallelism, multiple data independent code blocks can each be placed inside a parallel region with the section compiler directive placed before each block of code.

= OpenMP 3.0 =

In May 2008, OpenMP version 3.0 was released as an upgrade with some features such as tasks, synchronization primitives among others. For the scope of this article, nothing significant was upgraded regarding the execution of DOALL, DOACROSS, and DOPIPE parallelism, reduction, or function parallelism. Therefore, see sections above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* https://computing.llnl.gov/tutorials/pthreads/

* http://www.humanfactor.com/pthreads/

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* http://openmp.org/wp/

* https://computing.llnl.gov/tutorials/openMP/

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-23T21:04:56Z

Wjfisher: /* OpenMP 3.0 */

Supplement to Chapter 3: Support for parallel-programming models. Discuss how DOACROSS, DOPIPE, DOALL, etc. are implemented in packages such as Posix threads, Intel Thread Building Blocks, OpenMP 2.0 and 3.0.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exits. All that is necessary is that the threads have to be created.

= Intel Threading Building Blocks =

According the to Intel® Software [http://software.intel.com/en-us/intel-tbb/ site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

== Parallel Loops ==

A DOALL parallel construct can be specified using the ''parallel_for()'' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is the a solid chunk of operations that can be processed as a units that are safe to run concurrently. For a DOALL loop, this second parameter should include all possible loop indices. Also, an optional third parameter can be specified to define the chunk size of the loop and information about cache affinity.

= OpenMP 2.0 =

= OpenMP 3.0 =

Some features such as tasks, synchronization primitives among a few others were included in the upgrades from OpenMP 2.0 but nothing significant was upgraded that fits the scope of this article. Therefore, see section above about OpenMP 2.0 since all discussions there are applicable to version 3.0.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* https://computing.llnl.gov/tutorials/pthreads/

* http://www.humanfactor.com/pthreads/

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* http://openmp.org/wp/

* https://computing.llnl.gov/tutorials/openMP/

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-23T21:01:36Z

Wjfisher: /* References */

Supplement to Chapter 3: Support for parallel-programming models. Discuss how DOACROSS, DOPIPE, DOALL, etc. are implemented in packages such as Posix threads, Intel Thread Building Blocks, OpenMP 2.0 and 3.0.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==Creating and Terminating Pthreads==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

==Mutexes==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==Conditional Variables==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS Parallelism==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

==DOPIPE Parallelism==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

==DOALL Parallelism==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exits. All that is necessary is that the threads have to be created.

= Intel Threading Building Blocks =

According the to Intel® Software [http://software.intel.com/en-us/intel-tbb/ site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

== Parallel Loops ==

A DOALL parallel construct can be specified using the ''parallel_for()'' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is the a solid chunk of operations that can be processed as a units that are safe to run concurrently. For a DOALL loop, this second parameter should include all possible loop indices. Also, an optional third parameter can be specified to define the chunk size of the loop and information about cache affinity.

= OpenMP 2.0 =

= OpenMP 3.0 =

Some features such as tasks, synchronization primitives among a few others were included in the upgrades from OpenMP 2.0 but nothing significant was upgraded that fits the scope of this article.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* https://computing.llnl.gov/tutorials/pthreads/

* http://www.humanfactor.com/pthreads/

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* http://openmp.org/wp/

* https://computing.llnl.gov/tutorials/openMP/

* Mark Bull, University of Edinburgh, "OpenMP 3.0 Overview", [http://www.compunity.org/futures/Mark_SC06BOF.pdf http://www.compunity.org/futures/Mark_SC06BOF.pdf]

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-23T21:00:16Z

Wjfisher: /* References */

Supplement to Chapter 3: Support for parallel-programming models. Discuss how DOACROSS, DOPIPE, DOALL, etc. are implemented in packages such as Posix threads, Intel Thread Building Blocks, OpenMP 2.0 and 3.0.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

==CREATING AND TERMINATING PTHREADS==

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

==MUTEXES==

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

==CONDITIONAL VARIABLES==

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

==DOACROSS==

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

==DOPIPE==

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

==DOALL==

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exits. All that is necessary is that the threads have to be created.

= Intel Threading Building Blocks =

According the to Intel® Software [http://software.intel.com/en-us/intel-tbb/ site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

== Parallel Loops ==

A DOALL parallel construct can be specified using the ''parallel_for()'' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is the a solid chunk of operations that can be processed as a units that are safe to run concurrently. For a DOALL loop, this second parameter should include all possible loop indices. Also, an optional third parameter can be specified to define the chunk size of the loop and information about cache affinity.

= OpenMP 2.0 =

= OpenMP 3.0 =

Some features such as tasks, synchronization primitives among a few others were included in the upgrades from OpenMP 2.0 but nothing significant was upgraded that fits the scope of this article.

= References =
* Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

* https://computing.llnl.gov/tutorials/pthreads/

* http://www.humanfactor.com/pthreads/

* Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

* Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

* http://openmp.org/wp/

* https://computing.llnl.gov/tutorials/openMP/

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-23T20:57:15Z

Wjfisher: /* OpenMP 3.0 */

Supplement to Chapter 3: Support for parallel-programming models. Discuss how DOACROSS, DOPIPE, DOALL, etc. are implemented in packages such as Posix threads, Intel Thread Building Blocks, OpenMP 2.0 and 3.0.

= Posix threads =
POSIX thread, also referred to as a pthread is used in shared address space architectures for parallel programs. Through the use of the pthread API, various functions can be used to create and manage pthreads. In order to fully, understand how pthreads can be used to exploit DOACROSS, DOPIPE, and DOALL parallelism a brief introduction to creating/terminating pthreads, mutexes, and conditional variables are necessary.

CREATING AND TERMINATING PTHREADS

In order to create a pthread, the API provides the pthread_create() function. The pthread function accepts 4 arguments: thread, attr, start_routine, and arg. The thread argument is used to provide a unique identifier for the thread you are creating. The attr argument is used to specify a threads attribute object, or use default attributes by passing NULL. For the examples discussed the default attributes will be sufficient; for more information on setting thread attributes please see references. The start_routine argument is the program subroutine that will be executed by the thread being created. The arg argument is used to pass an argument to the subroutine that is being executed by the thread being created (value can be set to NULL if no argument is being pass to the subroutine).

In order to terminate pthreads, the API provides the pthread_exit() function. In order to terminate a pthread, the thread being run simply has to call this function (even the main thread). NOTE: there are alternate methods for terminating pthreads not discussed here for convenience and simplicity.

MUTEXES

A mutex variable is a variable that must only be accessed by a single thread (mutex is short for mutual exclusion). The API provides the pthread_mutex_t data type in order statically create a mutex variable, and the pthread_mutex_init() function to create it dynamically.

Mutex variables are used to implement locks, so that multiple pthreads do not access critical data in a program. The API provides the pthread_mutex_lock() and pthread_mutex_unlock() functions. These functions simply lock or unlock the mutex variable specified.

CONDITIONAL VARIABLES

Conditional variables allow for point-to-point synchronization between threads.
The API provides a few useful functions in order to synchronize threads: pthread_cond_wait() and pthread_cond_signal(). The pthread_cond_wait() blocks all threads until the specified condition is satisfied. The pthread_cond_signal() wakes up another thread that is waiting on the condition to be satisfied. These functions are the pthread specific functions that are analogous to the general wait() and post() function discussed in the Sohlihin text.

DOACROSS

In order to exploit DOACROSS parallelism using pthreads, conditions are needed in order to synchronize the threads. Since instructions are executed across iterations, and data dependencies exist across iterations are assumed (See Sohlin Text), the conditional variables shown above are used to ensure the correct execution of the code.

Lets take a simple example where each thread calculates A[i] = A[i-1] + B[i]. Point-to-point synchronization is necessary in order to make sure A[i-1] is not read before its value is written. This is where the pthread’s conditional variable comes is useful. We put pthread_cond_wait() and pthread_cond_signal() around the instruction above, in order to make sure the previous thread has signaled that its completed before the current thread performs its own computation.

DOPIPE

In order to exploit DOPIPE parallelism, conditions are also needed in order to synchronize threads. Instead of instructions being implemented across threads, instructions are implemented on a single thread (i.e. instruction 1 is executed by thread 1, instruction 2 is executed by thread 2, etc). However, there are loop independent data dependencies which require the conditional variables.

Here we use the pthreads differently from the DOACROSS parallelism. The DOPIPE parallelism has each thread call a different function. Each function may have some loop independent dependence with some other function. Lets say function 2 depends on function 1, so function 1 will call pthread_cond_signal() once it is finished, and function 2 will call pthread_cond_wait().

The differences between DOPIPE and DOACROSS are that DOPIPE executes different functions on each thread, and it has the signal and wait functions called from different functions. Whereas DOACROSS executes the same function on each thread (it just uses different data), and it has the signal and wait functions called from the same function.

DOALL

Since DOALL parallelism just means that all iterations are executed in parallel, and no dependences exits. All that is necessary is that the threads have to be created.

= Intel Threading Building Blocks =

According the to Intel® Software [http://software.intel.com/en-us/intel-tbb/ site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

== Parallel Loops ==

A DOALL parallel construct can be specified using the ''parallel_for()'' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is the a solid chunk of operations that can be processed as a units that are safe to run concurrently. For a DOALL loop, this second parameter should include all possible loop indices. Also, an optional third parameter can be specified to define the chunk size of the loop and information about cache affinity.

= OpenMP 2.0 =

= OpenMP 3.0 =

Some features such as tasks, synchronization primitives among a few others were included in the upgrades from OpenMP 2.0 but nothing significant was upgraded that fits the scope of this article.

= References =
- Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

- https://computing.llnl.gov/tutorials/pthreads/

- http://www.humanfactor.com/pthreads/

- Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

- Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

- http://openmp.org/wp/

- https://computing.llnl.gov/tutorials/openMP/

CSC/ECE 506 Spring 2010/ch 2 aj/Data Parallel Programming

2010-02-23T20:54:26Z

Wjfisher: /* Example of Data Parallel Programing Model */

Data-Parallel Programming Model

= Introduction =

The data-parallel programming model first appeared in the eighties as a programming model for SIMD (Single Instruction, Multiple Data) parallel machines. It's defined as multiple processing elements performing an action simultaneously on different parts of a data set and exchanging information globally before processing more code synchronously. Even though this model is offered in contrast to the shared memory model and the message passing model, data-parallel processing can actually be accomplished by passing messages or sharing variables. The main distinction between the data-parallel model and the other two models has to do with the outcome of the individual steps instead of the method of communication.

Tasks in this model work in conjunction on the same data structure. A distinct quality of the data parallelism model is that each task performs actions on different partitions of this data structure.

The model is generally associated with applications that involve a data set which is typically organized into a common structure, such as an array or matrix. Data-parallel processing has been found to be effective in situations where computations involve every element of a matrix in a uniform way. This allows the processing to be divided spatially over memories. Originally, the need for this model was discovered in scientific calculations and is still used today.

= Data parallelism vs Task parallelism =
One important feature of data-parallel programming model or data parallelism (SIMD) is the single control flow: there is only one control processor that directs the activities of all the processing elements. In stark contrast to this is task parallelism (MIMD: Multiple Instruction, Multiple Data): characterized by its multiple control flows, it allows the concurrent execution of multiple instruction streams, each manipulates its own data and services separate functions. Below is a contrast between the data parallelism and task parallelism models from wikipedia: [http://en.wikipedia.org/wiki/SIMD SIMD] and [http://en.wikipedia.org/wiki/MIMD MIMD]. In the following subsections we continue to compare and contrast different features of data-parallel model and task-parallel model to help reader understand the unique characteristics of data-parallel programming model.
[[Image:Smid.png|frame|center|425px|contrast between data parallelism and task parallelism]]

== Synchronous vs Asynchronous ==
While the [http://en.wikipedia.org/wiki/Lockstep_(computing) lockstep] imposed by data parallelism on all data streams ensures synchronous computation (all PEs perform their tasks at the exact same pace), every processor in task parallelism performs its task at their own pace, which we call asynchronous computation. Thus, at a certain point of a task parallel program's execution, communication and synchronization primitives are needed to allow different instruction streams to coordinate their efforts, and that is where variable-sharing and message-passing come into play.

== Determinism vs. Non-Determinism ==
Data parallelism's synchronous nature and task parallelism's asynchronism give rise to another pair of features that add to the difference between these two models: determinism versus non-determinism. Data parallelism is deterministic, i.e. computing with the same input will always yield the same result, since its synchronism ensures that issues like relative timing between PEs will not arise. In contrast, task parallelism's asynchronous updates of common data can give rise to non-determinism, i.e, the same input won't always yield the same computation result (the result of a computation will depend also on factors outside the program control, such as scheduling and timing of other PEs). Obviously, non-determinism makes it harder to write and maintain correct programs. This partially explains the advantage of data parallel programming model over data parallelism in terms of development effort (also discussed in section 4.2).

= Example of Data Parallel Programing Model =

This section shows a simple example adapted from Solihin textbook (pp. 24 - 27) that illustrates the data-parallel programming model. Each of the codes below are written in pseudo-code style.

Suppose we want to perform the following task on an array <code>a</code>: updating each element of <code>a</code> by the product of itself and its index, and adding together the elements of <code>a</code> into the variable <code>sum</code>. The corresponding code is shown below.

// simple sequential task
sum = 0;
'''for''' (i = 0; i < a.length; i++)
{
a[i] = a[i] * i;
sum = sum + a[i];
}

When we implement the task using the data-parallel programming model, the program can be divided into two parts. The first part performs the same operations on separate elements of the array for each processing element (sometimes referred to as PE or pe), and the second part reorganizes data among all processing elements (In our example data reorganization is summing up values across different processing elements). Since data-parallel programming model only defines the overall effects of parallel steps, the second part can be accomplished either through shared memory or message passing. The three code fragments below are examples for the first part of the program, shared-memory version of the second part, and message passing for the second part, respectively.

// data parallel programming: let each PE perform the same task on different pieces of distributed data
pe_id = getid();
my_sum = 0;
'''for''' (i = pe_id; i < a.length; i += number_of_pe) //separate elements of the array are assigned to each PE
{
a[i] = a[i] * i;
my_sum = my_sum + a[i]; //all PEs accumulate elements assigned to them into local variable my_sum
}

In the above code, data parallelism is achieved by letting each processing element perform actions on array's separate elements, which are identified using the PE's id. For instance, if three processing elements are used then one processing element would start at i = 0, one would start at i = 1, and the last would start at i = 2. Since there are three processing elements then the index of the array for each will increase by three on each iteration until the task is complete (note that in our example elements assigned to each PE are interleaved instead of continuous). If the length of the array is a multiple of three then each processing element takes the same amount of time to execute its portion of the task.

The picture below illustrates how elements of the array are assigned among different PEs for the specific case: length of the array is 7 and there are 3 PEs available. Elements in the array are marked by their indexes(0 to 6). As shown in the picture, PE0 will work on elements with index 0, 3, 6; PE1 is in charge of elements with index 1, 4; and elements with index 2, 5 are assigned to PE2. In this way, these 3 PEs work collectively on the array, while each PE works on different elements. Thus, data parallelism is achieved.

[[Image:506wiki1.png|frame|center|150px|Illustration of data parallel programming(adapted from [http://computing.llnl.gov/tutorials/parallel_comp/#ModelsData Introduction to Parallel Computing])]]

= Other Parallel Programming Models =
== Example Continued ==
As previously mentioned, the shared memory and message passing parallel programming models are commonly used. The following descriptions and code snippets are intended to develop a contrast between the data-parallel programming model and the other two models.

The code below shows how data reorganization is done by the shared memory programming model. A shared variable is declared to store the global sum. Each PE accumulates their local my_sum into this variable. In this example, the lock() and unlock() routines are used to prevent race conditions and barrier ensures that the all the local my_sum variables have been accumulated into the shared variable sum before the code proceeds. Notice that in this model there is only one copy of the data like in the data-parallel model. However, in the shared memory model, the data must be protected since any PE could be using it at any point, whereas data is associated with a specific PE in the data-parallel model.

// data reorganization via shared memory
'''shared''' sum;
lock(); //prevent race condition
sum = sum + my_sum; //each PE adds up their local my_sum to shared variable sum
unlock();
barrier;

The final section of code shows how data reorganization is done by the message passing programming model. PE0 acts as the one that collects the local my_sum variable from all the other PEs. This is done by the send_msg() and the recv_msg() routines: PEs other than PE0 send their my_sum variable as a message to PE0. PE0 receives the messages by specifying the sender in the '''for''' loop. PE ids are used in the message passing model in the same way they are in the data-parallel model. On the contrary, in the data-parallel model there is only one copy of data that is processed and there is no need to do any final accumulation by a single PE.

// data reorganization via message passing
'''if''' (pe_id != 0) send_msg (0, my_sum); //PE other than PE0 send their local my_sum
'''else''' //PE0 does this
{
'''for''' (i = 1; i < number_of_pe; i++) //for each other PE
{
recv_msg (i, temp); //receive local sum from other PEs
my_sum = my_sum + temp; //accumulate into total
}
sum = my_sum;
}

== Comparison in Different Aspects ==
The table below is extended from Table 2.1 on page 22 of the Solihin text. A column has been added for the data-parallel model. The table compares some key characteristics of each programming model. As you can see, the complexity of the data-parallel model comes mainly in the form of dividing up the work among PEs. Most of this work is done by the programmer and does not necessarily require special hardware, although, specific types of hardware can optimize the benefits of the model. For instance, SIMD and SPMD hardware are examples that have efficiently utilized the data-parallel model.

{| align="center" class="wikitable" style="text-align:center;" border="1" !cellpadding = "2"
|+
! Aspects !! Data-Parallel !! Shared Memory !! Message Passing
|-
! Communication
| implicit - via loads/stores || implicit - via loads/stores || explicit messages
|-
! Synchronization
| none || explicit || implicit - via messages
|-
! Hardware support
| none || typically required || none
|-
! Development effort
| low || low || high
|-
! Tuning effort
| high || high || low
|}

= References =

1) David E. Culler, Jaswinder Pal Singh, and Anoop Gupta, ''Parallel Computer Architecture: A Hardware/Software Approach'', Gulf Professional Publishing, August 1998.

2) Yan Solihin, ''Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems'', Solihin Books, August 2009.

3) Philip J. Hatcher, Michael Jay Quinn, ''Data-Parallel Programming on MIMD Computers'', The MIT Press, 1991.

4) Blaise Barney, "Introduction to Parallel Computing: Data Parallel Model", Lawrence Livermore National Laboratory, [https://computing.llnl.gov/tutorials/parallel_comp/#ModelsData https://computing.llnl.gov/tutorials/parallel_comp/#ModelsData], January 2009.

5) Guy Blelloch, "Is Parallel Programming Hard?", Carnegie Mellon University, [http://www.cilk.com/multicore-blog/bid/9108/Is-Parallel-Programming-Hard http://www.cilk.com/multicore-blog/bid/9108/Is-Parallel-Programming-Hard], April 2009.

6) Björn Lisper, ''Data parallelism and functional programming'', Lecture Notes in Computer Science, Volume 1132/1996, pp. 220-251, Springer Berlin, 1996.

7) ''SIMD'', Wikipedia, the free encyclopedia, [http://en.wikipedia.org/wiki/SIMD http://en.wikipedia.org/wiki/SIMD].

8) ''MIMD'', Wikipedia, the free encyclopedia, [http://en.wikipedia.org/wiki/MIMD http://en.wikipedia.org/wiki/MIMD].

9) ''Lockstep'', Wikipedia, the free encyclopedia, [http://en.wikipedia.org/wiki/Lockstep_(computing) http://en.wikipedia.org/wiki/Lockstep_(computing)].

10) ''SPMD'', Wikipedia, the free encyclopedia, [http://en.wikipedia.org/wiki/SPMD http://en.wikipedia.org/wiki/SPMD].

CSC/ECE 506 Spring 2010/ch 2 aj/Data Parallel Programming

2010-02-23T20:54:13Z

Wjfisher: /* Example of Data Parallel Programing Model */

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-23T03:38:39Z

Wjfisher: /* Intel Threading Building Blocks */

Supplement to Chapter 3: Support for parallel-programming models. Discuss how DOACROSS, DOPIPE, DOALL, etc. are implemented in packages such as Posix threads, Intel Thread Building Blocks, OpenMP 2.0 and 3.0.

= Posix threads =

= Intel Threading Building Blocks =

According the to Intel® Software [http://software.intel.com/en-us/intel-tbb/ site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

== Parallel Loops ==

A DOALL parallel construct can be specified using the ''parallel_for()'' construct. This construct takes two parameters. The first is the range of indices of the loop that can be run in parallel. The second is the a solid chunk of operations that can be processed as a units that are safe to run concurrently. For a DOALL loop, this second parameter should include all possible loop indices. Also, an optional third parameter can be specified to define the chunk size of the loop and information about cache affinity.

= OpenMP 2.0 =

= OpenMP 3.0 =

= References =
- Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

- https://computing.llnl.gov/tutorials/pthreads/

- http://www.humanfactor.com/pthreads/

- Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

- Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

- http://openmp.org/wp/

- https://computing.llnl.gov/tutorials/openMP/

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-22T21:59:42Z

Wjfisher: /* References */

Supplement to Chapter 3: Support for parallel-programming models. Discuss how DOACROSS, DOPIPE, DOALL, etc. are implemented in packages such as Posix threads, Intel Thread Building Blocks, OpenMP 2.0 and 3.0.

= Posix threads =

= Intel Threading Building Blocks =

According the to Intel® Software [http://software.intel.com/en-us/intel-tbb/ site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

= OpenMP 2.0 =

= OpenMP 3.0 =

= References =
- Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

- https://computing.llnl.gov/tutorials/pthreads/

- http://www.humanfactor.com/pthreads/

- Intel® Corporation, "Intel® TBB - Intel® Software Network", [http://software.intel.com/en-us/intel-tbb/ http://software.intel.com/en-us/intel-tbb/]

- Intel® Corporation, "Intel® Threading Building Blocks 2.2 for Open Source", [http://www.threadingbuildingblocks.org/ http://www.threadingbuildingblocks.org/]

- http://openmp.org/wp/

- https://computing.llnl.gov/tutorials/openMP/

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-22T21:51:46Z

Wjfisher:

Supplement to Chapter 3: Support for parallel-programming models. Discuss how DOACROSS, DOPIPE, DOALL, etc. are implemented in packages such as Posix threads, Intel Thread Building Blocks, OpenMP 2.0 and 3.0.

= Posix threads =

= Intel Threading Building Blocks =

According the to Intel® Software [http://software.intel.com/en-us/intel-tbb/ site], Intel® Threading Building Blocks (Intel® TBB) is a C++ template library that abstracts thread to tasks to create reliable, portable, and scalable parallel applications.

= OpenMP 2.0 =

= OpenMP 3.0 =

= References =
1) Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

2) https://computing.llnl.gov/tutorials/pthreads/

3) http://www.threadingbuildingblocks.org/

4) http://openmp.org/wp/

5) https://computing.llnl.gov/tutorials/openMP/

6) http://www.humanfactor.com/pthreads/

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-22T19:04:05Z

Wjfisher:

Supplement to Chapter 3: Support for parallel-programming models. Discuss how DOACROSS, DOPIPE, DOALL, etc. are implemented in packages such as Posix threads, Intel Thread Building Blocks, OpenMP 2.0 and 3.0.

= Posix threads =

= Intel Thread Building Blocks =

= OpenMP 2.0 =

= OpenMP 3.0 =

= References =
1) Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.

2) https://computing.llnl.gov/tutorials/pthreads/

3) http://www.threadingbuildingblocks.org/

4) http://openmp.org/wp/

5) https://computing.llnl.gov/tutorials/openMP/

6) http://www.humanfactor.com/pthreads/

CSC/ECE 506 Spring 2010/ch 3 jb/Parallel Programming Model Support

2010-02-22T18:57:28Z

Wjfisher:

Supplement to Chapter 3: Support for parallel-programming models. Discuss how DOACROSS, DOPIPE, DOALL, etc. are implemented in packages such as Posix threads, Intel Thread Building Blocks, OpenMP 2.0 and 3.0.

= Introduction =

= Packages =

== Posix threads ==

== Intel Thread Building Blocks ==

== OpenMP 2.0 ==

== OpenMP 3.0 ==

= References =
1) Yan Solihin, Fundamentals of Parallel Computer Architecture: Multichip and Multicore Systems, Solihin Books, August 2009.