Expertiza_Wiki - User contributions [en]

CSC/ECE 506 Fall 2007/wiki4 7 2815

2007-11-29T02:03:28Z

Mvrao: /* Prefetch loads/stores */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main program. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

* Make hard to predict branch predictions early

* Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB. The Branch Prediction Hardware is represented as in the block diagram below

[[Image:Branch.jpg]]

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetch loads/stores===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application. The figure shown below gives an idea of how the helper thread is constructed. Red spot indicates the problem load.

[[Image:Helper.jpg]]

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

* Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

* Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

File:Helper.jpg

2007-11-29T02:02:03Z

Mvrao: An example of how a helper thread is created. The red spot indicated problem load.

An example of how a helper thread is created. The red spot indicated problem load.

CSC/ECE 506 Fall 2007/wiki4 7 2815

2007-11-29T01:47:40Z

Mvrao: /* Predict difficult branches */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main program. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

* Make hard to predict branch predictions early

* Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB. The Branch Prediction Hardware is represented as in the block diagram below

[[Image:Branch.jpg]]

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetch loads/stores===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

* Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

* Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

File:Branch.jpg

2007-11-29T01:45:39Z

Mvrao: The helper thread construction hardware. Microthread in this figure means helper thread. Microthread Construction Buffer decides what instructions go into the helper thread

The helper thread construction hardware. Microthread in this figure means helper thread. Microthread Construction Buffer decides what instructions go into the helper thread

CSC/ECE 506 Fall 2007/wiki4 7 2815

2007-11-29T01:38:15Z

Mvrao:

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main program. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

* Make hard to predict branch predictions early

* Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetch loads/stores===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

* Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

* Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki4 7 2815

2007-11-29T01:37:33Z

Mvrao:

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main program. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

* Make hard to predict branch predictions early

* Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetch loads/stores===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

* Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

* Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki4 7 2815

2007-11-29T01:36:58Z

Mvrao:

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main program. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

* Make hard to predict branch predictions early

* Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetch loads/stores===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

* Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

* Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki4 7 2815

2007-11-29T01:36:09Z

Mvrao: Helper Threads and Applications

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main program. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

* Make hard to predict branch predictions early

* Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetch loads/stores===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

* Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

* Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:31:29Z

Mvrao: /* The Slipstream Approach */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main program. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

* Make hard to predict branch predictions early

* Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetch loads/stores===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

* Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

* Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:30:38Z

Mvrao: /* Helper Threads */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main program. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

* Make hard to predict branch predictions early

* Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetch loads/stores===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:24:34Z

Mvrao: /* Helper Threads */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main program. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetch loads/stores===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:23:37Z

Mvrao: /* Prefetching */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main programs. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetch loads/stores===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:22:24Z

Mvrao: /* Further reading */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main programs. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetching===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:22:09Z

Mvrao: /* References */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main programs. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetching===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:21:52Z

Mvrao: /* Conclusion */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main programs. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetching===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:21:33Z

Mvrao: /* The Slipstream Approach */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main programs. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetching===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:21:13Z

Mvrao: /* Applications of Helper threads */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main programs. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetching===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:20:47Z

Mvrao: /* Construction of Helper Threads */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main programs. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetching===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:20:12Z

Mvrao: /* Construction of Helper Threads */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main programs. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetching===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:19:54Z

Mvrao: /* Helper Threads */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main programs. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetching===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:19:20Z

Mvrao: /* Using Hardware */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main programs. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetching===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:18:58Z

Mvrao: /* Using Complier */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main programs. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetching===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:18:00Z

Mvrao: /* Introduction */

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main programs. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetching===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 7 2815

2007-11-29T01:17:38Z

Mvrao: Helper Threads and Applications

==Introduction==

The sole aim of all Computer Architects today is to increase the speed of the processor. If Instruction level parallelism (ILP) along with pipelining seems to be a potential solution for this motive, data and control dependencies seem to drag the speed by causing a much lower than ideal instructions per cycle (IPC). Deeper pipelines, which aim at increasing clock rate to increase processor speed of execution, turn out to be a hindrance when branch misprediction penalty becomes larger as more stages of pipeline need to be flushed and refilled.

To reduce the impact of these critical instructions, loads that miss in cache and difficult to predict branches, we need to note that the main program does calculate the values crucial for the address of load or for branch outcome, but not in a timely manner. If we somehow had the branch predictions and loads done much before the actual program needs it, the program could run faster. This requires that an additional program be run along with the main program that does such jobs and only such jobs. This additional program does everything to help main program run faster. This program, when implemented along with the main program on an advanced multiprocessor like CMP (where communication between multiple programs is not a big issue), there is significant increase in the running speed.

Such helpful programs that run along with the main program are called Helper threads.

==Helper Threads==

Helper threads are additional programs usually running the same code as the main programs. It runs faster than the main program to fetch values needed by the main program well in advance. Since the main motive of a helper thread is to “help” the main program, it doesn’t have to do everything that the main program does. In fact, if it does everything that the main program does, it would run as slow/fast as the main program which wouldn’t help much.

We need the helper thread mainly to take care of the following jobs:

1. Make hard to predict branch predictions early

2. Pre fetch irregular data (data which would definitely be a cache miss) into on-chip caches

Now, the challenge is to select the instructions included in the helper thread. It should be selected to make sure that instructions are not redundant but are yet relevant. Another challenge is to ensure that the helper thread is able to calculate the value needed to resolve a critical instruction in time. Timeliness is critical since a branch prediction is useless after the branch has been resolved and a pre fetch would simply be extra computation if the main thread already executed the load.

==Construction of Helper Threads==

Helper threads are seldom manually coded mainly because manual construction of helper threads is cumbersome and error-prone. Hence, it is desirable to automate this process. There are several ways proposed to construct these threads. We discuss the two main techniques here:

===Using Complier===

Construction of helper threads using compilers involves several steps:

1. '''Delinquent load identification''': Delinquent loads are those loads which cause the most cache misses. These are identified. This may be done by using performance analyzers such as Intel’s VTuneTM. In identifying such loads, the profile information is extracted where the compiler also keeps tracks of the cycle costs associated with those delinquent loads i.e., memory stall time. Finally, delinquent loads that account for a large portion of execution time are selected to be targeted by helper threads.

2. '''Loop Selection''': Research has shown that delinquent loads most likely occur within heavily traversed loop nest. Thus, loop structures can naturally be used for picking an appropriate region where helper threads will be constructed. In order to minimize the overhead of thread management, there are two goals. One is to minimize no. of helper thread invocations which can be done by keeping trip count of outer loop that encompasses the targeted loop small. Another goal is to make sure that once started, a helper thread runs for a considerable amount of time. This is to minimize thread activation cost. This can be achieved by having the targeted loop trip count maximum. This can be optimized by searching from the innermost loop which is most preferred target and going outward.

3. '''Slicing''': Next, the compiler identifies instructions to be executed in the helper thread. This process is called slicing. This process aims at keeping the instruction count to minimum to construct efficient and light weighted helper threads. Only the statements that affect the address computation of the delinquent load, including the change of the control flow, are selected as a slice and everything else is filtered out.

4. '''Live in variable identification''': Live in variables are those which form a part of helper thread’s available variables. They are global in nature. These variables are explicitly passed through global variables that are declared solely for the use of helper threads.

5. '''Code Generation''': After the compiler analysis phases, the constructed helper threads are attached to the application program as a separate code. In addition, the codes to create, schedule, and invoke the helper thread are inserted as well. Whenever the main thread encounters a trigger point, it first passes the function pointer of the corresponding helper thread and the live-in variables, and wakes up the helper thread. After being woken up, the helper thread indirectly jumps to the designated helper thread code region, reads in the live-ins, and starts execution. When the execution ends, the helper thread returns back to the dispatcher loop and goes to sleep again.

===Using Hardware===

As opposed to the compiler generation, this is the hardware method of creating helper threads. This makes use of post retirement buffers and scanners. Retire is a process in the pipeline when the execution of an instruction is complete and it is about to get the status updated. If there is an instruction path at the end of which there is a branch which is always difficult to predict, then it indicates a set of instructions that should go into the helper thread.
There is a buffer called post retirement buffer. Instructions from this targeted path are stored in the buffer after retirement. When such a path is identified, it is scanned before deciding what exactly goes into helper set. The scanner removes any instructions irrelevant to the branch, and puts the remaining instructions into the helper thread construction buffer. This is stored in a small content addressable memory chip called MircoRAM.

==Applications of Helper threads==

As discussed previously, helper threads are mainly used for the following applications:

===Predict difficult branches===

This uses the hardware mechanism of thread generation. A branch prediction is considered difficult based on the context in which the branch occurs. Taking the context of a branch into account makes these helper threads more efficient since helper threads will only be used when branches are truly difficult to predict. Addition of context also adds complexity into hardware.

The hardware used to make difficult branch predictions is called the Microthread Builder. There is a Post Retirement Buffer (PRB) which is used to store the last i instructions to retire from the primary or the main thread. Instructions enter the PRB after they retire and are pushed out as younger instructions are added. Dependency information, computed during instruction execution, is also stored in each PRB entry.

When a decision to predict branches with helper threads is made, the Microthread Builder extracts the data flow tree needed to compute the branch outcome. The PRB is frozen and scanned from youngest to oldest (the branch will always be the youngest instruction, as it just retired). Instructions making up the data flow tree are identified and extracted into the Microthread Construction Buffer (MCB). Microthread Construction Buffer is the place where the helper thread is constructed. The identification is not difficult, as the dependency information is already stored in the PRB.

Next step, an instruction is inserted at the end of the helper thread to communicate the branch outcome generated by the helper thread back to the front-end of the machine.

Last step is to recognize what is called ‘Spawn Point’. Spawn point determines when exactly a helper thread is called. It is crucial to start the helper thread as early as possible so the information is available prior to the main thread reaching the critical section. The spawn point should be chosen such that helper is not started very early which ends up making the helper thread have a lot of instructions. Helper thread should be started close enough to the critical situation so as to remain short and far enough to ensure that the critical situation has not passed by.

===Prefetching===

This makes use of a helper thread construction algorithm. This is pretty much similar to the compiler based thread generation discussed above.

The goal of the helper thread construction algorithm is to make the helper thread as short as possible, while still containing necessary instructions for generating correct prefetches to target loads/stores that likely miss in the L2 cache. To achieve this, prefetching regions are identified first. These are the regions of code that have a high concentration of L2 cache miss. The prefetching sections are typically loop nests consisting of several millions of dynamic instructions. Reads and writes that likely miss in a prefetching section will then be converted into prefetch instructions in the helper thread. The helper thread is spawned only once at the beginning of the application.

The following is the algorithm used to extract the prefetching helper thread for the identified loops:

1. Inline function calls in the loop. This means, the function body is replaced by function call instructions.

2. Identify target reads and writes to array elements or structures’ fields that likely miss in the L2 cache.

3. Starting from these read and writes, identify address computations. These are added into the address chain which determines the instruction addresses that form helper thread.

4. Privatize the locations that are written in address chain. If the location is an array element, substitute read/writes of the array in the address chain with read/writes to a new privatized array that belongs to the helper thread.

5. Replace target read and writes by prefetch instructions that access the same addresses.

6. Remove unnecessary computations, branches, and redundant prefetch instructions.

The helper thread is spawned at the beginning of the application. One complexity is that prefetching threads need to be spawned early to tolerate spawning overhead and cache miss latency.

==The Slipstream Approach==

The Slipstream paradigm proposes that only a subset of original dynamic instruction stream is needed to make full correct forward progress.

We know that processors run the entire set of dynamic instructions to determine the final output. The slip stream idea is to run a subset of this dynamic instruction stream in parallel with the original but a little ahead of it on a Chip Multiprocessor. This shorter program is similar to the helper thread we have been discussing.

This shorter program speculatively runs ahead of the full program and supplies the full program with control and data outcomes. The full program executes efficiently due to the communicated outcomes, at the same time validating the speculative, shorter program. The two programs combined run faster than the original program alone.

This can be better understood when we see how the name ‘Slipstream’ was originated. This was named after slipstreaming in stock car racing. At speeds in excess of 190 m.p.h., high air pressure forms at the front of a race car and a partial vacuum forms behind it. This creates drag and limits the car’s top speed. A second car can position itself close behind the first (a process called slipstreaming or drafting). This fills the vacuum behind the lead car, reducing its drag. And the trailing car now has less wind resistance in front (and by some accounts, the vacuum behind the lead car actually helps pull the trailing car). As a result, both cars speed up by several m.p.h.: the two combined go faster than either can alone.

With slipstream processors, the operating system creates two redundant processes. The two programs simultaneously run on a single chip multiprocessor. One of the programs always runs ahead of the other. The leading program is called Advanced Stream (A-stream) and the trailing program is called Redundant Stream (R-stream). Hardware monitors the R-stream and detects the following:

1. Dynamic instructions that repeatedly and predictably have no observable effect. (e.g., unreferenced writes, non-modifying writes)

2. Dynamic branches whose outcomes are consistently predicted correctly.

Such instructions are bypassed in the A-stream since it needs better jobs than just predict what can be predicted by R-stream as well.

This reduced A-stream is sped up because it fetches, executes, and retires fewer instructions than it would otherwise. Also, all values and branch outcomes produced in the leading A-stream are communicated to the trailing R-stream. Although the R-stream is not reduced in terms of retired instructions, it has an accurate picture of the future and fetches/executes more efficiently. In summary, the A-stream is sped up because it is shorter and the R-stream is sped up because it receives accurate predictions from the A-stream. The two redundant programs combined run faster than either can alone.

Hence we see that as in stock car racing, A-stream and R-stream mutually improve one another’s performance. The A-stream could not be accurately reduced without the trailing R-stream. And the R-stream is helped along in the slipstream (control and data flow outcomes) of the A-stream. The user perceives an overall speedup because both programs finish earlier.

==Conclusion==

From statistics, we can learn that the number of applications that can use helper threads in minimal. Most integer applications have fairly modest memory footprints, and therefore have few cache misses to eliminate. Many of those with large memory footprints, such as databases, are fairly easy to parallelize into true threads, which are always a better choice than “helper” threads if it is possible to create them. On the floating-point side, many applications are easily parallelizable or have fairly regular access patterns that can be prefetched using hardware mechanisms or occasional software prefetch instructions right in the main thread. As a result of these fundamental application characteristics, the selection of applications that can really be helped by helper threads is fairly limited.

A second problem is that very tight synchronization is needed between the main thread and its helpers in order to keep them the proper distance ahead of the main thread. Too far ahead, and they will cause cache thrashing by prefetching data and then replacing it with subsequent prefetches before the main thread can even use it. On the other hand, if they are not far enough ahead they might not be able to prefetch cache lines in time. Inserting just enough synchronization to keep these threads properly paced without slowing down the main thread significantly is still an active area for research.

However, applications like slipstreaming, Master-Slave Spec. Parallelization and Simultaneous Subordinate Multithreading have proven that helper thread are to remain in vogue.

==References==

1. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

2. [http://www.cs.ucsd.edu/~jbrown/research-exam/talk-print.pdf Helper Threading: Improving Single-Thread Performance Using Additional Execution Contexts]

3. [http://www.ece.rochester.edu/~mihuang/TEACHING/OLD/ECE404_SPRING03/HTsurvey.pdf A Survey on Helper Threads and Their Implementations]

4. [http://www.cgo.org/cgo2004/papers/02_80_Kim_D_REVISED.pdf Physical Experimentation with Prefetching Helper Threads on Intels Hyper-Threaded Processors]

5. [http://www.ece.ncsu.edu/arpers/Papers/ipdps06-threadpref.pdf Helper Thread Prefetching for Loosely-Coupled Multiprocessor Systems]

==Further reading==

1. [http://www-sal.cs.uiuc.edu/~zilles/papers/mssp.micro2002.pdf Master-Slave Spec. Parallelization]

2. [http://ieeexplore.ieee.org/iel5/6210/16584/00765950.pdf Simultaneous Subordinate Multithreading]

3. [http://www.tinker.ncsu.edu/ericro/publications/conference_ASPLOS-9.pdf Slipstream Processors]

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T02:22:03Z

Mvrao: /* Message Passing Interface (MPI) */

== Message Passing ==

When we have multiple processors, there needs to be a way to communicate between those processors. Message Passing forms a part of this communication architecture. There are other methods of communication like Shared Address Space and Data Parallel Processing, which along with Message Passing contribute to the communication abstraction. Communication abstraction is essentially a layer in between the application software and the communication hardware where the programmer uses available libraries to initiate communication between processors though programs.

===Message Passing Model===

Message Passing Model is defined as:

1. Set of Processes having only local memory

2. Processes communicate by sending and receiving messages

3. Transfer of data between processes requires cooperative operations to be performed by each process (a send operation must have a matching receive)

The message passing model has gained wide use in the field of parallel computing due to advantages that include:

1. Hardware match - The message passing model fits well on parallel supercomputers and clusters of workstations which are composed of separate processors connected by a communications network.

2. Functionality - Message passing offers a full set of functions for expressing parallel algorithms, providing the control not found in data-parallel model.

3. Performance - Message passing gives the programmer explicit control of data locality. This in turn enables effective management of memory and caches in CPUs.

===Latest Developments in Message Passing===

Although Message Passing Model as a whole has not changed over time, the Message Passing Interface (MPI) has undergone continuous change. MPI is a communications protocol used to program parallel computers. MPI is not sanctioned by any major standards body; nevertheless, it has become the de facto standard for communication among processes that model a parallel program running on a distributed memory system.

== Message Passing Interface (MPI) ==

It is a specification for message passing libraries, designed to be a standard for distributed memory, message passing and parallel computing. The goal of the Message Passing Interface simply stated is to provide a widely used standard for writing message-passing programs. The interface attempts to establish a practical, portable, efficient, and flexible standard for message passing.

===Advantages===

MPI is preferred over other implementations for several reasons like:

1. Standardization - MPI is the only message passing library which can be considered a standard. It is supported on virtually all High Performance Computing (HPC) platforms.

2. Portability – Modification of source code not required when the application is ported to a different platform that supports MPI.

3. Performance - vendor implementations should be able to exploit native hardware features to optimize performance.

4. Functionality (over 115 routines)

5. Availability - a variety of implementations are available, both vendor and public domain.

===MPI Implementations===

Some of the implementations of MPI include:

1. Classical Cluster and Supercomputer implementations

2. Python

3. OCaml

4. Java

5. Microsoft Windows

6. MATLAB

7. Hardware implementations

== Blade Servers ==

A blade server is a server chassis housing multiple thin, modular electronic circuit boards, known as server blades. Each blade is an independent server, often dedicated to a single application. The blades are servers on a card, containing processors, memory, integrated network controllers, an optional fiber channel host bus adaptor (HBA) and other input/output (IO) ports.

Blade servers allow more processing power in less rack space, simplifying cabling and reducing power consumption. According to [http://winsystems.com/ WinSystems] article on server technology, enterprises moving to blade servers can experience as much as an 85% reduction in cabling for blade installations over conventional 1U or tower servers. With so much less cabling, IT administrators can spend less time managing the infrastructure and more time ensuring high availability.

A blade server is sometimes referred to as a high-density server and is typically used in a clustering of servers that are dedicated to a single task, such as:

1. File sharing

2. Web page serving and caching

3. SSL encrypting of Web communication

4. The transcoding of Web page content for smaller displays

5. Streaming audio and video content

===Architecture===

A general blade server architecture consists of hardware components like the switch blade (for network signal switch functions), chassis (with fans, temperature sensors, etc), and multiple compute blades (for computer server functions). Blades that are application specific are positioned between switch blade and compute blades.

The outside world connects through the rear of the chassis to a switch card in the blade server. The switch card is provisioned to distribute packets to blades within the blade server. All these components are wrapped together with network management system software provided by the blade server vendor. The network management could be done through Message Passing which essentially makes blade servers an extension of message passing.

===Evolution===

Blade servers date back to 1970s. The evolution chronology as provided by [http://en.wikipedia.org/wiki/Blade_server#History Wikipedia(Article: Blade Servers)] can be summarized as below:

1. In the 1970s, soon after the introduction of 8-bit microprocessors, complete microcomputers were placed on cards and packaged in standard 19-inch racks. This architecture was used in the industrial process control industry as an alternative to minicomputer control systems. Programs were stored in EPROM on early models and were limited to a single function.

2. In 1981 the VMEBus architecture was designed in California. VMEBus architecture defined a computer interface which included implementation of a board-level computer that was installed in a chassis backplane with multiple slots for pluggable boards that provide I/O, memory, or additional computing. This architecture introduced the use of a chassis which forms the backbone of Blade servers today.

3. Later, PCI Industrial Computer Manufacturers Group (PICMG) developed a chassis/blade structure for Peripheral Component Interconnect bus PCI which was called CompactPCI. Though these chassis based computers included multiple computing elements to provide desired level of performance, there was always one master board coordinating the operation of the entire system.

4. In the next phase, PICMG expanded the CompactPCI specification with the use of standard Ethernet connectivity between boards across the backplane.

5. The first open architecture for a multi-server chassis was provided in Sept 2001 with the adoption of PICMG 2.16 CompactPCI Packet Switching Backplane specification. This was the closest form of present day blade server.

The name blade server is given to a card including the processor, memory, I/O and non-volatile program storage (flash memory or small hard disk(s)). This represents a complete server, with its operating system and applications packaged on a single card / board / blade. These blades operate independently within a common chassis, doing the work of multiple separate server boxes efficiently. Less space consumption is the most obvious benefit of this packaging, but additional efficiency benefits have become clear in power, cooling, management, and networking due to the sharing of common infrastructure to support the entire chassis, rather than providing each of these on a per server box basis.

=== Future ===

Early versions of server blades will be primarily high-density, low-power devices with relatively low performance. This type of blade is suited for first-tier applications such as static Web servers, security, network services, and streaming media because the applications can be easily and inexpensively load balanced. The performance of an application depends on the aggregate performance of the servers rather than the performance of an individual server.

Higher performance, less dense blade designs will help drive blade usage into more mainstream applications in the corporate data center. These designs can offer the individual performance characteristics and features available in today's rack-dense servers along with the cost, deployment, serviceability, and density benefits of server blades. The blades will be well suited to high-performance Web servers, dedicated application servers, server-based or thin-client computing, and high-performance computing (HPC) clusters. The introduction of server blades and associated technology like IB (InfiniBand) will usher in a new IT infrastructure.

== References ==

1. [http://www.hku.hk/cc/sp2/workshop/html/message_passing/message_passing.html#message1 Message Passing]

2. [http://www-unix.mcs.anl.gov/mpi/mpich2/#related MPI Implementation]

3. [http://www.dell.com/content/topics/global.aspx/power/en/ps1q02_blades?c=us&cs=555&l=en&s=biz Blade Servers]

4. [http://www.compactpci-systems.com/dl.php?pdf=/columns/software_corner/pdfs/3.03.pdf Blade Server Evolution]

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T02:18:42Z

Mvrao: /* Message Passing */

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T02:18:15Z

Mvrao: /* Message Passing */

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T02:17:51Z

Mvrao: /* References */

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T02:14:37Z

Mvrao: /* References */

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T02:14:21Z

Mvrao: /* Blade Servers */

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T02:13:01Z

Mvrao: /* Blade Servers */

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T02:12:35Z

Mvrao: /* Message Passing Interface (MPI) */

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T02:12:00Z

Mvrao: /* Message Passing */

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T02:11:22Z

Mvrao: /* Future */

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T02:07:57Z

Mvrao: /* Evolution */

== Message Passing ==

When we have multiple processors, there needs to be a way to communicate between those processors. Message Passing forms a part of this communication architecture. There are other methods of communication like Shared Address Space and Data Parallel Processing, which along with Message Passing contribute to the communication abstraction. Communication abstraction is essentially a layer in between the application software and the communication hardware where the programmer uses available libraries to initiate communication between processors though programs.

===Message Passing Model===

Message Passing Model is defined as:

1. Set of Processes having only local memory

2. Processes communicate by sending and receiving messages

3. Transfer of data between processes requires cooperative operations to be performed by each process (a send operation must have a matching receive)

The message passing model has gained wide use in the field of parallel computing due to advantages that include:

1. Hardware match - The message passing model fits well on parallel supercomputers and clusters of workstations which are composed of separate processors connected by a communications network.

2. Functionality - Message passing offers a full set of functions for expressing parallel algorithms, providing the control not found in data-parallel model.

3. Performance - Message passing gives the programmer explicit control of data locality. This in turn enables effective management of memory and caches in CPUs.

===Latest Developments in Message Passing===

Although Message Passing Model as a whole has not changed over time, the Message Passing Interface (MPI) has undergone continuous change. MPI is a communications protocol used to program parallel computers. MPI is not sanctioned by any major standards body; nevertheless, it has become the de facto standard for communication among processes that model a parallel program running on a distributed memory system.

== Message Passing Interface (MPI) ==

It is a specification for message passing libraries, designed to be a standard for distributed memory, message passing and parallel computing. The goal of the Message Passing Interface simply stated is to provide a widely used standard for writing message-passing programs. The interface attempts to establish a practical, portable, efficient, and flexible standard for message passing.

===Advantages===

MPI is preferred over other implementations for several reasons like:

1. Standardization - MPI is the only message passing library which can be considered a standard. It is supported on virtually all High Performance Computing (HPC) platforms.

2. Portability – Modification of source code not required when the application is ported to a different platform that supports MPI.

3. Performance - vendor implementations should be able to exploit native hardware features to optimize performance.

4. Functionality (over 115 routines)

5. Availability - a variety of implementations are available, both vendor and public domain.

===MPI Implementations===

Some of the implementations of MPI include:

1. Classical Cluster and Supercomputer implementations

2. Python

3. OCaml

4. Java

5. Microsoft Windows

6. MATLAB

7. Hardware implementations

== Blade Servers ==

A blade server is a server chassis housing multiple thin, modular electronic circuit boards, known as server blades. Each blade is an independent server, often dedicated to a single application. The blades are servers on a card, containing processors, memory, integrated network controllers, an optional fiber channel host bus adaptor (HBA) and other input/output (IO) ports.

Blade servers allow more processing power in less rack space, simplifying cabling and reducing power consumption. According to [http://winsystems.com/ WinSystems] article on server technology, enterprises moving to blade servers can experience as much as an 85% reduction in cabling for blade installations over conventional 1U or tower servers. With so much less cabling, IT administrators can spend less time managing the infrastructure and more time ensuring high availability.

A blade server is sometimes referred to as a high-density server and is typically used in a clustering of servers that are dedicated to a single task, such as:

1. File sharing

2. Web page serving and caching

3. SSL encrypting of Web communication

4. The transcoding of Web page content for smaller displays

5. Streaming audio and video content

===Architecture===

A general blade server architecture consists of hardware components like the switch blade (for network signal switch functions), chassis (with fans, temperature sensors, etc), and multiple compute blades (for computer server functions). Blades that are application specific are positioned between switch blade and compute blades.

The outside world connects through the rear of the chassis to a switch card in the blade server. The switch card is provisioned to distribute packets to blades within the blade server. All these components are wrapped together with network management system software provided by the blade server vendor. The network management could be done through Message Passing which essentially makes blade servers an extension of message passing.

===Evolution===

Blade servers date back to 1970s. The evolution chronology as provided by [http://en.wikipedia.org/wiki/Blade_server#History Wikipedia(Article: Blade Servers)] can be summarized as below:

1. In the 1970s, soon after the introduction of 8-bit microprocessors, complete microcomputers were placed on cards and packaged in standard 19-inch racks. This architecture was used in the industrial process control industry as an alternative to minicomputer control systems. Programs were stored in EPROM on early models and were limited to a single function.

2. In 1981 the VMEBus architecture was designed in California. VMEBus architecture defined a computer interface which included implementation of a board-level computer that was installed in a chassis backplane with multiple slots for pluggable boards that provide I/O, memory, or additional computing. This architecture introduced the use of a chassis which forms the backbone of Blade servers today.

3. Later, PCI Industrial Computer Manufacturers Group (PICMG) developed a chassis/blade structure for Peripheral Component Interconnect bus PCI which was called CompactPCI. Though these chassis based computers included multiple computing elements to provide desired level of performance, there was always one master board coordinating the operation of the entire system.

4. In the next phase, PICMG expanded the CompactPCI specification with the use of standard Ethernet connectivity between boards across the backplane.

5. The first open architecture for a multi-server chassis was provided in Sept 2001 with the adoption of PICMG 2.16 CompactPCI Packet Switching Backplane specification. This was the closest form of present day blade server.

The name blade server is given to a card including the processor, memory, I/O and non-volatile program storage (flash memory or small hard disk(s)). This represents a complete server, with its operating system and applications packaged on a single card / board / blade. These blades operate independently within a common chassis, doing the work of multiple separate server boxes efficiently. Less space consumption is the most obvious benefit of this packaging, but additional efficiency benefits have become clear in power, cooling, management, and networking due to the sharing of common infrastructure to support the entire chassis, rather than providing each of these on a per server box basis.

=== Future ===

Early versions of server blades will be primarily high-density, low-power devices with relatively low performance. This type of blade is suited for first-tier applications such as static Web servers, security, network services, and streaming media because the applications can be easily and inexpensively load balanced. The performance of an application depends on the aggregate performance of the servers rather than the performance of an individual server.
Higher performance, less dense blade designs will help drive blade usage into more mainstream applications in the corporate data center. These designs can offer the individual performance characteristics and features available in today's rack-dense servers along with the cost, deployment, serviceability, and density benefits of server blades. The blades will be well suited to high-performance Web servers, dedicated application servers, server-based or thin-client computing, and high-performance computing (HPC) clusters.

The introduction of server blades and associated technology like IB will usher in a new IT infrastructure. IT managers should start planning now for server blade installations by evaluating IP-based storage solutions, remote software provisioning and management solutions, scale-out architectures, and load-balancing technologies

== References ==

1. [http://www.hku.hk/cc/sp2/workshop/html/message_passing/message_passing.html#message1 Message Passing]

2. [http://www-unix.mcs.anl.gov/mpi/mpich2/#related MPI Implementation]

3. [http://www.dell.com/content/topics/global.aspx/power/en/ps1q02_blades?c=us&cs=555&l=en&s=biz Blade Servers]

4. [http://www.compactpci-systems.com/dl.php?pdf=/columns/software_corner/pdfs/3.03.pdf Blade Server Evolution]

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T02:06:26Z

Mvrao: /* Evolution */

== Message Passing ==

When we have multiple processors, there needs to be a way to communicate between those processors. Message Passing forms a part of this communication architecture. There are other methods of communication like Shared Address Space and Data Parallel Processing, which along with Message Passing contribute to the communication abstraction. Communication abstraction is essentially a layer in between the application software and the communication hardware where the programmer uses available libraries to initiate communication between processors though programs.

===Message Passing Model===

Message Passing Model is defined as:

1. Set of Processes having only local memory

2. Processes communicate by sending and receiving messages

3. Transfer of data between processes requires cooperative operations to be performed by each process (a send operation must have a matching receive)

The message passing model has gained wide use in the field of parallel computing due to advantages that include:

1. Hardware match - The message passing model fits well on parallel supercomputers and clusters of workstations which are composed of separate processors connected by a communications network.

2. Functionality - Message passing offers a full set of functions for expressing parallel algorithms, providing the control not found in data-parallel model.

3. Performance - Message passing gives the programmer explicit control of data locality. This in turn enables effective management of memory and caches in CPUs.

===Latest Developments in Message Passing===

Although Message Passing Model as a whole has not changed over time, the Message Passing Interface (MPI) has undergone continuous change. MPI is a communications protocol used to program parallel computers. MPI is not sanctioned by any major standards body; nevertheless, it has become the de facto standard for communication among processes that model a parallel program running on a distributed memory system.

== Message Passing Interface (MPI) ==

It is a specification for message passing libraries, designed to be a standard for distributed memory, message passing and parallel computing. The goal of the Message Passing Interface simply stated is to provide a widely used standard for writing message-passing programs. The interface attempts to establish a practical, portable, efficient, and flexible standard for message passing.

===Advantages===

MPI is preferred over other implementations for several reasons like:

1. Standardization - MPI is the only message passing library which can be considered a standard. It is supported on virtually all High Performance Computing (HPC) platforms.

2. Portability – Modification of source code not required when the application is ported to a different platform that supports MPI.

3. Performance - vendor implementations should be able to exploit native hardware features to optimize performance.

4. Functionality (over 115 routines)

5. Availability - a variety of implementations are available, both vendor and public domain.

===MPI Implementations===

Some of the implementations of MPI include:

1. Classical Cluster and Supercomputer implementations

2. Python

3. OCaml

4. Java

5. Microsoft Windows

6. MATLAB

7. Hardware implementations

== Blade Servers ==

A blade server is a server chassis housing multiple thin, modular electronic circuit boards, known as server blades. Each blade is an independent server, often dedicated to a single application. The blades are servers on a card, containing processors, memory, integrated network controllers, an optional fiber channel host bus adaptor (HBA) and other input/output (IO) ports.

Blade servers allow more processing power in less rack space, simplifying cabling and reducing power consumption. According to [http://winsystems.com/ WinSystems] article on server technology, enterprises moving to blade servers can experience as much as an 85% reduction in cabling for blade installations over conventional 1U or tower servers. With so much less cabling, IT administrators can spend less time managing the infrastructure and more time ensuring high availability.

A blade server is sometimes referred to as a high-density server and is typically used in a clustering of servers that are dedicated to a single task, such as:

1. File sharing

2. Web page serving and caching

3. SSL encrypting of Web communication

4. The transcoding of Web page content for smaller displays

5. Streaming audio and video content

===Architecture===

A general blade server architecture consists of hardware components like the switch blade (for network signal switch functions), chassis (with fans, temperature sensors, etc), and multiple compute blades (for computer server functions). Blades that are application specific are positioned between switch blade and compute blades.

The outside world connects through the rear of the chassis to a switch card in the blade server. The switch card is provisioned to distribute packets to blades within the blade server. All these components are wrapped together with network management system software provided by the blade server vendor. The network management could be done through Message Passing which essentially makes blade servers an extension of message passing.

===Evolution===

Blade servers date back to 1970s. The evolution chronology as provided by Wikipedia (Article: Blade Servers) can be summarized as below:

1. In the 1970s, soon after the introduction of 8-bit microprocessors, complete microcomputers were placed on cards and packaged in standard 19-inch racks. This architecture was used in the industrial process control industry as an alternative to minicomputer control systems. Programs were stored in EPROM on early models and were limited to a single function.

2. In 1981 the VMEBus architecture was designed in California. VMEBus architecture defined a computer interface which included implementation of a board-level computer that was installed in a chassis backplane with multiple slots for pluggable boards that provide I/O, memory, or additional computing. This architecture introduced the use of a chassis which forms the backbone of Blade servers today.

3. Later, PCI Industrial Computer Manufacturers Group (PICMG) developed a chassis/blade structure for Peripheral Component Interconnect bus PCI which was called CompactPCI. Though these chassis based computers included multiple computing elements to provide desired level of performance, there was always one master board coordinating the operation of the entire system.

4. In the next phase, PICMG expanded the CompactPCI specification with the use of standard Ethernet connectivity between boards across the backplane.

5. The first open architecture for a multi-server chassis was provided in Sept 2001 with the adoption of PICMG 2.16 CompactPCI Packet Switching Backplane specification. This was the closest form of present day blade server.

The name blade server is given to a card including the processor, memory, I/O and non-volatile program storage (flash memory or small hard disk(s)). This represents a complete server, with its operating system and applications packaged on a single card / board / blade. These blades operate independently within a common chassis, doing the work of multiple separate server boxes efficiently. Less space consumption is the most obvious benefit of this packaging, but additional efficiency benefits have become clear in power, cooling, management, and networking due to the sharing of common infrastructure to support the entire chassis, rather than providing each of these on a per server box basis.

=== Future ===

Early versions of server blades will be primarily high-density, low-power devices with relatively low performance. This type of blade is suited for first-tier applications such as static Web servers, security, network services, and streaming media because the applications can be easily and inexpensively load balanced. The performance of an application depends on the aggregate performance of the servers rather than the performance of an individual server.
Higher performance, less dense blade designs will help drive blade usage into more mainstream applications in the corporate data center. These designs can offer the individual performance characteristics and features available in today's rack-dense servers along with the cost, deployment, serviceability, and density benefits of server blades. The blades will be well suited to high-performance Web servers, dedicated application servers, server-based or thin-client computing, and high-performance computing (HPC) clusters.

The introduction of server blades and associated technology like IB will usher in a new IT infrastructure. IT managers should start planning now for server blade installations by evaluating IP-based storage solutions, remote software provisioning and management solutions, scale-out architectures, and load-balancing technologies

== References ==

1. [http://www.hku.hk/cc/sp2/workshop/html/message_passing/message_passing.html#message1 Message Passing]

2. [http://www-unix.mcs.anl.gov/mpi/mpich2/#related MPI Implementation]

3. [http://www.dell.com/content/topics/global.aspx/power/en/ps1q02_blades?c=us&cs=555&l=en&s=biz Blade Servers]

4. [http://www.compactpci-systems.com/dl.php?pdf=/columns/software_corner/pdfs/3.03.pdf Blade Server Evolution]

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T01:04:12Z

Mvrao: /* Architecture */

== Message Passing ==

When we have multiple processors, there needs to be a way to communicate between those processors. Message Passing forms a part of this communication architecture. There are other methods of communication like Shared Address Space and Data Parallel Processing, which along with Message Passing contribute to the communication abstraction. Communication abstraction is essentially a layer in between the application software and the communication hardware where the programmer uses available libraries to initiate communication between processors though programs.

===Message Passing Model===

Message Passing Model is defined as:

1. Set of Processes having only local memory

2. Processes communicate by sending and receiving messages

3. Transfer of data between processes requires cooperative operations to be performed by each process (a send operation must have a matching receive)

The message passing model has gained wide use in the field of parallel computing due to advantages that include:

1. Hardware match - The message passing model fits well on parallel supercomputers and clusters of workstations which are composed of separate processors connected by a communications network.

2. Functionality - Message passing offers a full set of functions for expressing parallel algorithms, providing the control not found in data-parallel model.

3. Performance - Message passing gives the programmer explicit control of data locality. This in turn enables effective management of memory and caches in CPUs.

===Latest Developments in Message Passing===

Although Message Passing Model as a whole has not changed over time, the Message Passing Interface (MPI) has undergone continuous change. MPI is a communications protocol used to program parallel computers. MPI is not sanctioned by any major standards body; nevertheless, it has become the de facto standard for communication among processes that model a parallel program running on a distributed memory system.

== Message Passing Interface (MPI) ==

It is a specification for message passing libraries, designed to be a standard for distributed memory, message passing and parallel computing. The goal of the Message Passing Interface simply stated is to provide a widely used standard for writing message-passing programs. The interface attempts to establish a practical, portable, efficient, and flexible standard for message passing.

===Advantages===

MPI is preferred over other implementations for several reasons like:

1. Standardization - MPI is the only message passing library which can be considered a standard. It is supported on virtually all High Performance Computing (HPC) platforms.

2. Portability – Modification of source code not required when the application is ported to a different platform that supports MPI.

3. Performance - vendor implementations should be able to exploit native hardware features to optimize performance.

4. Functionality (over 115 routines)

5. Availability - a variety of implementations are available, both vendor and public domain.

===MPI Implementations===

Some of the implementations of MPI include:

1. Classical Cluster and Supercomputer implementations

2. Python

3. OCaml

4. Java

5. Microsoft Windows

6. MATLAB

7. Hardware implementations

== Blade Servers ==

A blade server is a server chassis housing multiple thin, modular electronic circuit boards, known as server blades. Each blade is an independent server, often dedicated to a single application. The blades are servers on a card, containing processors, memory, integrated network controllers, an optional fiber channel host bus adaptor (HBA) and other input/output (IO) ports.

Blade servers allow more processing power in less rack space, simplifying cabling and reducing power consumption. According to [http://winsystems.com/ WinSystems] article on server technology, enterprises moving to blade servers can experience as much as an 85% reduction in cabling for blade installations over conventional 1U or tower servers. With so much less cabling, IT administrators can spend less time managing the infrastructure and more time ensuring high availability.

A blade server is sometimes referred to as a high-density server and is typically used in a clustering of servers that are dedicated to a single task, such as:

1. File sharing

2. Web page serving and caching

3. SSL encrypting of Web communication

4. The transcoding of Web page content for smaller displays

5. Streaming audio and video content

===Architecture===

A general blade server architecture consists of hardware components like the switch blade (for network signal switch functions), chassis (with fans, temperature sensors, etc), and multiple compute blades (for computer server functions). Blades that are application specific are positioned between switch blade and compute blades.

The outside world connects through the rear of the chassis to a switch card in the blade server. The switch card is provisioned to distribute packets to blades within the blade server. All these components are wrapped together with network management system software provided by the blade server vendor. The network management could be done through Message Passing which essentially makes blade servers an extension of message passing.

===Evolution===

The name blade server appeared when a card included the processor, memory, I/O and non-volatile program storage (flash memory or small hard disk(s)). This allowed a complete server, with its operating system and applications, to be packaged on a single card / board / blade. These blades could then operate independently within a common chassis, doing the work of multiple separate server boxes more efficiently. Less space consumption is the most obvious benefit of this packaging, but additional efficiency benefits have become clear in power, cooling, management, and networking due to the pooling or sharing of common infrastructure to supports the entire chassis, rather than providing each of these on a per server box basis.
Blade servers date back to 1970s. The evolution chronology as provided by Wikipedia (Article: Blade Servers) is given below:
Complete microcomputers were placed on cards and packaged in standard 19-inch racks in the 1970s soon after the introduction of 8-bit microprocessors. This architecture was used in the industrial process control industry as an alternative to minicomputer control systems. Programs were stored in EPROM on early models and were limited to a single function with a small realtime executive.

The VMEBus architecture (ca. 1981) defined a computer interface which included implementation of a board-level computer that was installed in a chassis backplane with multiple slots for pluggable boards that provide I/O, memory, or additional computing. The PCI Industrial Computer Manufacturers Group PICMG developed a chassis/blade structure for the then emerging Peripheral Component Interconnect bus PCI which is called CompactPCI. Common among these chassis based computers was the fact that the entire chassis was a single system. While a chassis might include multiple computing elements to provide the desired level of performance and redundancy, there was always one board in charge, one master board coordinating the operation of the entire system. PICMG expanded the CompactPCI specification with the use of standard Ethernet connectivity between boards across the backplane. The PICMG 2.16 CompactPCI Packet Switching Backplane specification was adopted in Sept 2001 (PICMG specifications). This provided the first open architecture for a multi-server chassis. PICMG followed with the larger and more feature rich AdvancedTCA specification targeting the telecom industry's need for a high availability and dense computing platform with extended product life (10+ years). While AdvancedTCA system and board pricing is typically higher than blade servers, AdvancedTCA suppliers claim that low operating expenses and total cost of ownership can make AdvancedTCA-based solutions a cost effective alternative for many building blocks of the next generation telecom network.

=== Future ===

Early versions of server blades will be primarily high-density, low-power devices with relatively low performance. This type of blade is suited for first-tier applications such as static Web servers, security, network services, and streaming media because the applications can be easily and inexpensively load balanced. The performance of an application depends on the aggregate performance of the servers rather than the performance of an individual server.
Higher performance, less dense blade designs will help drive blade usage into more mainstream applications in the corporate data center. These designs can offer the individual performance characteristics and features available in today's rack-dense servers along with the cost, deployment, serviceability, and density benefits of server blades. The blades will be well suited to high-performance Web servers, dedicated application servers, server-based or thin-client computing, and high-performance computing (HPC) clusters.

The introduction of server blades and associated technology like IB will usher in a new IT infrastructure. IT managers should start planning now for server blade installations by evaluating IP-based storage solutions, remote software provisioning and management solutions, scale-out architectures, and load-balancing technologies

== References ==

1. [http://www.hku.hk/cc/sp2/workshop/html/message_passing/message_passing.html#message1 Message Passing]

2. [http://www-unix.mcs.anl.gov/mpi/mpich2/#related MPI Implementation]

3. [http://www.dell.com/content/topics/global.aspx/power/en/ps1q02_blades?c=us&cs=555&l=en&s=biz Blade Servers]

4. [http://www.compactpci-systems.com/dl.php?pdf=/columns/software_corner/pdfs/3.03.pdf Blade Server Evolution]

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T00:50:48Z

Mvrao: /* Blade Servers */

== Message Passing ==

When we have multiple processors, there needs to be a way to communicate between those processors. Message Passing forms a part of this communication architecture. There are other methods of communication like Shared Address Space and Data Parallel Processing, which along with Message Passing contribute to the communication abstraction. Communication abstraction is essentially a layer in between the application software and the communication hardware where the programmer uses available libraries to initiate communication between processors though programs.

===Message Passing Model===

Message Passing Model is defined as:

1. Set of Processes having only local memory

2. Processes communicate by sending and receiving messages

3. Transfer of data between processes requires cooperative operations to be performed by each process (a send operation must have a matching receive)

The message passing model has gained wide use in the field of parallel computing due to advantages that include:

1. Hardware match - The message passing model fits well on parallel supercomputers and clusters of workstations which are composed of separate processors connected by a communications network.

2. Functionality - Message passing offers a full set of functions for expressing parallel algorithms, providing the control not found in data-parallel model.

3. Performance - Message passing gives the programmer explicit control of data locality. This in turn enables effective management of memory and caches in CPUs.

===Latest Developments in Message Passing===

Although Message Passing Model as a whole has not changed over time, the Message Passing Interface (MPI) has undergone continuous change. MPI is a communications protocol used to program parallel computers. MPI is not sanctioned by any major standards body; nevertheless, it has become the de facto standard for communication among processes that model a parallel program running on a distributed memory system.

== Message Passing Interface (MPI) ==

It is a specification for message passing libraries, designed to be a standard for distributed memory, message passing and parallel computing. The goal of the Message Passing Interface simply stated is to provide a widely used standard for writing message-passing programs. The interface attempts to establish a practical, portable, efficient, and flexible standard for message passing.

===Advantages===

MPI is preferred over other implementations for several reasons like:

1. Standardization - MPI is the only message passing library which can be considered a standard. It is supported on virtually all High Performance Computing (HPC) platforms.

2. Portability – Modification of source code not required when the application is ported to a different platform that supports MPI.

3. Performance - vendor implementations should be able to exploit native hardware features to optimize performance.

4. Functionality (over 115 routines)

5. Availability - a variety of implementations are available, both vendor and public domain.

===MPI Implementations===

Some of the implementations of MPI include:

1. Classical Cluster and Supercomputer implementations

2. Python

3. OCaml

4. Java

5. Microsoft Windows

6. MATLAB

7. Hardware implementations

== Blade Servers ==

A blade server is a server chassis housing multiple thin, modular electronic circuit boards, known as server blades. Each blade is an independent server, often dedicated to a single application. The blades are servers on a card, containing processors, memory, integrated network controllers, an optional fiber channel host bus adaptor (HBA) and other input/output (IO) ports.

Blade servers allow more processing power in less rack space, simplifying cabling and reducing power consumption. According to [http://winsystems.com/ WinSystems] article on server technology, enterprises moving to blade servers can experience as much as an 85% reduction in cabling for blade installations over conventional 1U or tower servers. With so much less cabling, IT administrators can spend less time managing the infrastructure and more time ensuring high availability.

A blade server is sometimes referred to as a high-density server and is typically used in a clustering of servers that are dedicated to a single task, such as:

1. File sharing

2. Web page serving and caching

3. SSL encrypting of Web communication

4. The transcoding of Web page content for smaller displays

5. Streaming audio and video content

===Architecture===

A general blade server architecture is shown in the figure below. The hardware components of a blade server are the switch blade, chassis (with fans, temperature sensors, etc), and multiple compute blades. Some vendors offer, partner, or plan to partner with companies that provide application specific blades that provide traffic conditioning, protection, or network processing prior to the traffic reaching the compute blades. Often, these application specific blades may be functionally positioned between the switch blade and compute blades. However, these blades reside in a standard compute blade slot.
The outside world connects through the rear of the chassis to a switch card in the blade server. The switch card is provisioned to distribute packets to blades within the blade server. All these components are wrapped together with network management system software provided by the blade server vendor. The network management could be done through Message Passing which essentially makes blade servers an extension of message passing.

===Evolution===

The name blade server appeared when a card included the processor, memory, I/O and non-volatile program storage (flash memory or small hard disk(s)). This allowed a complete server, with its operating system and applications, to be packaged on a single card / board / blade. These blades could then operate independently within a common chassis, doing the work of multiple separate server boxes more efficiently. Less space consumption is the most obvious benefit of this packaging, but additional efficiency benefits have become clear in power, cooling, management, and networking due to the pooling or sharing of common infrastructure to supports the entire chassis, rather than providing each of these on a per server box basis.
Blade servers date back to 1970s. The evolution chronology as provided by Wikipedia (Article: Blade Servers) is given below:
Complete microcomputers were placed on cards and packaged in standard 19-inch racks in the 1970s soon after the introduction of 8-bit microprocessors. This architecture was used in the industrial process control industry as an alternative to minicomputer control systems. Programs were stored in EPROM on early models and were limited to a single function with a small realtime executive.

The VMEBus architecture (ca. 1981) defined a computer interface which included implementation of a board-level computer that was installed in a chassis backplane with multiple slots for pluggable boards that provide I/O, memory, or additional computing. The PCI Industrial Computer Manufacturers Group PICMG developed a chassis/blade structure for the then emerging Peripheral Component Interconnect bus PCI which is called CompactPCI. Common among these chassis based computers was the fact that the entire chassis was a single system. While a chassis might include multiple computing elements to provide the desired level of performance and redundancy, there was always one board in charge, one master board coordinating the operation of the entire system. PICMG expanded the CompactPCI specification with the use of standard Ethernet connectivity between boards across the backplane. The PICMG 2.16 CompactPCI Packet Switching Backplane specification was adopted in Sept 2001 (PICMG specifications). This provided the first open architecture for a multi-server chassis. PICMG followed with the larger and more feature rich AdvancedTCA specification targeting the telecom industry's need for a high availability and dense computing platform with extended product life (10+ years). While AdvancedTCA system and board pricing is typically higher than blade servers, AdvancedTCA suppliers claim that low operating expenses and total cost of ownership can make AdvancedTCA-based solutions a cost effective alternative for many building blocks of the next generation telecom network.

=== Future ===

Early versions of server blades will be primarily high-density, low-power devices with relatively low performance. This type of blade is suited for first-tier applications such as static Web servers, security, network services, and streaming media because the applications can be easily and inexpensively load balanced. The performance of an application depends on the aggregate performance of the servers rather than the performance of an individual server.
Higher performance, less dense blade designs will help drive blade usage into more mainstream applications in the corporate data center. These designs can offer the individual performance characteristics and features available in today's rack-dense servers along with the cost, deployment, serviceability, and density benefits of server blades. The blades will be well suited to high-performance Web servers, dedicated application servers, server-based or thin-client computing, and high-performance computing (HPC) clusters.

The introduction of server blades and associated technology like IB will usher in a new IT infrastructure. IT managers should start planning now for server blade installations by evaluating IP-based storage solutions, remote software provisioning and management solutions, scale-out architectures, and load-balancing technologies

== References ==

1. [http://www.hku.hk/cc/sp2/workshop/html/message_passing/message_passing.html#message1 Message Passing]

2. [http://www-unix.mcs.anl.gov/mpi/mpich2/#related MPI Implementation]

3. [http://www.dell.com/content/topics/global.aspx/power/en/ps1q02_blades?c=us&cs=555&l=en&s=biz Blade Servers]

4. [http://www.compactpci-systems.com/dl.php?pdf=/columns/software_corner/pdfs/3.03.pdf Blade Server Evolution]

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T00:38:14Z

Mvrao: /* Message Passing Interface (MPI) */

== Message Passing ==

When we have multiple processors, there needs to be a way to communicate between those processors. Message Passing forms a part of this communication architecture. There are other methods of communication like Shared Address Space and Data Parallel Processing, which along with Message Passing contribute to the communication abstraction. Communication abstraction is essentially a layer in between the application software and the communication hardware where the programmer uses available libraries to initiate communication between processors though programs.

===Message Passing Model===

Message Passing Model is defined as:

1. Set of Processes having only local memory

2. Processes communicate by sending and receiving messages

3. Transfer of data between processes requires cooperative operations to be performed by each process (a send operation must have a matching receive)

The message passing model has gained wide use in the field of parallel computing due to advantages that include:

1. Hardware match - The message passing model fits well on parallel supercomputers and clusters of workstations which are composed of separate processors connected by a communications network.

2. Functionality - Message passing offers a full set of functions for expressing parallel algorithms, providing the control not found in data-parallel model.

3. Performance - Message passing gives the programmer explicit control of data locality. This in turn enables effective management of memory and caches in CPUs.

===Latest Developments in Message Passing===

Although Message Passing Model as a whole has not changed over time, the Message Passing Interface (MPI) has undergone continuous change. MPI is a communications protocol used to program parallel computers. MPI is not sanctioned by any major standards body; nevertheless, it has become the de facto standard for communication among processes that model a parallel program running on a distributed memory system.

== Message Passing Interface (MPI) ==

It is a specification for message passing libraries, designed to be a standard for distributed memory, message passing and parallel computing. The goal of the Message Passing Interface simply stated is to provide a widely used standard for writing message-passing programs. The interface attempts to establish a practical, portable, efficient, and flexible standard for message passing.

===Advantages===

MPI is preferred over other implementations for several reasons like:

1. Standardization - MPI is the only message passing library which can be considered a standard. It is supported on virtually all High Performance Computing (HPC) platforms.

2. Portability – Modification of source code not required when the application is ported to a different platform that supports MPI.

3. Performance - vendor implementations should be able to exploit native hardware features to optimize performance.

4. Functionality (over 115 routines)

5. Availability - a variety of implementations are available, both vendor and public domain.

===MPI Implementations===

Some of the implementations of MPI include:

1. Classical Cluster and Supercomputer implementations

2. Python

3. OCaml

4. Java

5. Microsoft Windows

6. MATLAB

7. Hardware implementations

== Blade Servers ==

A blade server is a server chassis housing multiple thin, modular electronic circuit boards, known as server blades. Each blade is a server in its own right, often dedicated to a single application. The blades are literally servers on a card, containing processors, memory, integrated network controllers, an optional fiber channel host bus adaptor (HBA) and other input/output (IO) ports.

Blade servers allow more processing power in less rack space, simplifying cabling and reducing power consumption. According to a Search, WinSystems.com article on server technology, enterprises moving to blade servers can experience as much as an 85% reduction in cabling for blade installations over conventional 1U or tower servers. With so much less cabling, IT administrators can spend less time managing the infrastructure and more time ensuring high availability

A blade server is sometimes referred to as a high-density server and is typically used in a clustering of servers that are dedicated to a single task, such as:

1. File sharing

2. Web page serving and caching

3. SSL encrypting of Web communication

4. The transcoding of Web page content for smaller displays

5. Streaming audio and video content

===Architecture===

A general blade server architecture is shown in the figure below. The hardware components of a blade server are the switch blade, chassis (with fans, temperature sensors, etc), and multiple compute blades. Some vendors offer, partner, or plan to partner with companies that provide application specific blades that provide traffic conditioning, protection, or network processing prior to the traffic reaching the compute blades. Often, these application specific blades may be functionally positioned between the switch blade and compute blades. However, these blades reside in a standard compute blade slot.
The outside world connects through the rear of the chassis to a switch card in the blade server. The switch card is provisioned to distribute packets to blades within the blade server. All these components are wrapped together with network management system software provided by the blade server vendor. The network management could be done through Message Passing which essentially makes blade servers an extension of message passing.

===Evolution===

The name blade server appeared when a card included the processor, memory, I/O and non-volatile program storage (flash memory or small hard disk(s)). This allowed a complete server, with its operating system and applications, to be packaged on a single card / board / blade. These blades could then operate independently within a common chassis, doing the work of multiple separate server boxes more efficiently. Less space consumption is the most obvious benefit of this packaging, but additional efficiency benefits have become clear in power, cooling, management, and networking due to the pooling or sharing of common infrastructure to supports the entire chassis, rather than providing each of these on a per server box basis.
Blade servers date back to 1970s. The evolution chronology as provided by Wikipedia (Article: Blade Servers) is given below:
Complete microcomputers were placed on cards and packaged in standard 19-inch racks in the 1970s soon after the introduction of 8-bit microprocessors. This architecture was used in the industrial process control industry as an alternative to minicomputer control systems. Programs were stored in EPROM on early models and were limited to a single function with a small realtime executive.

The VMEBus architecture (ca. 1981) defined a computer interface which included implementation of a board-level computer that was installed in a chassis backplane with multiple slots for pluggable boards that provide I/O, memory, or additional computing. The PCI Industrial Computer Manufacturers Group PICMG developed a chassis/blade structure for the then emerging Peripheral Component Interconnect bus PCI which is called CompactPCI. Common among these chassis based computers was the fact that the entire chassis was a single system. While a chassis might include multiple computing elements to provide the desired level of performance and redundancy, there was always one board in charge, one master board coordinating the operation of the entire system. PICMG expanded the CompactPCI specification with the use of standard Ethernet connectivity between boards across the backplane. The PICMG 2.16 CompactPCI Packet Switching Backplane specification was adopted in Sept 2001 (PICMG specifications). This provided the first open architecture for a multi-server chassis. PICMG followed with the larger and more feature rich AdvancedTCA specification targeting the telecom industry's need for a high availability and dense computing platform with extended product life (10+ years). While AdvancedTCA system and board pricing is typically higher than blade servers, AdvancedTCA suppliers claim that low operating expenses and total cost of ownership can make AdvancedTCA-based solutions a cost effective alternative for many building blocks of the next generation telecom network.

=== Future ===

Early versions of server blades will be primarily high-density, low-power devices with relatively low performance. This type of blade is suited for first-tier applications such as static Web servers, security, network services, and streaming media because the applications can be easily and inexpensively load balanced. The performance of an application depends on the aggregate performance of the servers rather than the performance of an individual server.
Higher performance, less dense blade designs will help drive blade usage into more mainstream applications in the corporate data center. These designs can offer the individual performance characteristics and features available in today's rack-dense servers along with the cost, deployment, serviceability, and density benefits of server blades. The blades will be well suited to high-performance Web servers, dedicated application servers, server-based or thin-client computing, and high-performance computing (HPC) clusters.

The introduction of server blades and associated technology like IB will usher in a new IT infrastructure. IT managers should start planning now for server blade installations by evaluating IP-based storage solutions, remote software provisioning and management solutions, scale-out architectures, and load-balancing technologies

== References ==

1. [http://www.hku.hk/cc/sp2/workshop/html/message_passing/message_passing.html#message1 Message Passing]

2. [http://www-unix.mcs.anl.gov/mpi/mpich2/#related MPI Implementation]

3. [http://www.dell.com/content/topics/global.aspx/power/en/ps1q02_blades?c=us&cs=555&l=en&s=biz Blade Servers]

4. [http://www.compactpci-systems.com/dl.php?pdf=/columns/software_corner/pdfs/3.03.pdf Blade Server Evolution]

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T00:37:02Z

Mvrao: /* History */

== Message Passing ==

When we have multiple processors, there needs to be a way to communicate between those processors. Message Passing forms a part of this communication architecture. There are other methods of communication like Shared Address Space and Data Parallel Processing, which along with Message Passing contribute to the communication abstraction. Communication abstraction is essentially a layer in between the application software and the communication hardware where the programmer uses available libraries to initiate communication between processors though programs.

===Message Passing Model===

Message Passing Model is defined as:

1. Set of Processes having only local memory

2. Processes communicate by sending and receiving messages

3. Transfer of data between processes requires cooperative operations to be performed by each process (a send operation must have a matching receive)

The message passing model has gained wide use in the field of parallel computing due to advantages that include:

1. Hardware match - The message passing model fits well on parallel supercomputers and clusters of workstations which are composed of separate processors connected by a communications network.

2. Functionality - Message passing offers a full set of functions for expressing parallel algorithms, providing the control not found in data-parallel model.

3. Performance - Message passing gives the programmer explicit control of data locality. This in turn enables effective management of memory and caches in CPUs.

===Latest Developments in Message Passing===

Although Message Passing Model as a whole has not changed over time, the Message Passing Interface (MPI) has undergone continuous change. MPI is a communications protocol used to program parallel computers. MPI is not sanctioned by any major standards body; nevertheless, it has become the de facto standard for communication among processes that model a parallel program running on a distributed memory system.

== Message Passing Interface (MPI) ==

It is a specification for message passing libraries, designed to be a standard for distributed memory, message passing and parallel computing. The goal of the Message Passing Interface simply stated is to provide a widely used standard for writing message-passing programs. The interface attempts to establish a practical, portable, efficient, and flexible standard for message passing.

===Advantages===

MPI is preferred over other implementations for several reasons like:

1. Standardization - MPI is the only message passing library which can be considered a standard. It is supported on virtually all High Performance Computing (HPC) platforms.

2. Portability – Modification of source code not required when the application is ported to a different platform that supports MPI.

3. Performance - vendor implementations should be able to exploit native hardware features to optimize performance.

4. Functionality (over 115 routines)

5. Availability - a variety of implementations are available, both vendor and public domain.

===MPI Implementations===

Some of the implementations of MPI include:

1. Classical Cluster and Supercomputer implementations

2. Python

3. OCaml

4. Java

5. Microsoft Windows

6. MATLAB

7. Hardware implementations

== Blade Servers ==

A blade server is a server chassis housing multiple thin, modular electronic circuit boards, known as server blades. Each blade is a server in its own right, often dedicated to a single application. The blades are literally servers on a card, containing processors, memory, integrated network controllers, an optional fiber channel host bus adaptor (HBA) and other input/output (IO) ports.

Blade servers allow more processing power in less rack space, simplifying cabling and reducing power consumption. According to a Search, WinSystems.com article on server technology, enterprises moving to blade servers can experience as much as an 85% reduction in cabling for blade installations over conventional 1U or tower servers. With so much less cabling, IT administrators can spend less time managing the infrastructure and more time ensuring high availability

A blade server is sometimes referred to as a high-density server and is typically used in a clustering of servers that are dedicated to a single task, such as:

1. File sharing

2. Web page serving and caching

3. SSL encrypting of Web communication

4. The transcoding of Web page content for smaller displays

5. Streaming audio and video content

===Architecture===

A general blade server architecture is shown in the figure below. The hardware components of a blade server are the switch blade, chassis (with fans, temperature sensors, etc), and multiple compute blades. Some vendors offer, partner, or plan to partner with companies that provide application specific blades that provide traffic conditioning, protection, or network processing prior to the traffic reaching the compute blades. Often, these application specific blades may be functionally positioned between the switch blade and compute blades. However, these blades reside in a standard compute blade slot.
The outside world connects through the rear of the chassis to a switch card in the blade server. The switch card is provisioned to distribute packets to blades within the blade server. All these components are wrapped together with network management system software provided by the blade server vendor. The network management could be done through Message Passing which essentially makes blade servers an extension of message passing.

===Evolution===

The name blade server appeared when a card included the processor, memory, I/O and non-volatile program storage (flash memory or small hard disk(s)). This allowed a complete server, with its operating system and applications, to be packaged on a single card / board / blade. These blades could then operate independently within a common chassis, doing the work of multiple separate server boxes more efficiently. Less space consumption is the most obvious benefit of this packaging, but additional efficiency benefits have become clear in power, cooling, management, and networking due to the pooling or sharing of common infrastructure to supports the entire chassis, rather than providing each of these on a per server box basis.
Blade servers date back to 1970s. The evolution chronology as provided by Wikipedia (Article: Blade Servers) is given below:
Complete microcomputers were placed on cards and packaged in standard 19-inch racks in the 1970s soon after the introduction of 8-bit microprocessors. This architecture was used in the industrial process control industry as an alternative to minicomputer control systems. Programs were stored in EPROM on early models and were limited to a single function with a small realtime executive.

The VMEBus architecture (ca. 1981) defined a computer interface which included implementation of a board-level computer that was installed in a chassis backplane with multiple slots for pluggable boards that provide I/O, memory, or additional computing. The PCI Industrial Computer Manufacturers Group PICMG developed a chassis/blade structure for the then emerging Peripheral Component Interconnect bus PCI which is called CompactPCI. Common among these chassis based computers was the fact that the entire chassis was a single system. While a chassis might include multiple computing elements to provide the desired level of performance and redundancy, there was always one board in charge, one master board coordinating the operation of the entire system. PICMG expanded the CompactPCI specification with the use of standard Ethernet connectivity between boards across the backplane. The PICMG 2.16 CompactPCI Packet Switching Backplane specification was adopted in Sept 2001 (PICMG specifications). This provided the first open architecture for a multi-server chassis. PICMG followed with the larger and more feature rich AdvancedTCA specification targeting the telecom industry's need for a high availability and dense computing platform with extended product life (10+ years). While AdvancedTCA system and board pricing is typically higher than blade servers, AdvancedTCA suppliers claim that low operating expenses and total cost of ownership can make AdvancedTCA-based solutions a cost effective alternative for many building blocks of the next generation telecom network.

=== Future ===

Early versions of server blades will be primarily high-density, low-power devices with relatively low performance. This type of blade is suited for first-tier applications such as static Web servers, security, network services, and streaming media because the applications can be easily and inexpensively load balanced. The performance of an application depends on the aggregate performance of the servers rather than the performance of an individual server.
Higher performance, less dense blade designs will help drive blade usage into more mainstream applications in the corporate data center. These designs can offer the individual performance characteristics and features available in today's rack-dense servers along with the cost, deployment, serviceability, and density benefits of server blades. The blades will be well suited to high-performance Web servers, dedicated application servers, server-based or thin-client computing, and high-performance computing (HPC) clusters.

The introduction of server blades and associated technology like IB will usher in a new IT infrastructure. IT managers should start planning now for server blade installations by evaluating IP-based storage solutions, remote software provisioning and management solutions, scale-out architectures, and load-balancing technologies

== References ==

1. [http://www.hku.hk/cc/sp2/workshop/html/message_passing/message_passing.html#message1 Message Passing]

2. [http://www-unix.mcs.anl.gov/mpi/mpich2/#related MPI Implementation]

3. [http://www.dell.com/content/topics/global.aspx/power/en/ps1q02_blades?c=us&cs=555&l=en&s=biz Blade Servers]

4. [http://www.compactpci-systems.com/dl.php?pdf=/columns/software_corner/pdfs/3.03.pdf Blade Server Evolution]

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-11T00:30:37Z

Mvrao: /* Message Passing Model */

== Message Passing ==

When we have multiple processors, there needs to be a way to communicate between those processors. Message Passing forms a part of this communication architecture. There are other methods of communication like Shared Address Space and Data Parallel Processing, which along with Message Passing contribute to the communication abstraction. Communication abstraction is essentially a layer in between the application software and the communication hardware where the programmer uses available libraries to initiate communication between processors though programs.

===Message Passing Model===

Message Passing Model is defined as:

1. Set of Processes having only local memory

2. Processes communicate by sending and receiving messages

3. Transfer of data between processes requires cooperative operations to be performed by each process (a send operation must have a matching receive)

The message passing model has gained wide use in the field of parallel computing due to advantages that include:

1. Hardware match - The message passing model fits well on parallel supercomputers and clusters of workstations which are composed of separate processors connected by a communications network.

2. Functionality - Message passing offers a full set of functions for expressing parallel algorithms, providing the control not found in data-parallel model.

3. Performance - Message passing gives the programmer explicit control of data locality. This in turn enables effective management of memory and caches in CPUs.

===Latest Developments in Message Passing===

Although Message Passing Model as a whole has not changed over time, the Message Passing Interface (MPI) has undergone continuous change. MPI is a communications protocol used to program parallel computers. MPI is not sanctioned by any major standards body; nevertheless, it has become the de facto standard for communication among processes that model a parallel program running on a distributed memory system.

== Message Passing Interface (MPI) ==

It is a specification for message passing libraries, designed to be a standard for distributed memory, message passing and parallel computing. The goal of the Message Passing Interface simply stated is to provide a widely used standard for writing message-passing programs. The interface attempts to establish a practical, portable, efficient, and flexible standard for message passing.

===History===

MPI resulted from the efforts of numerous individuals and groups over the course of 2 years, dated back in 1980’s. Given below is a chronology of developments in MPI according to documentation provided by the Maui High Performance Computing Center. (www.mhpcc.edu)

1. 1980s - early 1990s: Distributed memory, parallel computing develops, as do a number of incompatible software tools for writing such programs - usually with tradeoffs between portability, performance, functionality and price. Recognition of the need for a standard arose.

2. April, 1992: Workshop on Standards for Message Passing in a Distributed Memory Environment, sponsored by the Center for Research on Parallel Computing, Williamsburg, Virginia. The basic features essential to a standard message passing interface were discussed, and a working group established to continue the standardization process. Preliminary draft proposal developed subsequently.

3. November 1992: - Working group meets in Minneapolis. MPI draft proposal (MPI1) from ORNL presented. Group adopts procedures and organization to form the MPI Forum. MPIF eventually comprised of about 175 individuals from 40 organizations including parallel computer vendors, software writers, academia and application scientists.

4. November 1993: Supercomputing 93 conference - draft MPI standard presented.

5. Final version of draft released in May, 1994 - available on the WWW at: http://www.mcs.anl.gov/Projects/mpi/standard.html

===Advantages===

MPI is preferred over other implementations for several reasons like:

1. Standardization - MPI is the only message passing library which can be considered a standard. It is supported on virtually all High Performance Computing (HPC) platforms.

2. Portability – Modification of source code not required when the application is ported to a different platform that supports MPI.

3. Performance - vendor implementations should be able to exploit native hardware features to optimize performance.

4. Functionality (over 115 routines)

5. Availability - a variety of implementations are available, both vendor and public domain.

===MPI Implementations===

Some of the implementations of MPI include:

1. Classical Cluster and Supercomputer implementations

2. Python

3. OCaml

4. Java

5. Microsoft Windows

6. MATLAB

7. Hardware implementations

== Blade Servers ==

A blade server is a server chassis housing multiple thin, modular electronic circuit boards, known as server blades. Each blade is a server in its own right, often dedicated to a single application. The blades are literally servers on a card, containing processors, memory, integrated network controllers, an optional fiber channel host bus adaptor (HBA) and other input/output (IO) ports.

Blade servers allow more processing power in less rack space, simplifying cabling and reducing power consumption. According to a Search, WinSystems.com article on server technology, enterprises moving to blade servers can experience as much as an 85% reduction in cabling for blade installations over conventional 1U or tower servers. With so much less cabling, IT administrators can spend less time managing the infrastructure and more time ensuring high availability

A blade server is sometimes referred to as a high-density server and is typically used in a clustering of servers that are dedicated to a single task, such as:

1. File sharing

2. Web page serving and caching

3. SSL encrypting of Web communication

4. The transcoding of Web page content for smaller displays

5. Streaming audio and video content

===Architecture===

A general blade server architecture is shown in the figure below. The hardware components of a blade server are the switch blade, chassis (with fans, temperature sensors, etc), and multiple compute blades. Some vendors offer, partner, or plan to partner with companies that provide application specific blades that provide traffic conditioning, protection, or network processing prior to the traffic reaching the compute blades. Often, these application specific blades may be functionally positioned between the switch blade and compute blades. However, these blades reside in a standard compute blade slot.
The outside world connects through the rear of the chassis to a switch card in the blade server. The switch card is provisioned to distribute packets to blades within the blade server. All these components are wrapped together with network management system software provided by the blade server vendor. The network management could be done through Message Passing which essentially makes blade servers an extension of message passing.

===Evolution===

The name blade server appeared when a card included the processor, memory, I/O and non-volatile program storage (flash memory or small hard disk(s)). This allowed a complete server, with its operating system and applications, to be packaged on a single card / board / blade. These blades could then operate independently within a common chassis, doing the work of multiple separate server boxes more efficiently. Less space consumption is the most obvious benefit of this packaging, but additional efficiency benefits have become clear in power, cooling, management, and networking due to the pooling or sharing of common infrastructure to supports the entire chassis, rather than providing each of these on a per server box basis.
Blade servers date back to 1970s. The evolution chronology as provided by Wikipedia (Article: Blade Servers) is given below:
Complete microcomputers were placed on cards and packaged in standard 19-inch racks in the 1970s soon after the introduction of 8-bit microprocessors. This architecture was used in the industrial process control industry as an alternative to minicomputer control systems. Programs were stored in EPROM on early models and were limited to a single function with a small realtime executive.

The VMEBus architecture (ca. 1981) defined a computer interface which included implementation of a board-level computer that was installed in a chassis backplane with multiple slots for pluggable boards that provide I/O, memory, or additional computing. The PCI Industrial Computer Manufacturers Group PICMG developed a chassis/blade structure for the then emerging Peripheral Component Interconnect bus PCI which is called CompactPCI. Common among these chassis based computers was the fact that the entire chassis was a single system. While a chassis might include multiple computing elements to provide the desired level of performance and redundancy, there was always one board in charge, one master board coordinating the operation of the entire system. PICMG expanded the CompactPCI specification with the use of standard Ethernet connectivity between boards across the backplane. The PICMG 2.16 CompactPCI Packet Switching Backplane specification was adopted in Sept 2001 (PICMG specifications). This provided the first open architecture for a multi-server chassis. PICMG followed with the larger and more feature rich AdvancedTCA specification targeting the telecom industry's need for a high availability and dense computing platform with extended product life (10+ years). While AdvancedTCA system and board pricing is typically higher than blade servers, AdvancedTCA suppliers claim that low operating expenses and total cost of ownership can make AdvancedTCA-based solutions a cost effective alternative for many building blocks of the next generation telecom network.

=== Future ===

Early versions of server blades will be primarily high-density, low-power devices with relatively low performance. This type of blade is suited for first-tier applications such as static Web servers, security, network services, and streaming media because the applications can be easily and inexpensively load balanced. The performance of an application depends on the aggregate performance of the servers rather than the performance of an individual server.
Higher performance, less dense blade designs will help drive blade usage into more mainstream applications in the corporate data center. These designs can offer the individual performance characteristics and features available in today's rack-dense servers along with the cost, deployment, serviceability, and density benefits of server blades. The blades will be well suited to high-performance Web servers, dedicated application servers, server-based or thin-client computing, and high-performance computing (HPC) clusters.

The introduction of server blades and associated technology like IB will usher in a new IT infrastructure. IT managers should start planning now for server blade installations by evaluating IP-based storage solutions, remote software provisioning and management solutions, scale-out architectures, and load-balancing technologies

== References ==

1. [http://www.hku.hk/cc/sp2/workshop/html/message_passing/message_passing.html#message1 Message Passing]

2. [http://www-unix.mcs.anl.gov/mpi/mpich2/#related MPI Implementation]

3. [http://www.dell.com/content/topics/global.aspx/power/en/ps1q02_blades?c=us&cs=555&l=en&s=biz Blade Servers]

4. [http://www.compactpci-systems.com/dl.php?pdf=/columns/software_corner/pdfs/3.03.pdf Blade Server Evolution]

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-05T22:42:49Z

Mvrao: /* Message Passing and Blade Servers */

== Message Passing ==

When we have multiple processors, there needs to be a way to communicate between those processors. Message Passing forms a part of this communication architecture. There are other methods of communication like Shared Address Space and Data Parallel Processing, which along with Message Passing contribute to the communication abstraction. Communication abstraction is essentially a layer in between the application software and the communication hardware where the programmer uses available libraries to initiate communication between processors though programs.

===Message Passing Model===

Message Passing Model is defined as:

1. Set of Processes having only local memory

2. Processes communicate by sending and receiving messages

3. Transfer of data between processes requires cooperative operations to be performed by each process (a send operation must have a matching receive)

The message passing model has gained wide use in the field of parallel computing due to advantages that include:

1. Hardware match - The message passing model fits well on parallel supercomputers and clusters of workstations which are composed of separate processors connected by a communications network.

2. Functionality - Message passing offers a full set of functions for expressing parallel algorithms, providing the control not found in data-parallel and compiler-based models.

3. Performance - Effective use of modern CPUs requires management of their memory hierarchy, especially their caches. Message passing achieves this by giving programmer explicit control of data locality.

The principle drawback of message passing is the responsibility it places on the programmer. The programmer must explicitly implement a data distribution scheme and all interprocess communication and synchronization. In so doing, it is the programmer's responsibility to resolve data dependencies and avoid deadlock and race conditions.

===Latest Developments in Message Passing===

Although Message Passing Model as a whole has not changed over time, the Message Passing Interface (MPI) has undergone continuous change. MPI is a communications protocol used to program parallel computers. MPI is not sanctioned by any major standards body; nevertheless, it has become the de facto standard for communication among processes that model a parallel program running on a distributed memory system.

== Message Passing Interface (MPI) ==

It is a specification for message passing libraries, designed to be a standard for distributed memory, message passing and parallel computing. The goal of the Message Passing Interface simply stated is to provide a widely used standard for writing message-passing programs. The interface attempts to establish a practical, portable, efficient, and flexible standard for message passing.

===History===

MPI resulted from the efforts of numerous individuals and groups over the course of 2 years, dated back in 1980’s. Given below is a chronology of developments in MPI according to documentation provided by the Maui High Performance Computing Center. (www.mhpcc.edu)

1. 1980s - early 1990s: Distributed memory, parallel computing develops, as do a number of incompatible software tools for writing such programs - usually with tradeoffs between portability, performance, functionality and price. Recognition of the need for a standard arose.

2. April, 1992: Workshop on Standards for Message Passing in a Distributed Memory Environment, sponsored by the Center for Research on Parallel Computing, Williamsburg, Virginia. The basic features essential to a standard message passing interface were discussed, and a working group established to continue the standardization process. Preliminary draft proposal developed subsequently.

3. November 1992: - Working group meets in Minneapolis. MPI draft proposal (MPI1) from ORNL presented. Group adopts procedures and organization to form the MPI Forum. MPIF eventually comprised of about 175 individuals from 40 organizations including parallel computer vendors, software writers, academia and application scientists.

4. November 1993: Supercomputing 93 conference - draft MPI standard presented.

5. Final version of draft released in May, 1994 - available on the WWW at: http://www.mcs.anl.gov/Projects/mpi/standard.html

===Advantages===

MPI is preferred over other implementations for several reasons like:

1. Standardization - MPI is the only message passing library which can be considered a standard. It is supported on virtually all High Performance Computing (HPC) platforms.

2. Portability – Modification of source code not required when the application is ported to a different platform that supports MPI.

3. Performance - vendor implementations should be able to exploit native hardware features to optimize performance.

4. Functionality (over 115 routines)

5. Availability - a variety of implementations are available, both vendor and public domain.

===MPI Implementations===

Some of the implementations of MPI include:

1. Classical Cluster and Supercomputer implementations

2. Python

3. OCaml

4. Java

5. Microsoft Windows

6. MATLAB

7. Hardware implementations

== Blade Servers ==

A blade server is a server chassis housing multiple thin, modular electronic circuit boards, known as server blades. Each blade is a server in its own right, often dedicated to a single application. The blades are literally servers on a card, containing processors, memory, integrated network controllers, an optional fiber channel host bus adaptor (HBA) and other input/output (IO) ports.

Blade servers allow more processing power in less rack space, simplifying cabling and reducing power consumption. According to a Search, WinSystems.com article on server technology, enterprises moving to blade servers can experience as much as an 85% reduction in cabling for blade installations over conventional 1U or tower servers. With so much less cabling, IT administrators can spend less time managing the infrastructure and more time ensuring high availability

A blade server is sometimes referred to as a high-density server and is typically used in a clustering of servers that are dedicated to a single task, such as:

1. File sharing

2. Web page serving and caching

3. SSL encrypting of Web communication

4. The transcoding of Web page content for smaller displays

5. Streaming audio and video content

===Architecture===

A general blade server architecture is shown in the figure below. The hardware components of a blade server are the switch blade, chassis (with fans, temperature sensors, etc), and multiple compute blades. Some vendors offer, partner, or plan to partner with companies that provide application specific blades that provide traffic conditioning, protection, or network processing prior to the traffic reaching the compute blades. Often, these application specific blades may be functionally positioned between the switch blade and compute blades. However, these blades reside in a standard compute blade slot.
The outside world connects through the rear of the chassis to a switch card in the blade server. The switch card is provisioned to distribute packets to blades within the blade server. All these components are wrapped together with network management system software provided by the blade server vendor. The network management could be done through Message Passing which essentially makes blade servers an extension of message passing.

===Evolution===

The name blade server appeared when a card included the processor, memory, I/O and non-volatile program storage (flash memory or small hard disk(s)). This allowed a complete server, with its operating system and applications, to be packaged on a single card / board / blade. These blades could then operate independently within a common chassis, doing the work of multiple separate server boxes more efficiently. Less space consumption is the most obvious benefit of this packaging, but additional efficiency benefits have become clear in power, cooling, management, and networking due to the pooling or sharing of common infrastructure to supports the entire chassis, rather than providing each of these on a per server box basis.
Blade servers date back to 1970s. The evolution chronology as provided by Wikipedia (Article: Blade Servers) is given below:
Complete microcomputers were placed on cards and packaged in standard 19-inch racks in the 1970s soon after the introduction of 8-bit microprocessors. This architecture was used in the industrial process control industry as an alternative to minicomputer control systems. Programs were stored in EPROM on early models and were limited to a single function with a small realtime executive.

The VMEBus architecture (ca. 1981) defined a computer interface which included implementation of a board-level computer that was installed in a chassis backplane with multiple slots for pluggable boards that provide I/O, memory, or additional computing. The PCI Industrial Computer Manufacturers Group PICMG developed a chassis/blade structure for the then emerging Peripheral Component Interconnect bus PCI which is called CompactPCI. Common among these chassis based computers was the fact that the entire chassis was a single system. While a chassis might include multiple computing elements to provide the desired level of performance and redundancy, there was always one board in charge, one master board coordinating the operation of the entire system. PICMG expanded the CompactPCI specification with the use of standard Ethernet connectivity between boards across the backplane. The PICMG 2.16 CompactPCI Packet Switching Backplane specification was adopted in Sept 2001 (PICMG specifications). This provided the first open architecture for a multi-server chassis. PICMG followed with the larger and more feature rich AdvancedTCA specification targeting the telecom industry's need for a high availability and dense computing platform with extended product life (10+ years). While AdvancedTCA system and board pricing is typically higher than blade servers, AdvancedTCA suppliers claim that low operating expenses and total cost of ownership can make AdvancedTCA-based solutions a cost effective alternative for many building blocks of the next generation telecom network.

=== Future ===

Early versions of server blades will be primarily high-density, low-power devices with relatively low performance. This type of blade is suited for first-tier applications such as static Web servers, security, network services, and streaming media because the applications can be easily and inexpensively load balanced. The performance of an application depends on the aggregate performance of the servers rather than the performance of an individual server.
Higher performance, less dense blade designs will help drive blade usage into more mainstream applications in the corporate data center. These designs can offer the individual performance characteristics and features available in today's rack-dense servers along with the cost, deployment, serviceability, and density benefits of server blades. The blades will be well suited to high-performance Web servers, dedicated application servers, server-based or thin-client computing, and high-performance computing (HPC) clusters.

The introduction of server blades and associated technology like IB will usher in a new IT infrastructure. IT managers should start planning now for server blade installations by evaluating IP-based storage solutions, remote software provisioning and management solutions, scale-out architectures, and load-balancing technologies

== References ==

1. [http://www.hku.hk/cc/sp2/workshop/html/message_passing/message_passing.html#message1 Message Passing]

2. [http://www-unix.mcs.anl.gov/mpi/mpich2/#related MPI Implementation]

3. [http://www.dell.com/content/topics/global.aspx/power/en/ps1q02_blades?c=us&cs=555&l=en&s=biz Blade Servers]

4. [http://www.compactpci-systems.com/dl.php?pdf=/columns/software_corner/pdfs/3.03.pdf Blade Server Evolution]

CSC/ECE 506 Fall 2007/wiki1 8 s5

2007-09-05T18:10:33Z

Mvrao: Message Passing and Blade Servers

== Message Passing and Blade Servers ==

'''Introduction:'''

When we have multiple processors, there needs to be a way to communicate between those processors. Message Passing forms a part of this communication architecture. There are other methods of communication like Shared Address Space and Data Parallel Processing, which along with Message Passing contribute to the communication abstraction. Communication abstraction is essentially a layer in between the application software and the communication hardware where the programmer uses available libraries to initiate communication between processors though programs.

'''Message Passing:'''

Message Passing Model is defined as:

1. Set of Processes having only local memory

2. Processes communicate by sending and receiving messages

3. Transfer of data between processes requires cooperative operations to be performed by each process (a send operation must have a matching receive)

The message passing model has gained wide use in the field of parallel computing due to advantages that include:

1. Hardware match - The message passing model fits well on parallel supercomputers and clusters of workstations which are composed of separate processors connected by a communications network.

2. Functionality - Message passing offers a full set of functions for expressing parallel algorithms, providing the control not found in data-parallel and compiler-based models.

3. Performance - Effective use of modern CPUs requires management of their memory hierarchy, especially their caches. Message passing achieves this by giving programmer explicit control of data locality.

The principle drawback of message passing is the responsibility it places on the programmer. The programmer must explicitly implement a data distribution scheme and all interprocess communication and synchronization. In so doing, it is the programmer's responsibility to resolve data dependencies and avoid deadlock and race conditions.

Latest Developments in Message Passing:

Although Message Passing Model as a whole has not changed over time, the Message Passing Interface (MPI) has undergone continuous change. MPI is a communications protocol used to program parallel computers. MPI is not sanctioned by any major standards body; nevertheless, it has become the de facto standard for communication among processes that model a parallel program running on a distributed memory system.

'''Message Passing Interface (MPI)'''

It is a specification for message passing libraries, designed to be a standard for distributed memory, message passing and parallel computing. The goal of the Message Passing Interface simply stated is to provide a widely used standard for writing message-passing programs. The interface attempts to establish a practical, portable, efficient, and flexible standard for message passing.

History:

MPI resulted from the efforts of numerous individuals and groups over the course of 2 years, dated back in 1980’s. Given below is a chronology of developments in MPI according to documentation provided by the Maui High Performance Computing Center. (www.mhpcc.edu)

1. 1980s - early 1990s: Distributed memory, parallel computing develops, as do a number of incompatible software tools for writing such programs - usually with tradeoffs between portability, performance, functionality and price. Recognition of the need for a standard arose.

2. April, 1992: Workshop on Standards for Message Passing in a Distributed Memory Environment, sponsored by the Center for Research on Parallel Computing, Williamsburg, Virginia. The basic features essential to a standard message passing interface were discussed, and a working group established to continue the standardization process. Preliminary draft proposal developed subsequently.

3. November 1992: - Working group meets in Minneapolis. MPI draft proposal (MPI1) from ORNL presented. Group adopts procedures and organization to form the MPI Forum. MPIF eventually comprised of about 175 individuals from 40 organizations including parallel computer vendors, software writers, academia and application scientists.

4. November 1993: Supercomputing 93 conference - draft MPI standard presented.

5. Final version of draft released in May, 1994 - available on the WWW at: http://www.mcs.anl.gov/Projects/mpi/standard.html

Advantages:

MPI is preferred over other implementations for several reasons like:

1. Standardization - MPI is the only message passing library which can be considered a standard. It is supported on virtually all High Performance Computing (HPC) platforms.

2. Portability – Modification of source code not required when the application is ported to a different platform that supports MPI.

3. Performance - vendor implementations should be able to exploit native hardware features to optimize performance.

4. Functionality (over 115 routines)

5. Availability - a variety of implementations are available, both vendor and public domain.

MPI Implementations:

Some of the implementations of MPI include:

1. Classical Cluster and Supercomputer implementations

2. Python

3. OCaml

4. Java

5. Microsoft Windows

6. MATLAB

7. Hardware implementations

'''Blade Servers'''

A blade server is a server chassis housing multiple thin, modular electronic circuit boards, known as server blades. Each blade is a server in its own right, often dedicated to a single application. The blades are literally servers on a card, containing processors, memory, integrated network controllers, an optional fiber channel host bus adaptor (HBA) and other input/output (IO) ports.

Blade servers allow more processing power in less rack space, simplifying cabling and reducing power consumption. According to a Search, WinSystems.com article on server technology, enterprises moving to blade servers can experience as much as an 85% reduction in cabling for blade installations over conventional 1U or tower servers. With so much less cabling, IT administrators can spend less time managing the infrastructure and more time ensuring high availability

A blade server is sometimes referred to as a high-density server and is typically used in a clustering of servers that are dedicated to a single task, such as:

1. File sharing

2. Web page serving and caching

3. SSL encrypting of Web communication

4. The transcoding of Web page content for smaller displays

5. Streaming audio and video content

Architecture:

A general blade server architecture is shown in the figure below. The hardware components of a blade server are the switch blade, chassis (with fans, temperature sensors, etc), and multiple compute blades. Some vendors offer, partner, or plan to partner with companies that provide application specific blades that provide traffic conditioning, protection, or network processing prior to the traffic reaching the compute blades. Often, these application specific blades may be functionally positioned between the switch blade and compute blades. However, these blades reside in a standard compute blade slot.
The outside world connects through the rear of the chassis to a switch card in the blade server. The switch card is provisioned to distribute packets to blades within the blade server. All these components are wrapped together with network management system software provided by the blade server vendor. The network management could be done through Message Passing which essentially makes blade servers an extension of message passing.

[[Image:D:\bladeserver.jpg]]

Fig 1: Blade Server Architecture (Courtesy: Blade Servers: Evolution and revolution by Curtis A Schwaderer)

Evolution:

The name blade server appeared when a card included the processor, memory, I/O and non-volatile program storage (flash memory or small hard disk(s)). This allowed a complete server, with its operating system and applications, to be packaged on a single card / board / blade. These blades could then operate independently within a common chassis, doing the work of multiple separate server boxes more efficiently. Less space consumption is the most obvious benefit of this packaging, but additional efficiency benefits have become clear in power, cooling, management, and networking due to the pooling or sharing of common infrastructure to supports the entire chassis, rather than providing each of these on a per server box basis.
Blade servers date back to 1970s. The evolution chronology as provided by Wikipedia (Article: Blade Servers) is given below:
Complete microcomputers were placed on cards and packaged in standard 19-inch racks in the 1970s soon after the introduction of 8-bit microprocessors. This architecture was used in the industrial process control industry as an alternative to minicomputer control systems. Programs were stored in EPROM on early models and were limited to a single function with a small realtime executive.

The VMEBus architecture (ca. 1981) defined a computer interface which included implementation of a board-level computer that was installed in a chassis backplane with multiple slots for pluggable boards that provide I/O, memory, or additional computing. The PCI Industrial Computer Manufacturers Group PICMG developed a chassis/blade structure for the then emerging Peripheral Component Interconnect bus PCI which is called CompactPCI. Common among these chassis based computers was the fact that the entire chassis was a single system. While a chassis might include multiple computing elements to provide the desired level of performance and redundancy, there was always one board in charge, one master board coordinating the operation of the entire system. PICMG expanded the CompactPCI specification with the use of standard Ethernet connectivity between boards across the backplane. The PICMG 2.16 CompactPCI Packet Switching Backplane specification was adopted in Sept 2001 (PICMG specifications). This provided the first open architecture for a multi-server chassis. PICMG followed with the larger and more feature rich AdvancedTCA specification targeting the telecom industry's need for a high availability and dense computing platform with extended product life (10+ years). While AdvancedTCA system and board pricing is typically higher than blade servers, AdvancedTCA suppliers claim that low operating expenses and total cost of ownership can make AdvancedTCA-based solutions a cost effective alternative for many building blocks of the next generation telecom network.

Future:

Early versions of server blades will be primarily high-density, low-power devices with relatively low performance. This type of blade is suited for first-tier applications such as static Web servers, security, network services, and streaming media because the applications can be easily and inexpensively load balanced. The performance of an application depends on the aggregate performance of the servers rather than the performance of an individual server.
Higher performance, less dense blade designs will help drive blade usage into more mainstream applications in the corporate data center. These designs can offer the individual performance characteristics and features available in today's rack-dense servers along with the cost, deployment, serviceability, and density benefits of server blades. The blades will be well suited to high-performance Web servers, dedicated application servers, server-based or thin-client computing, and high-performance computing (HPC) clusters.

The introduction of server blades and associated technology like IB will usher in a new IT infrastructure. IT managers should start planning now for server blade installations by evaluating IP-based storage solutions, remote software provisioning and management solutions, scale-out architectures, and load-balancing technologies