CSC 456 Spring 2012/11b AB

From PG_Wiki
Jump to: navigation, search


Large-Scale Multiprocessors

With modern technology, large-scale multiprocessors (LSMs) have become more prevalent. There has been considerable research into networking topologies for connecting the processors, and several methods have been conceived to ensure coherence. Additionally, there are numerous manufacturers who make the materials necessary to build LSMs. In this article, we show examples of each of these.


In order to build a LSM, you will need to choose the right processors, as well as the most appropriate cabinet(s) to place them in. There are several different manufacturers of processors and cabinets that can be used in LSM configurations. For example, Fujitsu's K computer (the number one ranked supercomputer on TOP500's November 2011 list) uses a configuration of 88,128 SPARC64 VIIIfx processors. This means it has a total of 705,024 cores at its use.[1] Additional examples of processors used in LSMs can be found in Table 1.

alt text
Fujitsu's K Computer[2]
Table 1: Processor Manufacturers
Manufacturer Processor Year Cores Clock Rate Architecture
Fujitsu SPARC64 VIIIfx[3] 2009 8 2.0 GHz SPARC
Intel Xeon 7500[4] 2010 8 1.733-2.667 GHz Nehalem
IBM POWER7[5] 2010 8 2.4-4.25 GHz Power ISA v.2.06
AMD Opteron 6100[6] 2010 12 1.7-2.4 GHz Direct Connect 2.0

Like processors, different manufacturers offer varying cabinet/server types, such as IBM's BladeCenter HT. This particular model uses their CoolBlue technology, a set of tools that allows the user to have greater control over cooling and power use. There are also some standard cabinet frames, such as 19-inch racks, which get their name from the 19-inch panels used in their design. Typically, these racks allow for easy processor/server installation and removal. [7]

Table 2: Cabinet Manufacturers
Manufacturer Cabinet Blade Count
SuperMicro MP Superserver 8064B-TRLF[8] 4
HP Integrity Superdome 2[9] 32
IBM BladeCenter HT[10] 12


Network Topology

There are many different ways to connect the network of processors. Each network type has different properties and values related to their diameter, bisection bandwidth, and degree. The diameter of a network is the longest number of network hops between any pair of nodes. Bisection bandwidth refers to the minimum number of links that need to be cut to divide the network in half. The degree of a network refers to the number of in/out links on each node. The following figure displays some examples.

An example of possible network structures.[11]

The following table gives some more detail on the different characteristics of some network types. "p" is the number of nodes, "d" is dimensions, and "k" is the number of nodes in each dimension.

Table 3: Network Properties
Topology Diameter Bandwidth Degree Example(s)
Ring p/2 2 2 KSR-1, NUMA-chine [12]
k-ary d Mesh 2(sqrt(p) - 1) sqrt(p) 4 Intel Paragon, Cray T3D [13]
Butterfly log_2(p) p/2 4 BBN Butterfly[14]
k-ary Fat Tree 2 x log_k(p) p/2 k+1 Xtreme-X[15]
Hypercube log_2(p) p/2 log_2(p) nCUBE 1[16]


For LSMs that use a Distributed Shared Memory (DSM) architecture, cache coherence is an important issue. In 1990, researchers at the Massachusetts Institute of Technology showed that it was possible to build to build a coherent LSM using a directory-based approach with the Alewife multiprocessor.[17] A modern example is the Pittsburgh Supercomputing Center's Blacklight, a supercomputer with hardware-enabled shared coherent memory.[18]

On the other hand, some LSMs use distributed memory systems, meaning that each of the processors has its own private memory, making cache coherency a non-issue. Fujitsu's K computer is an example of such a system.[1]

Another example of a memory design used by LSMs is Non Uniform Memory Access (NUMA). NUMA has a coherent version of its system, called cache coherent NUMA (ccNUMA), where data and memory is accessed globally.[19] The 2008 IBM Roadrunner supercomputer, which has 6480 Opteron processors and 12960 IBM Cell processors, uses ccNUMA.[20]


  1. 1.0 1.1 K Computer
  2. K Computer Image
  3. Fujitsu SPARC64 VIIIfx Processor
  4. Intel Xeon Processor
  5. IBM Power 7 Processor
  6. AMD Opteron Processor
  7. 19 inch Rack
  8. Supermicro Chassis
  9. HP Chassis
  10. IBM Chassis
  11. Network Topologies
  12. Ring Network Example
  13. Mesh Network Example
  14. BBN Butterfly Supercomputer
  15. Xtreme-X
  16. nCUBE 1
  17. Cache Coherence Protocols for Large Scale Multiprocessors
  18. Blacklight Multiprocessor
  19. ccNuma Machines
  20. Supercomputer Architecture
Personal tools