podc00_architecture.ppt 228KB Jun 23 2011 12:32:04 PM
How Computer Architecture Trends
May Affect Future Distributed Systems
Mark D. Hill
Computer Sciences Department
University of Wisconsin--Madison
http://www.cs.wisc.edu/~markhill
PODC ‘00 Invited Talk
(C) 2000 Mark D. Hill
University of Wisconsin-Madison
Three Questions
• What is a System Area Network (SAN)
and how will it affect clusters?
– E.g., InfiniBand
• How fat will multiprocessor servers be
and how to we build larger ones?
– E.g. Wisconsin Multifacet’s Multicast & Timestamp Snooping
• Future of multiprocessor servers & clusters?
– A merging of both?
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Outline
• Motivation
• System Area Networks
• Designing Multiprocessor Servers
• Server & Cluster Trends
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Technology Push: Moore’s Law
• What do following intervals have in common?
– Prehistory to 2000
– 2001 to 2002
• Answer: Equal progress in absolute processor speed
(and more doubling 2003-4, 2005-6, etc.)
– Consider salary doubling
• Corollary: Cost halves every two years
– Jim Gray: In a decade you can buy a computer
for less than its sales tax today
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Application Pull
• Should use computers in currently wasteful ways
– Already computers in electric razors & greeting cards
• New business models
– B2C, B2B, C2B, C2C
– Mass customization
• More proactive (beyond interactive) [Tennenhouse]
–
–
–
–
–
Today: P2C where P==Person & C==Computer
More C2P: mattress adjusts to save your back
More C2C: Agents surf the web for optimal deal
More sensors (physical/logic worlds coupled)
More hidden computers (c.f., electric motors)
• Furthermore, I am wrong
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
The Internet Iceberg
• Internet Components
–
–
–
–
Clients -- mobile, wireless
“On Ramp” -- LANs/DSL/Cable Modems
WAN Backbone -- IPv6, massive BW
and ...
• SERVICES
–
–
–
–
Scale Storage
Scale Bandwidth
Scale Computation
High Availability
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Outline
• Motivation
• System Area Networks
–
–
–
–
What is a SAN?
InfiniBand
Virtualizing I/O with Queue Pairs
Predictions
• Designing Multiprocessor Servers
• Server & Cluster Trends
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Regarding Storage/Bandwidth
• Currently resides on I/O Bus (PCI)
– HW & SW protocol stacks
– Must add hosts to add storage/bandwidth
proc
proc
memory interconnect
memory
bridge
i/o bus
i/o slot 0
(C) 2000 Mark D. Hill
i/o slot n-1
PODC00: Computer Architecture Trends
Want System Area Network (SAN)
• SAN vs. Local Area Nework (LAN)
–
–
–
–
–
Higher bandwidth (10 Gbps)
Lower latency (few microseconds or less)
More limited size
Other (e.g., single administrative domain, short distance)
Examples: Tandem Servernet & Myricom Myrinet
• Emerging Standard: InfiniBand
– www.inifinibandTA.org w/ spec 1.0 Summer 2000
– Compaq, Dell, HP, IBM, Intel, Microsoft, Sun, & others
– 2.5 Gbits/s times 1, 4, or 12 wires
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
InfiniBand Model (from website)
proc
proc
memory interconnect
memory
Other
networks
router
X
C
A
HCA (host channel adapter)
link
switch
T
C
A
target
(disks)
Other switches, hosts, targets, etc.
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Inifiniband Advantages
• Storage/Network made orthogonal from Computation
• Reduce “hardware” stack -- no i/o bridge
• Reduce “software” stack; hardware support for
–
–
–
–
–
Connected Reliable
Connected Unreliable
Datagram
Reliable Datagram
Raw Datagram
• Can eliminate system call for SAN use (next slide)
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Virtualizing InfiniBand
• I/O traditionally virtualized with system call
– System enforces isolation
– System permits authorized sharing
• Memory virtualized
– System trap/call for setup
– Virtual memory hardware for common-case translation
• Infiniband exploits “queue pairs” (QPs) in memory
– C.f., Intel Virtual Interface Architecture (VIA)
[IEEE Micro, Mar/Apr ‘98]
– Users issue sends, receives, & remote DMA reads/writes
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Queue Pair
proc
• QP setup system call
Main
Memory
dma-W4
dma-R3
send2 receive1
send1 receive2
– Connect with process
– Connect with remote QP
(not shown here)
• QP placed in “pinned”
virtual memory
• User directly access QP
HCA
(C) 2000 Mark D. Hill
– E.g., sends, receives &
remote DMA reads/writes
PODC00: Computer Architecture Trends
InfiniBand, cont.
• Roadmap
– NGIO/FIO merger in ‘99
– Spec in ‘00
– Products in ‘03-’10
• My Assessment
–
–
–
–
PCI needs successor
InfiniBand has the necessary features (but also many others)
InifiniBand has considerable industry buy-in (but it is recent)
Gigabit Ethernet will be only competitor
• Good name with backing from Cisco et al.
• But TCP/IP is a killer
– Infiniband for storage will be key
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
InfiniBand Research Issues
• Software Wide Open
– Industry will do local optimization
(e.g., still have device driver virtualized with system calls)
– But what is the “right” way to do software?
– Is there a theoretical model for this software?
• Other SAN Issues
–
–
–
–
A theoretical model of a service-providers site?
How to trade performance and availability?
Utility of broadcast or multicast support?
Obtaining quasi-real-time performance?
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Outline
• Motivation
• System Area Networks
• Designing Multiprocessor Servers
–
–
–
–
How Fat?
Coherence for Servers
E.g., Multicast Snooping
E.g., Timestamp Snooping
• Server & Cluster Trends
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
How Fat Should Servers Be?
• Use
– PCs -- cheap but small
– Workgroup servers -- medium cost; medium size
– Large servers -- premium cost & size
• One answer: “yes”
PCs w/
“soft” state
(C) 2000 Mark D. Hill
Servers running
databases for
“hard” state
PODC00: Computer Architecture Trends
How Do We Build the Big Servers?
• (Industry knows how to build the small ones)
• A key problem is the memory system
– Memory Wall: E.g., 100ns memory access =
400 instruction opportunities for 4-way 1GHz processor
• Use per-processor caches to reduce
– Effective Latency
– Effective Bandwidth Used
• But cache coherence problem ...
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
“4”
r0
May Affect Future Distributed Systems
Mark D. Hill
Computer Sciences Department
University of Wisconsin--Madison
http://www.cs.wisc.edu/~markhill
PODC ‘00 Invited Talk
(C) 2000 Mark D. Hill
University of Wisconsin-Madison
Three Questions
• What is a System Area Network (SAN)
and how will it affect clusters?
– E.g., InfiniBand
• How fat will multiprocessor servers be
and how to we build larger ones?
– E.g. Wisconsin Multifacet’s Multicast & Timestamp Snooping
• Future of multiprocessor servers & clusters?
– A merging of both?
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Outline
• Motivation
• System Area Networks
• Designing Multiprocessor Servers
• Server & Cluster Trends
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Technology Push: Moore’s Law
• What do following intervals have in common?
– Prehistory to 2000
– 2001 to 2002
• Answer: Equal progress in absolute processor speed
(and more doubling 2003-4, 2005-6, etc.)
– Consider salary doubling
• Corollary: Cost halves every two years
– Jim Gray: In a decade you can buy a computer
for less than its sales tax today
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Application Pull
• Should use computers in currently wasteful ways
– Already computers in electric razors & greeting cards
• New business models
– B2C, B2B, C2B, C2C
– Mass customization
• More proactive (beyond interactive) [Tennenhouse]
–
–
–
–
–
Today: P2C where P==Person & C==Computer
More C2P: mattress adjusts to save your back
More C2C: Agents surf the web for optimal deal
More sensors (physical/logic worlds coupled)
More hidden computers (c.f., electric motors)
• Furthermore, I am wrong
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
The Internet Iceberg
• Internet Components
–
–
–
–
Clients -- mobile, wireless
“On Ramp” -- LANs/DSL/Cable Modems
WAN Backbone -- IPv6, massive BW
and ...
• SERVICES
–
–
–
–
Scale Storage
Scale Bandwidth
Scale Computation
High Availability
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Outline
• Motivation
• System Area Networks
–
–
–
–
What is a SAN?
InfiniBand
Virtualizing I/O with Queue Pairs
Predictions
• Designing Multiprocessor Servers
• Server & Cluster Trends
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Regarding Storage/Bandwidth
• Currently resides on I/O Bus (PCI)
– HW & SW protocol stacks
– Must add hosts to add storage/bandwidth
proc
proc
memory interconnect
memory
bridge
i/o bus
i/o slot 0
(C) 2000 Mark D. Hill
i/o slot n-1
PODC00: Computer Architecture Trends
Want System Area Network (SAN)
• SAN vs. Local Area Nework (LAN)
–
–
–
–
–
Higher bandwidth (10 Gbps)
Lower latency (few microseconds or less)
More limited size
Other (e.g., single administrative domain, short distance)
Examples: Tandem Servernet & Myricom Myrinet
• Emerging Standard: InfiniBand
– www.inifinibandTA.org w/ spec 1.0 Summer 2000
– Compaq, Dell, HP, IBM, Intel, Microsoft, Sun, & others
– 2.5 Gbits/s times 1, 4, or 12 wires
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
InfiniBand Model (from website)
proc
proc
memory interconnect
memory
Other
networks
router
X
C
A
HCA (host channel adapter)
link
switch
T
C
A
target
(disks)
Other switches, hosts, targets, etc.
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Inifiniband Advantages
• Storage/Network made orthogonal from Computation
• Reduce “hardware” stack -- no i/o bridge
• Reduce “software” stack; hardware support for
–
–
–
–
–
Connected Reliable
Connected Unreliable
Datagram
Reliable Datagram
Raw Datagram
• Can eliminate system call for SAN use (next slide)
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Virtualizing InfiniBand
• I/O traditionally virtualized with system call
– System enforces isolation
– System permits authorized sharing
• Memory virtualized
– System trap/call for setup
– Virtual memory hardware for common-case translation
• Infiniband exploits “queue pairs” (QPs) in memory
– C.f., Intel Virtual Interface Architecture (VIA)
[IEEE Micro, Mar/Apr ‘98]
– Users issue sends, receives, & remote DMA reads/writes
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Queue Pair
proc
• QP setup system call
Main
Memory
dma-W4
dma-R3
send2 receive1
send1 receive2
– Connect with process
– Connect with remote QP
(not shown here)
• QP placed in “pinned”
virtual memory
• User directly access QP
HCA
(C) 2000 Mark D. Hill
– E.g., sends, receives &
remote DMA reads/writes
PODC00: Computer Architecture Trends
InfiniBand, cont.
• Roadmap
– NGIO/FIO merger in ‘99
– Spec in ‘00
– Products in ‘03-’10
• My Assessment
–
–
–
–
PCI needs successor
InfiniBand has the necessary features (but also many others)
InifiniBand has considerable industry buy-in (but it is recent)
Gigabit Ethernet will be only competitor
• Good name with backing from Cisco et al.
• But TCP/IP is a killer
– Infiniband for storage will be key
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
InfiniBand Research Issues
• Software Wide Open
– Industry will do local optimization
(e.g., still have device driver virtualized with system calls)
– But what is the “right” way to do software?
– Is there a theoretical model for this software?
• Other SAN Issues
–
–
–
–
A theoretical model of a service-providers site?
How to trade performance and availability?
Utility of broadcast or multicast support?
Obtaining quasi-real-time performance?
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
Outline
• Motivation
• System Area Networks
• Designing Multiprocessor Servers
–
–
–
–
How Fat?
Coherence for Servers
E.g., Multicast Snooping
E.g., Timestamp Snooping
• Server & Cluster Trends
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
How Fat Should Servers Be?
• Use
– PCs -- cheap but small
– Workgroup servers -- medium cost; medium size
– Large servers -- premium cost & size
• One answer: “yes”
PCs w/
“soft” state
(C) 2000 Mark D. Hill
Servers running
databases for
“hard” state
PODC00: Computer Architecture Trends
How Do We Build the Big Servers?
• (Industry knows how to build the small ones)
• A key problem is the memory system
– Memory Wall: E.g., 100ns memory access =
400 instruction opportunities for 4-way 1GHz processor
• Use per-processor caches to reduce
– Effective Latency
– Effective Bandwidth Used
• But cache coherence problem ...
(C) 2000 Mark D. Hill
PODC00: Computer Architecture Trends
“4”
r0