Myricom Myrinet/PCI-X Network Interface Cards User Manual

1www.myri.com© 2005 Myricom, Inc.Cluster InterconnectCharles L. SeitzCEO & CTO of Myricom, [email protected] June 2005ISC2005 Interconnect Tuto

10www.myri.com© 2005 Myricom, Inc.Myrinet NICs = Protocol Offload EnginesSRAMinterfacex72SRAMx72SRAMJTAGinterfaceEEPROMinterfacePCI-Xinterfacepacketin

11www.myri.com© 2005 Myricom, Inc.Current-production Myrinet-2000/PCI-X NICsOne-port NICs:"D card" (225MHz) & "F card" (333MHz

12www.myri.com© 2005 Myricom, Inc.One more thing about Myrinet NICs…A Myrinet NIC looks from the standpoint ofinstallation exactly like an Ethernet NI

13www.myri.com© 2005 Myricom, Inc.Myrinet Software InterfacesApplicationsUDP TCPIPEthernet driver Myrinet driverIn theHostOSEthernetNICFirmware in the

14www.myri.com© 2005 Myricom, Inc.Choice of Myrinet Software Interfaces• Low-level APIs– GM 1 (legacy), GM 2 (current standard), MX (new), 3rd party (

15www.myri.com© 2005 Myricom, Inc.Myrinet Express (MX)• MX was designed based on earlier experience in writing MPICH-GM and VI-GM middleware– MX does

16www.myri.com© 2005 Myricom, Inc.Adaptive Route Dispersion• MX takes advantage of the multiple paths through large networks(c.f., slides 6 & 8) t

17www.myri.com© 2005 Myricom, Inc.Myrinet-2000 Summary• Low latency– 2.6s (MPI user level) for the fastestNICs, or– 3.2s for the lowest-cost NICs• H

18www.myri.com© 2005 Myricom, Inc.Architecture: “The Mirror Image of Ethernet”• A network architecture, protocol, and technology– A descendant of pack

19www.myri.com© 2005 Myricom, Inc.Myrinet = ANSI/VITA 26-1998Sourcerouteusedby theswitches, which strip the bytes as they are used Type (allows multi

2www.myri.com© 2005 Myricom, Inc.What is Myrinet?• A network architecture, protocol, and technology– A descendant of packet communication and routing

20www.myri.com© 2005 Myricom, Inc.Why does Myrinet work so well for clusters? (1)• The "Processor & Firmware in the NIC" Architecture– V

21www.myri.com© 2005 Myricom, Inc.Why does Myrinet work so well for clusters? (2)• Source Routing– Versus Ethernet: Destination Routing, in which …• …

22www.myri.com© 2005 Myricom, Inc.Myrinet Milestones – Technical Progress• 1994: Myricom founded; first Myrinet-640 product shipments• 1996: First Myr

23www.myri.com© 2005 Myricom, Inc.Myri-10G• Convergence of 10-Gigabit Myrinet and 10-Gigabit Ethernet– 4th-generation Myrinet (Myrinet-10G)– Any 10-Gi

24www.myri.com© 2005 Myricom, Inc.The Future of Myrinet-2000We expect Myrinet-2000 products to continue to sell wellthrough 2006, and to co-exist in t

25www.myri.com© 2005 Myricom, Inc.Summary of 4th-Generation Myrinet (1)• Links are 10-Gigabit Ethernet– Any 10-Gigabit Ethernet PHY can be used for My

26www.myri.com© 2005 Myricom, Inc.Summary of 4th-Generation Myrinet (2)• Switches are 10-Gigabit Myrinet– Although we leverage 10-Gigabit Ethernet tec

27www.myri.com© 2005 Myricom, Inc.10-Gigabit Myrinet Switches• Based on the10G_XBar16 chippictured– 16 XAUI ports– Power: ~15W• Packaged similarlyto M

28www.myri.com© 2005 Myricom, Inc.Myri-10G NIC (PCI Express)10GBase-CX4connectorPCI Express connectorLanai ZEunder heat sinkSRAMEEPROMPhotograph of a

29www.myri.com© 2005 Myricom, Inc.The Lanai-ZE Chip• PCI Express port– 4-lane or 8-lane• Dual-protocolMyri-10G port(XAUI)• High throughput– Internal p

3www.myri.com© 2005 Myricom, Inc.Myrinet-2000 fiber links, 2+2 Gbit/s, full duplexAdvantages of fiber: small-diameter, lightweight, flexible cables; r

30www.myri.com© 2005 Myricom, Inc.Questions?LSU SuperMike ClusterLANL Lightning Cluster, 1408 dual Opterons,entered TOP500 in Nov-2003 ranked #6,8051

4www.myri.com© 2005 Myricom, Inc.16-Port Myrinet-2000 SwitchMyricom makes small Myrinet switches,but we are best known for our large switches

5www.myri.com© 2005 Myricom, Inc.Myrinet-2000 Switches for Large Clusters512 Portsin the Clos256+256 configurationpictured, 256 host ports plus 256int

6www.myri.com© 2005 Myricom, Inc.Internal Clos Topology of the Clos256+256(Optional) 256 inter-switch ports presented as 64 quad-fiber ports256 ports

7www.myri.com© 2005 Myricom, Inc.A glimpse of the technology inside these switches64 Myrinet-2000 links on 16 quad-fiber front-panel ports64 Myrinet-2

8www.myri.com© 2005 Myricom, Inc.Designed to Scale256hosts512hosts768hosts1024hosts1280hostsClos256Clos256+256SpineAll inter-switch cabling on quad-li

9www.myri.com© 2005 Myricom, Inc.A Myrinet Switch Network for 2560 HostsThe switch network for the MareNostrum cluster at the Barcelona Supercomputing