Myricom Myrinet/PCI-X Network Interface Cards Troubleshooting Guide

Myrinet-2000 Installation and Troubleshooting Guide Myricom, Inc. Draft: 07 April 2007 The most recent version of this document can be d

1. Plug in the power cord of the switch and the color TFT display (driven by the monitoring line card) will illuminate and exhibit a color-bar displa

Step 6: As soon as the ethernet port is connected, the upper green LED on the RJ45 connector will illuminate. Step 7: When the monitoring line card ha

Each time a monitoring line card is powered on, it will ask for its IP address (and netmask) via DHCP. You can specify a gateway with the DHCP "r

MX-2G or GM-2 software is required for use with the Myrinet-2000 M3-CLOS-ENCL-* and M3-SPINE-ENCL-* switches. MX-2G and GM 2.1.x support multi-path, d

$ ./configure --with-linux=<linux-source-dir> where <linux-source-dir> specifies the directory for the Linux kernel source. The k

MX libraries are installed in <install_path>/lib32 and the 64-bit MX libraries are installed in <install_path>/lib64. The <install_pat

The yellow "Lanai" LED is controlled by the Lanai processor, and will pulse like a heartbeat while the MCP/firmware is running. If an error

or (for RedHat Linux): chkconfig –add mx Alternatively, you may start and stop the driver manually using su root /etc/init.d/mx start /etc/ini

3. Enabling IP over Myrinet (Ethernet emulation) (OPTIONAL) If you wish to run IP over Myrinet (ethernet emulation), the Linux command to enable IP

If you would like to have FMS diagnostic monitoring with GM-2, refer to the FMS Download page (http://www.myri.com/scs/fms/) for installation instucti

Table of Contents I. Introduction...

• If you do not see any green “link” LEDs illuminated, is the switch powered on? • If you do not see green “link” LEDs illuminated on only a specifi

/var/run/gm_mapper/pid.{board_id}, and the map files are stored in /var/run/gm_mapper/map.{board_id}. Further details about the mapper in GM-2 can be

• Shuts down the gm_mapper daemon • ifconfig’s down the myri* ethernet devices • Unloads the GM module (using rmmod) The gm "restart" sc

VII. GM-1 Software Installation GM-1 installation is performed in four easy steps: 1. Configuring and compiling GM-1. 2. Installing the GM-1 driv

<install_path>/sbin/gm_install_drivers /etc/init.d/gm start on each machine to install/copy the driver on that machine. When the hardware is

If you wish the driver to auto-load at boot, you must create appropriate links in the /etc/rcN directories to the /etc/init.d/gm script. Alternative

Important points to note: • The GM-1 mapper is ONLY run on one node in the cluster. You should choose one node in the cluster to be the mapper node

If the GM tree is not mounted by NFS, copy the 3 files created by this command (static.map, static.routes, and static.hosts) to each <install_path&

VIII. Testing/Validation Once the MX, GM-2, or GM-1 firmware is running on all hosts in the cluster, and all host-to-switch and switch-to-switch cab

$ fm_switch -a <switch_name> where <switch_name> is the DNS name or IP address for the monitoring line card in the specific switch encl

I. Introduction This Myrinet-2000 Installation and Troubleshooting Guide describes the hardware and software installation procedures for a Myrine

the other end. On the host, there will be a green LED illuminated and a flashing yellow/amber LED illuminated on each NIC. If the LED of a connected

If you must have two PCI devices sharing the same PCI bus, and both devices are able to run at 133MHz, but the PCI bus is not running at 133MHz, are

mpicc to compile mx/unit_test/src/mpi/mpi_stress.c. The executable mpi_stress can then be run like any other MPI program using mpirun.ch_mx or mpirun

Appendix A: Determining if a Problem is Hardware or Software Related Diagnosing a problem as hardware- or software-related can be difficult. The fi

• Is there a monitoring line card installed in each Myrinet-2000 switch? If yes, do you see a high number of bad crcs reported in the switch count

• Did the firmware (MX or GM) load properly on all nodes in the cluster? Were there any error messages in the system log (dmesg or /var/log/messages)

If you are using M3-E* switches, two other useful hardware counters for diagnosing hardware failures are the switch counters called serdesFaultTrap a

Appendix B: Isolating the Cause of a Hardware Problem The following diagnostic procedures will need to be used if you are unable to install the Fab

mx_stop_mapper mx_msg_loop -n mx_counters | grep Bad where <hostname> is the name of the host on which the test is being run. Note that after

If the badcrc_cnt (reported in gm_counters) increased significantly after the test on any of the hosts, then you have identified a possible hardware t

"Guide to Switches and Switch Networks” http://www.myri.com/myrinet/m3switch/guide/ For Myrinet-2000 M3-CLOS-ENCL-* or M3-SPINE-ENCL-* switches

B.3. How do I determine if a Myrinet NIC has failed? If exchanging the cable and the port on the switch line card do not eliminate the errors, then

4. If you're using GM-1, run the gm_allsize "hardware loopback test" as follows: gm_counters [--board=n] gm_simpleroute --loopback

Appendix C: Troubleshooting Performance If you suspect a performance anomaly, we suggest: 1. Run mx_dmabench or gm_debug -L on each node in the c

significantly degraded performance. For a list of all options to gm_allsize, type gm_allsize –help or refer to the FAQ. For sample output of gm_alls

and on host2 type: gm_allsize --both-ways --bandwidth \ --remote-host=host1 --size=15 –geometric where the length of the messages se

• If your Myrinet-2000 M3-E* switch is equipped with a monitoring line card (located in the top-slot of the switch), this monitoring line card contai

• The M3-CLOS-ENCL-B (or M3-SPINE-ENCL-B) enclosure contains two 840W power supplies that can be individually hot-swapped, and operate in an auto-pa

• http://helics.iwr.uni-heidelberg.de/gallery/index.html Installation of the Myrinet PCI-X/PCI Network Interface Cards (NICs) Following the instal

• Have you tried using a different riser card and/or a different brand of riser card? • Have you tried using a newer BIOS for this motherboard? Ins

server will then serve this static IP address to the monitoring line card when it boots and asks for it. On Linux, this file is /etc/dhcpd.conf. The