************************************************************************
*           Myricom GM networking software and documentation           *
*                 Copyright (c) 2003 by Myricom, Inc.                  *
* All rights reserved.  See the file `COPYING' for copyright notice.   *
************************************************************************

README for GM-2.0

GM is a low-level communication layer for Myrinet.

************************************************************************
For updates to this software, visit `http://www.myri.com/scs'.

The FAQ file is located at `http://www.myri.com/scs/FAQ/'.

All Myrinet hardware and software technical questions should be directed
to help@myri.com.
************************************************************************

Table of Contents:

   I. Directory structure of GM distribution
  II. Installation
 III. GM Ports
  IV. Thread Safety
   V. GM Security 
  VI. GM with Different Topologies
 VII. GM Mapper and Mapper Tools
VIII. Recompiling GM MCP (optional)
  IX. Building User-Level GM programs
   X. GM Test Programs
  XI. GM Performance

=============================================
I. Directory Structure of the GM distribution
=============================================

The basic directory structure for the GM distribution is below:

gm-2.0
|-- doc                              GM API Documentation
|-- drivers                          GM Kernel Drivers
|   |-- aix
|   |-- freebsd
|   |-- hpux
|   |-- irix
|   |-- linux
|   |-- macosx
|   |-- nt4.0 
|   |-- nt5.0 
|   |-- solaris2.5
|   `-- vxworks
|-- examples                         GM Example Programs
|-- include                          Include directory
|-- libgm                            User-level GM API
|-- mcp                              GM Myrinet Control Program (MCP)
|   `-- libc
|-- mt                               GM Mapper
|   |-- ad_hoc_calculator
|   |-- barrel_simulation
|   |-- config
|   |-- deadlock
|   |-- dijkstra
|   |-- file_calculator
|   |-- force
|   |-- griddle
|   |-- libmt
|   |-- mapgen
|   |-- maps
|   |-- merlin
|   |-- mmapper
|   |-- nway
|   |-- ping
|   |-- simple
|   |-- smapper
|   |-- switch
|   |-- tools                        GM Mapper tools
|   `-- watch
|-- tests                            GM Test Suite
|-- util
`-- zlib

READMEs are located at various levels in this tree structure.

================
II. Installation
================

GM-2.0 is supported on the following operating systems and processors:

        Linux 2.4 for IA32, IA64, AMD64, Alpha, PowerPC, and 
           PowerPC64 (32-bit mode)
        Windows 2000 and Windows XP for IA32
        Windows XP for IA64

Supported interfaces: PCI64, PCI64A, PCI64B, PCI64C, PCIXD

        (If you have PCI32{A,B,C} interfaces, you will need to upgrade your
        interface, or use a previous version of GM (gm-1.2.3 for PCI32A,
        and gm-1.5.2.1 for PCI32B and PCI32C).

        For installation instructions of an earlier GM version
        please refer to the respective README and README-<arch> files.
        Please also note that Linux 2.4 is not supported on gm-1.2.3).

Installation instructions for GM 2.0 (and operating-system-specific
caveats) can be found in the respective README file:

     * README-linux

An earlier release of GM is supported on the following operating
systems and processors:

        AIX 4.3 and 5.1 for PowerPC
        FreeBSD v4.x for IA32 and Alpha
        IRIX 6.5 for Origin 200, Origin 2000, and Octane
        Linux 2.2 for IA32, Alpha, and PowerPC
        Linux 2.0 for IA32 and Alpha
        Mac OS X 10.1.x for PowerPC
        Solaris 6, 7, 8, 9 for UltraSPARC
        Tru64 (OSF1 V4.0e, V5.0, V5.1, V5.1A, V5.1B) for Alpha
        VxWorks for PowerPC Motorola MTX, Force, and CSPI boards.
        Windows NT 4.0 for IA32.


=============
III. GM Ports
=============

In the OS-bypass design, a port is a way for a process to communicate
directly with the Myrinet interface through GM.  Thus, the number of 
ports limits the number of processes concurrently using the Myrinet 
interface.  GM ports guarantee fairness and security between GM processes.

By default, this software supports 16 ports, and only 13 of these
(ports 2,4-15) are available to non-privileged user-level software.
Port 0 is for internal GM use and is never available.
Port 1 is used by the GM Mapper when running.
Port 3 is used by the IP-over-Myrinet driver, and the IP device
must be setup with
        ifconfig myri0 <ip_addr> up

To check the status of the GM ports:
   cd <GM_install_path>/bin
   ./gm_board_info

Note that these software ports are independent from the hardware ports 
on the Myrinet switch.

=================
IV. Thread Safety
=================

GM is not thread-safe but can be used successfully in a multi-threaded
application with a few constraints.  Basically, each call to GM should
be protected by a mutex.  As the host overhead of GM is particularly
small, the impact on performance of protecting the GM calls is
usually negligible.  Special provisions need to be made for the blocking
receive case.  Refer to the "Is GM thread-safe?" entry in the FAQ for 
further details.
    http://www.myri.com/scs/FAQ/

==============
V. GM Security
==============

By default, the security features of GM are enabled.  In particular,
opening privileged ports (like the mapper port) is restricted 
to privileged users.

Since GM_ENABLE_SECURITY is set in the driver source code, GM performs checks
to ensure that the Unix security model is not violated.  For example, only
root would be able to open the mapper port and configure the network.  With
GM_ENABLE_SECURITY set, GM is believed to maintain the integrity of Unix 
system security.

=================================
VI.  GM with Different Topologies
=================================

     * Loopback Configuration
     * Point-to-Point Configuration
     * Switch Configuration

GM will provide a communication layer in a Myrinet interface in these
three topologies: (1) connected to a loopback cable, (2) connected via
a cable to another installed interface (point-to-point), or (3)
connected via a cable to a switch.  GM implements a software loopback
so that a message leaving the network interface can return to this
network interface without a switch in between.  MPICH-GM will
work in loopback mode if the shared memory support is enabled as 
described in the MPICH-GM documentation, as will IP-over-GM since both 
of these protocols loop back messages above the GM messaging layer.

===============================
VII. GM Mapper and Mapper Tools
===============================

As of GM-2.0 and later, the GM Mapper is automatically run
when you load the GM module (/etc/init.d/gm start).  Refer to
the respective README-<arch> for installation details.

The Mapper Tools are available for GM-2.0.

========================================
VIII.  Recompiling the GM MCP (optional)
========================================

The GM distribution contains prebuilt GM Myrinet Control Programs 
(firmware) for all supported Myrinet interfaces.

The user does not need to compile the firmware.  The default prebuilts
cover the large majority of configurations.  However, you may need
to recompile the MCP if you use unsupported configure options that would
force a rebuild.  Myricom does not provide support if the firmware has been
modified.

The GM-2 firmware source is not included in the GM-2 distribution.
Access to the GM-2 firmware source software requires approval by Myricom
management.  Contact help@myri.com for further details.

Building the firmware requires lanai-tools, lanai-gcc and gendat.
These tools are available at http://www.myri.com/scs/L3.  It is
also necessary to configure with --enable-maintainer-mode.


===================================
IX. Building User-Level GM Programs
===================================

Building GM programs is straightforward.  Using the "gm.h" header file
and the associated library, one can build GM applications just as with 
any other library.  The only difference is that before the program can
be run, the GM driver and any dynamically linked GM library must be 
installed.

Example programs which call GM can be found in {GM_HOME}/examples/.

Architecture-specific details follow.

Unix
----

Under Unix, the header file is called "gm.h", the static library is
"libgm.a", and the dynamically linked library is "libgm.so".

Windows 2000
------------

Under Windows 2000, the header file is "gm.h," the import library is
"gm.lib," and the dynamically linked library is "gm.dll."


===================
X. GM Test Programs
===================

Once the GM software has been properly installed on all of the
hosts in your cluster, you are ready to validate your Myrinet
installation by performing the following sequence of tests.

   * Check the LEDs on each switch port and interface port
   * Run gm_board_info on one host
   * Run gm_debug to test the PCI bandwidth
   * Run gm_allsize to test the links in the network
   * Run gm_stress to test the network
   * Run Mute on the cluster to test for bad links

Each of these steps is detailed in the Troubleshooting section of the FAQ
   http://www.myri.com/scs/FAQ/

The test scripts (gm_board_info, gm_debug, gm_allsize, gm_stress) are
available in <GM_install_path>/bin in your GM installation. A README
describing each of these tests can be found in <GM_install_path>/bin/README.
Mute is not included in the GM distribution, but can be downloaded from
   http://www.myri.com/scs/

==================
XI. GM Performance
==================

As stated in the document "Guide to Myrinet/PCI Host Interfaces",
the Myrinet-2000 interfaces are capable of sustained PCI/PCI-X data rates 
approaching the limits of the PCI bus (528 MB/s for 64-bit, 66MHz;
264 MB/s for 64-bit, 33HMz or 32-bit, 66MHz; 132 MB/s for 32-bit,
33MHz) or the PCI-X bus (1067 MB/s for 64-bit, 133MHz).  However, the data
rate to/from system memory will depend upon the host's memory and PCI-bus/
PCI-X bus implementation.  The achievable performance can vary greatly from
one PCI chipset to another.

A host-to-host communication is limited by the weakest component
in the data path.  If the performance of the PCI/PCI-X bus does not
exceed the data rate of the Myrinet link, it will be a bottleneck
and thus limit the global data rate of the communication.  The
maximum performance of GM should be very close to the maximum
performance of the PCI/PCI-X bus if the latter is a limitation.

From the GM Test Suite, you can run `gm_debug -L' to determine the
maximum sustained bandwidth from the PCI/PCI-X bus, and determine if the
PCI/PCI-X bus is the bottleneck to your performance.  You can also obtain
information from the documentation of your motherboard, or run an
OS-specific command to determine the type of PCI chipset on the
motherboard.

For performance comparison, we refer you to the following webpage,
which collects performance data of several motherboards and PCI
chipsets,
  http://www.conservativecomputers.com/myrinet/perf.html
and encourage you to submit the specifications of your machine
(motherboard) with the result of the previous benchmark.
