************************************************************************
*           Myricom GM networking software and documentation           *
*                 Copyright (c) 2003 by Myricom, Inc.                  *
* All rights reserved.  See the file `COPYING' for copyright notice.   *
************************************************************************
 
README-Tru64 for gm-1.6.4

    Supported platforms:  Tru64 UNIX V4.0e, V5.0, V5.1, V5.1A, and V5.1B
                          for Alpha.

    Supported interfaces: LANai7 (PCI64, PCI64A), and LANai9 (PCI64B, PCI64C)
 
                         (If you have LANai4 with 256K, you will need to
                         upgrade your interface, or use a previous version
                         of GM (gm-1.2.3).  For installation
                         instructions of an earlier GM version please refer
                         to the respective README and README-<arch> files.)

Table of Contents:
-----------------
 
  I.  GM Binary Installation Instructions
      a.  Unpacking the GM driver
      b.  Installing the GM driver
      c.  Running the GM Mapper
      d.  Enabling IP over Myrinet (Ethernet emulation)
      e.  Testing the GM Installation
 II.  GM Source Installation Instructions
      a.  Configuring and compiling GM
      b.  Installing the GM driver
      c.  Running the GM Mapper
      d.  Enabling IP over Myrinet (Ethernet emulation)
      e.  Testing the GM Installation
III.  Verifying the GM performance
 IV.  Improving IP Performance
  V.  Cautions/Caveats
 
************************************************************************
If difficulties are encountered, please consult the FAQ
         http://www.myri.com/scs/FAQ/
and all technical support questions should be directed to help@myri.com.

If you have problems on any architectures, please send the output from
/var/adm/messages to <help@myri.com>.
************************************************************************

======================================
I. GM Binary Installation Instructions
======================================

Current cautions and common problems:

   * If the host is rebooted, you must rerun the GM Mapper. 
   * The GM Mapper must be run before any communication over Myrinet can occur. 
   * In most cases, the GM Mapper should not be run continuously. It
     is intrusive and can interfere with network traffic intensive jobs
     (e.g, MPI applications). 
   * We advise you to avoid unloading the GM module. This means one
     should NEVER run '/sbin/init.d/gm start' twice on any machine!

GM installation is performed in five easy steps:

1. Unpacking the GM driver. 
---------------------------

When reading the the instructions below, please replace VERSION with
your operating system version (V4.0E, V5.0, V5.1) as returned by sizer
-v, and please replace REVISION with your operating system revision as
returned by uname -r (V4.0, V5.0, V5.1).

        gunzip -c gm-1.6.4_for_Tru64-alpha-OSF1-VERSION-new-features-8ports.tar.gz | tar xvf -

2.  Installing the GM driver.
-----------------------------

Select an installation directory path <install_path>.  It is
usually best for <install_path> to be the path to an NFS
directory available on all machines that are to share this GM
installation.  The directory must be accessible using
<install_path> on all machines that are to share the
installation.  <install_path> must be an absolute path; it
must start with "/".  However, <install_path> may contain
symbolic links.

    cd gm-1.6.4_for_Tru64-alpha-OSF1-REVISION 
    ./GM_INSTALL <install_path>

If you omit <install_path>, the driver will be installed in
the default directory, /opt/gm/.

Next, you must run

    su root
    <install_path>/sbin/gm_install_drivers

If you have never installed GM on this host, you may load the driver
using:
       /sbin/init.d/gm start

If you have previously installed GM on this host, you should reboot
your system now.  After the system has been rebooted, you may load
the driver via:

       /sbin/init.d/gm start

Important note:  The gm_install_drivers script does not configure
the IP device.  If you wish to run IP over GM/Myrinet (ethernet
emulation), you must configure the device.  Refer to step 4.

3. Running the GM Mapper.
------------------------

   cd <install_path>/sbin/
   su root
   ./mapper ../etc/gm/map_once.args

There are 3 ways in which the GM Mapper can be run. Refer to section 2
'Running the Mapper' of the README-linux for details.

4. Enabling IP over Myrinet (Ethernet Emulation) (OPTIONAL)
-----------------------------------------------------------

The IP device is accessed via
   ifconfig myri0 <address> netmask <netmask> broadcast <bcast> up

where you must replace 'myri0' with the appropriate name (myri1, myr2, etc.)
if you have more than one Myrinet interface per host.  For more information,
please refer to the FAQ (http://www.myri.com/scs/FAQ/).

For suggestions on improving performance, please refer to section
"III. Improving IP Performance".

5. Testing the GM Installation.
------------------------------

Once the GM software has been properly installed on all of the
hosts in your cluster, you are ready to validate your Myrinet
installation by performing the following sequence of tests.

   * Check the LEDs on each switch port and interface port
   * Run gm_board_info on one host
   * Run gm_debug to test the PCI bandwidth
   * Run gm_allsize to test the links in the network
   * Run gm_stress to test the network
   * Run Mute on the cluster to test for bad links

Each of these steps is detailed in the Troubleshooting section of the FAQ
   http://www.myri.com/scs/FAQ/

The test scripts (gm_board_info, gm_debug, gm_allsize, gm_stress) are
available in <install_path>/bin in your GM installation. A README
describing each of these tests can be found in <install_path>/bin/README.
Mute is not included in the GM distribution, but can be downloaded from
   http://www.myri.com/scs/

=======================================
II. GM Source Installation Instructions 
=======================================

To compile GM on Tru64 Alpha platforms, you will need GNU make, the
Compaq C compiler, and either the GNU or Compaq C++ compilers.

Current cautions and common problems:

   * If the host is rebooted, you must rerun the GM Mapper. 
   * The GM Mapper must be run before any communication over Myrinet can occur. 
   * In most cases, the GM Mapper should not be run continuously. It
     is intrusive and can interfere with network traffic intensive jobs
     (e.g, MPI applications). 
   * We advise you to avoid unloading the GM module. This means one
     should NEVER run '/sbin/init.d/gm start' twice on any machine!

GM installation is performed in five easy steps:

1. Configuring and compiling the GM driver:
-------------------------------------------

        gunzip -c gm-1.6.4_for_Tru64.tar.gz | tar xvf -
        cd gm-1.6.4_for_Tru64
        ./configure
        make

2. Installing the GM driver:
----------------------------

Select an installation directory path <install_path>.  It is
usually best for <install_path> to be the path to an NFS
directory available on all machines that are to share this GM
installation.  The directory must be accessible using
<install_path> on all machines that are to share the
installation.  <install_path> must be an absolute path; it
must start with "/".  However, <install_path> may contain
symbolic links.

    cd gm-1.6.4_for_Tru64
    ./GM_INSTALL <install_path>

If you omit <install_path>, the driver will be installed in
the default directory, /opt/gm/.

Next, you must run

    su root
    <install_path>/sbin/gm_install_drivers

If you have never installed GM on this host, you may load the driver
using:
       /sbin/init.d/gm start

If you have previously installed GM on this host, you should reboot
your system now.  After the system has been rebooted, you may load
the driver via:

       /sbin/init.d/gm start

Important note:  The gm_install_drivers script does not configure
the IP device.  If you wish to run IP over GM/Myrinet (ethernet
emulation), you must configure the device.  Refer to step 4.

3. Running the GM Mapper.
------------------------

   cd <install_path>/sbin/
   su root
   ./mapper ../etc/gm/map_once.args

There are 3 ways in which the GM Mapper can be run. Refer to section 2
'Running the Mapper' of the README-linux for details.

4. Enabling IP over Myrinet (Ethernet Emulation) (OPTIONAL)
-----------------------------------------------------------

The IP device is accessed via
   ifconfig myri0 <address> netmask <netmask> broadcast <bcast> up

where you must replace 'myri0' with the appropriate name (myri1, myr2, etc.)
if you have more than one Myrinet interface per host.  For more information,
please refer to the FAQ (http://www.myri.com/scs/FAQ/).

For suggestions on improving performance, please refer to section
"III. Improving IP Performance".

5. Testing the GM Installation.
------------------------------

Once the GM software has been properly installed on all of the
hosts in your cluster, you are ready to validate your Myrinet
installation by performing the following sequence of tests.

   * Check the LEDs on each switch port and interface port
   * Run gm_board_info on one host
   * Run gm_debug to test the PCI bandwidth
   * Run gm_allsize to test the links in the network
   * Run gm_stress to test the network
   * Run Mute on the cluster to test for bad links

Each of these steps is detailed in the Troubleshooting section of the FAQ
   http://www.myri.com/scs/FAQ/

The test scripts (gm_board_info, gm_debug, gm_allsize, gm_stress) are
available in <install_path>/bin in your GM installation. A README
describing each of these tests can be found in <install_path>/bin/README.
Mute is not included in the GM distribution, but can be downloaded from
   http://www.myri.com/scs/

=================================
III. Verifying the GM Performance
=================================
 
We recommend the following test to verify the GM performance.
 
     cd {GM_HOME}/binary/bin
     ./gm_debug --no-counters

This gm_debug test displays the results of the hardware benchmark test of
the PCI bus with the DMA engine of the Myrinet interface. The output of this
command indicates the maximum sustained bandwidth that can be obtained from
the PCI bus, and thus provides an upper bound on GM performance. A detailed
description of this benchmark can be found in the FAQ
(http://www.myri.com/scs/FAQ/).

The output of this command also tells you if the Myrinet interface was
correctly detected as 64-bit / 66 MHz, for example. If the interface was
not correctly detected by the BIOS, you should suspect a riser card problem
or a PCI slot problem.

Performance graphs (http://www.myri.com/myrinet/performance) for GM on
Linux are available. The performance measurements were obtained by running
gm_allsize tests for latency and bandwidth as described in the FAQ entry
("What are the run-time options to gm_allsize?"). Refer to the section
entitled "GM Performance" in the <GM_HOME>/README for complete
details on expected GM performance.
 
============================
IV. Improving IP performance
============================

If you don't see the speed that you expect, you might consider adjusting
the TCP send and receive window sizes.

Tru64 5.0 and newer ship with a fairly reasonable set of defaults for
the tcp window size.  If you're running 4.x, you may want to increase
the window size to between 64k and 256k.

To adjust the window sizes, modify the inet:tcp_sendspace and
inet:tcp_recvspace like this:

# sysconfig -r inet tcp_sendspace=262144
# sysconfig -r inet tcp_recvspace=262144

You can make this happen automatically at boot via the sysconfigdb
command.  Read the sysconfigdb(8) man page for details.

===================
V. Cautions/Caveats
===================

*************************************************************************
Never run '/sbin/init.d/gm start' twice on any machine.
To simply update the driver binary (e.g. after you have run 
'/sbin/init.d/gm start' once at some point in time in the distant past
for a previous GM release),
	/sbin/sysconfigdb -d gm
        /sbin/reboot
        /sbin/init.d/gm start

You need to run the mapper at least once after loading the driver
on all nodes in your network.
cd sbin
mapper ../etc/gm/active.args

You might want to run
        gm_setname -h `hostname` 
to write a "useful" name into the card/driver. Otherwise, the
gm_name of the machine will be an ASCII-ized version of
its 48-bit ID.


COMPILING from SOURCE
By default, the configure will use cc and c++.
To compile from source you'll need to have a /usr/sys/<HOSTNAME>
or a /usr/sys/MYRI directory. We put both of those on the "-I" include
path list for building kernel code. If the machine has ever been used
for building a kernel, the /usr/sys/<HOSTNAME> directory is probably
already populated.

To compile from source using the Tru64 C and C++ compilers
 
#DIGITAL C++ V6.1-027
setenv CC /usr/ccs/bin/cc
setenv CFLAGS "-std  -msg_disable extrasemi,emptyfile "
setenv CXX /usr/bin/cxx
setenv CXXFLAGS "-O2 -inline speed -x cxx -using_std -w2 -msg_disable extrasemi,emptyfile -tune generic"
rm -f config.cache
./configure


#Compaq C++ V6.2-024 
setenv CC /usr/ccs/bin/cc
setenv CFLAGS "-std  -msg_disable extrasemi,emptyfile "
setenv CXX /usr/bin/cxx
setenv CXXFLAGS "-O2 -inline speed -x cxx -nocxxstd -using_std -w2 -msg_disable extrasemi,emptyfile -tune generic"
rm -f config.cache
./configure
*************************************************************************


*************************************************************************
The driver is theoretically unloadable, but we have not found an
"if_detach" function, so reloading the driver will result in multiple
myri0 interfaces showing up in the system. Most likely, any ifconfig or
IP traffic will cause a system panic.

The driver developer docs at
http://www.unix.digital.com/faqs/publications/dev_doc/usr/examples/ddk/lan/README

say this about unloading a network driver.

NOTES:
------

o LAN drivers can only be loaded.  There is currently no method for unloading
  LAN drivers.  We are planning on fixing this for a future release.
*************************************************************************


****************************************************************
If you have problems running the mapper and see something
like this
./mapper: /sbin/loader: Fatal Error: cannot map libstdc++.so.2.8.0


then set up a symbolic link from libstdc++.so.2.8.1 to libstdc++.so.2.8.0
assuming you can find libstdc++.so.2.8.0 in /usr/local/lib.

If not, a copy is at
ftp.myri.com
cd /pub/GM
<use the normal user and passwd you use for myrinet software>

www% ls -l lib*
-rw-r--r--  1 web        399700 Jul 16 16:57 libstdc++.so.2.8.0.gz
-rw-r--r--  1 web        399613 Aug 16 17:45 libstdc++.so.2.8.1.gz


This should let you use the mapper until we release a statically
linked version.

We're now compiling with libstdc++.so.2.8.1 so both are
available on our web site.

****************************************************************

To unload the driver you need to do the following
/sbin/sysconfigdb -d gm
/sbin/reboot
 
 The system will boot with the gm driver not loaded.
 You can then
        <install_path>/sbin/gm_install_drivers
to re-load it.

****************************************************************

If you see errors of this form -

gm: LANAI: gm_sdma.h: call handle_send_error(0x0x4f698,18) destid=0
gm: LANAI: handle_send_error: subport 4 handling send error 18
gm: LANAI: handle_send_error: disabling subport 4

then you will need to apply the latest aggregate patch for Tru64 5.0,
which is currently T64V50AS0003-20010221. This patch can be found at:

   http://ftp.support.compaq.com/patches/.new/unix.html

t64v50as0003-20010221.README, T64V50AS0003-20010221 Tru64 V5.0
                         ECO Summary
                         12-Mar-2001  

use 
uname -a 
to check your Tru64 version

and 
sys_check -config -nohtml > output_file

to check what patches you have currently installed (look
in the output_file for "Installed patch kit versions")

******************************************************************
