************************************************************************
*           Myricom GM networking software and documentation           *
*                 Copyright (c) 2001 by Myricom, Inc.                  *
* All rights reserved.  See the file `COPYING' for copyright notice.   *
************************************************************************
 
README-Tru64 for gm-1.5

    Supported platforms:  Tru64 UNIX V4.0e, V5.0, V5.1, and V5.1A for Alpha.

    Supported interfaces:  LANai4 with 1MB or 512K (PCI32, SBus),
                           LANai7 (PCI64, PCI64A), and LANai9 (PCI64B, PCI64C)
 
                         (If you have LANai4 with 256K, you will need to
                         upgrade your interface, or use a previous version
                         of GM (gm-1.2.3).  For installation
                         instructions of an earlier GM version please refer
                         to the respective README and README-<arch> files.
                         Please also note that Linux 2.4 is not supported on
                         earlier GM versions).

Table of Contents:
-----------------
 
  I.  GM Binary Installation Instructions
      a.  Unpacking and loading the GM driver
      b.  Running the GM Mapper
      c.  Testing the GM Installation
 II.  GM Source Installation Instructions
      a. Configuring, compiling, and loading the GM driver
      b.  Running the GM Mapper
      c.  Testing the GM Installation
III.  Verifying the GM performance
 IV.  Running IP over GM
  V.  Improving IP Performance
 VI.  Cautions/Caveats
 
************************************************************************
If difficulties are encountered, please consult the FAQ
         http://www.myri.com/scs/GM_FAQ.html
and all technical support questions should be directed to help@myri.com.

If you have problems on any architectures, please send the output from
/var/adm/messages to <help@myri.com>.
************************************************************************

======================================
I. GM Binary Installation Instructions
======================================

Current cautions and common problems:

   * If the host is rebooted, you must rerun the GM Mapper. 
   * The GM Mapper must be run before any communication over Myrinet can occur. 
   * In most cases, the GM Mapper should not be run continuously. It
     is intrusive and can interfere with network traffic intensive jobs
     (e.g, MPI applications). 
   * We advise you to avoid unloading the GM module. This means one
     should NEVER run GM_INSTALL twice on any machine!

GM installation is performed in three easy steps:

1. Unpacking and loading the GM driver. 
--------------------------------------

When reading the the instructions below, please replace VERSION with
your operating system version (V4.0E, V5.0, V5.1) as returned by sizer
-v, and please replace REVISION with your operating system revision as
returned by uname -r (V4.0, V5.0, V5.1).

        gunzip -c gm-1.5pre1_for_Tru64-alpha-OSF1-VERSION-new-features-8ports.tar.gz | tar xvf -
        cd gm-1.5pre1_for_Tru64-alpha-OSF1-REVISION 
        su root

If you have never installed GM on this host:

        ./GM_INSTALL

If you have previously installed GM on this host:

        ./GM_copy_driver
            shutdown -r now

2. Running the GM Mapper.
------------------------

        cd gm-1.5pre1_for_Tru64-alpha-OSF1-REVISION/sbin/
        su root
        ./mapper ../etc/gm/map_once.args

There are 3 ways in which the GM Mapper can be run. Refer to section 2
'Running the Mapper' of the README-linux for details.

3. Testing the GM Installation.
------------------------------

A variety of test scripts are available in binary/bin/ to test your GM
installation. We recommend the five tests described in tests/README.

=======================================
II. GM Source Installation Instructions 
=======================================

To compile GM on Tru64 Alpha platforms, you will need GNU make, the
Compaq C compiler, and either the GNU or Compaq C++ compilers.

Current cautions and common problems:

   * If the host is rebooted, you must rerun the GM Mapper. 
   * The GM Mapper must be run before any communication over Myrinet can occur. 
   * In most cases, the GM Mapper should not be run continuously. It
     is intrusive and can interfere with network traffic intensive jobs
     (e.g, MPI applications). 
   * We advise you to avoid unloading the GM module. This means one
     should NEVER run GM_INSTALL twice on any machine!

GM installation is performed in three easy steps:

1. Configuring, compiling, and loading the GM driver:
----------------------------------------------------

        gunzip -c gm-1.5pre1_for_Tru64.tar.gz | tar xvf -
        cd gm-1.5pre1_for_Tru64
        ./configure
        make
        cd binary
        su root

If you have never installed GM on this host:

        ./GM_INSTALL

If you have previously installed GM on this host:

        ./GM_copy_driver
            shutdown -r now

2. Running the GM Mapper.
------------------------

        cd binary/sbin/
        su root
        ./mapper ../etc/gm/map_once.args

There are 3 ways in which the GM Mapper can be run. Refer to section 2
'Running the Mapper' of the README-linux for details.

3. Testing the GM Installation.
------------------------------

A variety of test scripts are available in binary/bin/ to test your GM
installation. We recommend the five tests described in tests/README.

=================================
III. Verifying the GM Performance
=================================
 
   We recommend the following test to verify the GM performance.  View
   the results of the hardware benchmark test of the PCI bus with the
   DMA engine of the Myrinet adapter.
 
       cd {GM_HOME}/binary/bin
       ./gm_debug --no-counters
 
   Note: The output of this command gives the maximum sustained bandwidth
         that can be obtained from the PCI bus.  Refer to the section
         entitled "GM Performance" in the {GM_HOME}/README for complete
         details on expected GM performance.

======================
IV. Running IP over GM
======================

The IP device is accessed via
        ifconfig myri0 <address> netmask <netmask> broadcast <bcast> up

where you must replace 'myri0' with the appropriate name (myri1, myr2, etc.)
if you have more than one Myrinet interface per host.  For more information,
please refer to the FAQ (http://www.myri.com/scs/GM_FAQ.html).

 
===========================
V. Improving IP performance
===========================

If you don't see the speed that you expect, you might consider adjusting
the TCP send and receive window sizes.

Tru64 5.0 and newer ship with a fairly reasonable set of defaults for
the tcp window size.  If you're running 4.x, you may want to increase
the window size to between 64k and 256k.

To adjust the window sizes, modify the inet:tcp_sendspace and
inet:tcp_recvspace like this:

# sysconfig -r inet tcp_sendspace=262144
# sysconfig -r inet tcp_recvspace=262144

You can make this happen automatically at boot via the sysconfigdb
command.  Read the sysconfigdb(8) man page for details.

====================
VI. Cautions/Caveats
====================

*************************************************************************
Never run GM_INSTALL twice on any machine.
To simply update the driver binary 
(e.g. after you have run GM_INSTALL once at some point in time 
in the distant past for a previous GM release),
        ./GM_copy_driver
        sync
        /sbin/reboot

run
./GM_INSTALL 
to install/configure the driver.

You need to run the mapper at least once after loading the driver
on all nodes in your network.
cd sbin
mapper ../etc/gm/active.args

You might want to run
        gm_setname -h `hostname` 
to write a "useful" name into the card/driver. Otherwise, the
gm_name of the machine will be an ASCII-ized version of
its 48-bit ID.


COMPILING from SOURCE
By default, the configure will use cc and c++.
To compile from source you'll need to have a /usr/sys/<HOSTNAME>
or a /usr/sys/MYRI directory. We put both of those on the "-I" include
path list for building kernel code. If the machine has ever been used
for building a kernel, the /usr/sys/<HOSTNAME> directory is probably
already populated.

To compile from source using the Tru64 C and C++ compilers
 
#DIGITAL C++ V6.1-027
setenv CC /usr/ccs/bin/cc
setenv CFLAGS "-std  -msg_disable extrasemi,emptyfile "
setenv CXX /usr/bin/cxx
setenv CXXFLAGS "-O2 -inline speed -x cxx -using_std -w2 -msg_disable extrasemi,emptyfile -tune generic"
rm -f config.cache
./configure


#Compaq C++ V6.2-024 
setenv CC /usr/ccs/bin/cc
setenv CFLAGS "-std  -msg_disable extrasemi,emptyfile "
setenv CXX /usr/bin/cxx
setenv CXXFLAGS "-O2 -inline speed -x cxx -nocxxstd -using_std -w2 -msg_disable extrasemi,emptyfile -tune generic"
rm -f config.cache
./configure
*************************************************************************


*************************************************************************
The driver is theoretically unloadable, but we have not found an
"if_detach" function, so reloading the driver will result in multiple
myri0 interfaces showing up in the system. Most likely, any ifconfig or
IP traffic will cause a system panic.

The driver developer docs at
http://www.unix.digital.com/faqs/publications/dev_doc/usr/examples/ddk/lan/README

say this about unloading a network driver.

NOTES:
------

o LAN drivers can only be loaded.  There is currently no method for unloading
  LAN drivers.  We are planning on fixing this for a future release.
*************************************************************************


****************************************************************
If you have problems running the mapper and see something
like this
./mapper: /sbin/loader: Fatal Error: cannot map libstdc++.so.2.8.0


then set up a symbolic link from libstdc++.so.2.8.1 to libstdc++.so.2.8.0
assuming you can find libstdc++.so.2.8.0 in /usr/local/lib.

If not, a copy is at
ftp.myri.com
cd /pub/GM
<use the normal user and passwd you use for myrinet software>

www% ls -l lib*
-rw-r--r--  1 web        399700 Jul 16 16:57 libstdc++.so.2.8.0.gz
-rw-r--r--  1 web        399613 Aug 16 17:45 libstdc++.so.2.8.1.gz


This should let you use the mapper until we release a statically
linked version.

We're now compiling with libstdc++.so.2.8.1 so both are
available on our web site.

****************************************************************

To unload the driver you need to do the following
/sbin/sysconfigdb -d gm
/sbin/reboot

The system will boot with the gm driver not loaded.
You can then
        ./GM_INSTALL
to re-load it.

****************************************************************

If you see errors of this form -

gm: LANAI: gm_sdma.h: call handle_send_error(0x0x4f698,18) destid=0
gm: LANAI: handle_send_error: subport 4 handling send error 18
gm: LANAI: handle_send_error: disabling subport 4

then you will need to apply the latest aggregate patch for Tru64 5.0,
which is currently T64V50AS0003-20010221. This patch can be found at:

   http://ftp.support.compaq.com/patches/.new/unix.html

t64v50as0003-20010221.README, T64V50AS0003-20010221 Tru64 V5.0
                         ECO Summary
                         12-Mar-2001  

use 
uname -a 
to check your Tru64 version

and 
sys_check -config -nohtml > output_file

to check what patches you have currently installed (look
in the output_file for "Installed patch kit versions")

******************************************************************
