************************************************************************
*           Myricom GM networking software and documentation           *
*                 Copyright (c) 2003 by Myricom, Inc.                  *
* All rights reserved.  See the file `COPYING' for copyright notice.   *
************************************************************************
 
README-solaris for gm-2.0.6
 
    Supported platforms:  Solaris 8 and higher in 32-bit and 64-bit modes on
                          UltraSPARC, including the new UltraSPARC IIIi.

                          GM 2.0.6 software for Solaris on x86 is not supported.
 
    Supported interfaces: PCI64, PCI64A, PCI64B, PCI64C, PCIXD

			  Note:  gm-2.0 does not interoperate with gm-1.x.
			  A mixture of hosts with gm-1.x and gm-2.0 cannot
			  talk to each other.

                          If you have PCI32{A,B,C} you will need to upgrade 
			  your interface, or use a previous version of GM.
                          Use gm-1.6_Solaris for LANai4 with 1MB (PCI32, SBus), 
			  or gm-1.2.3 For LANai4 with 256K or 512K. 

                          For installation instructions of an earlier GM
                          version please refer to the respective  README
                          and README-<arch> files.
 
    Important notes:	  The GM-2 software release is meant to principally for
			  use with the new M3F-PCIXD interfaces. Althought this
			  GM-2 release also works with PCI64-series Myrinet 
			  interfaces, it is not yet as well optimized for 
			  performance as gm 1.6.4, and inevitably will not be as
			  stable. Thus, we do not recommend at this time upgrading 
			  clusters using PCI64-series interfaces to use GM 2. 
			  Within a few months, upgrading from GM 1.6.x to GM 2.x 
			  will makes send for sites that want to take advantage of 
			  the new features of GM 2.
 
			  PCIXD interfaces require GM-2.0 or later.
		
			  If you will be running ClusterTools' MPI application
			  over GM-2, you must use CT_Myrinet_PM_0.8 or later.


Table of Contents:
-----------------

  I.  GM Binary Installation Instructions
      a.  Unpacking the GM driver
      b.  Loading the GM driver
      c.  Enabling IP over Myrinet (Ethernet emulation) (OPTIONAL)
      d.  Testing the GM Installation
 II.  GM Source Installation Instructions
      a.  Configuring and compiling GM
III.  Verifying GM performance
 IV.  IP support via Ethernet Emulation 
  V.  Miscellaneous
      a.  Uninstallation of the GM driver
 
************************************************************************
If difficulties are encountered, please consult the FAQ
         http://www.myri.com/scs/FAQ/
and all technical support questions should be directed to help@myri.com.
 
************************************************************************

======================================
I. GM Binary Installation Instructions
======================================

1. Unpacking the GM driver
---------------------------

        gunzip -c gm-2.0.6_Solaris-sun4u-SunOS-5.8.tar.gz | tar xvf -
        cd gm-2.0.6_Solaris-sun4u-SunOS-5.8

2. Loading the GM driver
-------------------------

	Select an installation directory path <install_path>.  It is usually
	best for <install_path> to be the path to an NFS directory available
	on all machines that are to share this GM installation.  The directory
	must be accessible using <install_path> on all machines that are to
	share the installation.  <install_path> must be an absolute path;
	it must start with "/".  However, <install_path> may contain
	symbolic links.

        cd binary
        ./GM_INSTALL <install_path>

	If you omit the <install_path>, the driver will be installed in the
	default directory, /opt/gm/.

	Next, you must run

        su root
        <install_path>/sbin/gm_install_drivers
	/etc/init.d/gm start

	on each machine in your cluster.
 
        The GM_INSTALL script copies the GM binaries to the specified
	binary installation directory <install_path>.
	
        The gm_install_drivers script performs the following operations:

          * Copies gm into /kernel/drv/($SPARCV9)/gm
	  * Removes the previous installation by executing 
	    /sbin/gm_uninstall_drivers (rem_drv)
	  * Copies other files from the binary installation directory to 
	    an architecture-specific directory (/etc/init.d/).
          * Creates the devices (/dev/gm* and /dev/gmp*), one device
	    per interface
	  * Creates the mapper's per-host configuration directory
	    (/etc/gm_mapper) and possibly store configuration files there.
	    
        The gm "start" script performs the following operations:

	  * Loads the GM module (add_drv)
	  * Starts a mapper daemon called "gm_mapper" for each
	    Myrinet interface contained in the machine.  The PIDs of the
	    running gm_mappers are stored in /var/run/gm_mapper/pid.{board_id}.

        The gm "stop" script performs the following operations:

          * Shuts down the gm_mapper daemon
          * ifconfig's down the myri* ethernet devices
          * Unloads the GM module (rem_drv)


        Important note:  Stopping the mapper while GM is running is
	not supported.  The gm_mapper should be left running at
	all times, and it will not interfere with the performance of
	jobs running over Myrinet.

        Important note:  The installation scripts do not configure the
	IP device.  If you wish to run IP over GM/Myrinet (ethernet 
	emulation), you must configure the device.  Refer to step 3.

	If you wish for the driver to auto-load an boot, you can create
	appropriate links in the /etc/rcN directories to the /etc/init.d/gm
	and /etc/init.d/myri scripts.  

	Alternatively, you may start and stop the drivers manually using

        su root
        /etc/init.d/gm start
        /etc/init.d/gm stop

	or

        su root
        /etc/init.d/gm restart

	to start, stop, or restart the driver, respectively.

	For directions on how to uninstall the GM driver, refer to the 
	"Miscellaneous" section.

        Note:  If the host is rebooted, you must reload the GM driver.
 

3. Enabling IP over Myrinet (Ethernet Emulation) (OPTIONAL)
-----------------------------------------------------------

   If you wish to run IP over Myrinet (ethernet emulation), the Solaris
   command to enable IP over GM is as follows:

   /sbin/ifconfig myri0 <ip_address> up

   where you must replace myri0 with the appropriate name (myri1, myri2,
   etc.) if you have more than one Myrinet interface per host.


4. Testing the GM Installation.
------------------------------
 
   Once the GM software has been properly installed on all of the
   hosts in your cluster, you are ready to validate your Myrinet
   installation by performing the following sequence of tests.

      * Check the LEDs on each switch port and interface port
      * Run gm_board_info on one host
      * Run gm_debug to test the PCI bandwidth
      * Run gm_allsize to test the links in the network
      * Run gm_stress to test the network

   Each of these steps is detailed in the Troubleshooting section of the FAQ
      http://www.myri.com/scs/FAQ/

   The test scripts (gm_board_info, gm_debug, gm_allsize, gm_stress) are
   available in <install_path>/bin in your GM installation. A README
   describing each of these tests can be found in <install_path>/bin/README.


=======================================
II. GM Source Installation Instructions
=======================================

The source release of gm-2.0.6 for Solaris is prebuilt with
new-features and heart-beat enabled.  

Building GM requires the Sun C++ compiler and Gnu make.

GM installation from the source distribution differs from that of the
binary installation in only step 1:
 
1. Configuring and compiling GM:
--------------------------------
 
        gunzip -c gm-2.0.6_Solaris.tar.gz | tar xvf -
        cd gm-2.0.6_Solaris
	autoconf
	autoheader
        ./configure

	Add the "--disable-64b" option if you are building a 32-bit
	driver on a machine running a 64-bit kernel, or "--enable-64b"
	if you are building a 64-bit driver on a machine running a 32-bit
	kernel.  Without these options, a driver matching the running kernel
	on the compilation machine will be built.

	Add the --disable-sparc-streaming configure option if you are
	building the driver for a UltraSPARC IIIi machine. This option
	will porovide better performance compared with the default option.

	make depend
        make
	cd binary

The remaining steps in the installation procedure are identical to the
GM-2 binary installation. Start with step 2, and follow the steps until 
the installation is complete.



=============================
III. Verifying GM Performance
=============================
 
   We recommend the following test to verify the GM performance.
 
       cd <install_path>/bin
       ./gm_debug --no-counters
 
   This gm_debug test displays the results of the hardware benchmark test of
   the PCI bus with the DMA engine of the Myrinet interface. The output of this
   command indicates the maximum sustained bandwidth that can be obtained from
   the PCI bus, and thus provides an upper bound on GM performance. A detailed
   description of this benchmark can be found in the FAQ
   (http://www.myri.com/scs/FAQ/).

    The output of this command also tells you if the Myrinet interface was
    correctly detected as 64-bit / 66 MHz, for example. If the interface was
    not correctly detected by the BIOS, you should suspect a riser card problem
    or a PCI slot problem.

    Performance graphs (http://www.myri.com/scs/solaris/index-2.0.html) for 
    GM-2 on Solaris are available. The performance measurements were obtained 
    by running gm_allsize tests for latency and bandwidth as described in the 
    FAQ entry ("What are the run-time options to gm_allsize?"). Refer to the 
    section entitled "GM Performance" in the <GM_source_path>/README for 
    complete details on expected GM performance.

=====================================
IV. IP support via Ethernet Emulation 
=====================================

The IP device is accessed via

/sbin/ifconfig myri0 plumb	(installs the driver)
/sbin/ifconfig myri0 <ipaddr> up (configures the driver)

where you must replace 'myri0' with the appropriate name (myri1, myri2, etc.)
if you have more than one Myrinet interface per host.  


================
V. Miscellaneous
================

------------------------------------
a.  Uninstallation of the GM driver
------------------------------------

The gm_install_drivers script generates the script /sbin/gm_uninstall_drivers,
which can be used to uninstall the drivers.
 
The GM_INSTALL script generates the script <install_path>/sbin/GM_UNINSTALL,
which can be used to uninstall GM.

