************************************************************************
*           Myricom GM networking software and documentation           *
*                 Copyright (c) 2002 by Myricom, Inc.                  *
* All rights reserved.  See the file `COPYING' for copyright notice.   *
************************************************************************
 
README-MacOSX for gm-1.6.3

    Supported platforms:  Mac OS X 10.1 and 10.2 (darwin 1.4 - 6.0) for PowerPC

    Supported interfaces:  LANai7 (PCI64, PCI64A), and LANai9 (PCI64B, PCI64C)
 
Table of Contents:
-----------------
 
  I.  GM Binary Installation
      a.  Unpacking the GM driver
      b.  Installing the GM driver
      b.  Running the GM Mapper
      c.  Testing the GM installation
 II.  GM Source Installation
      a.  Configuring and compiling GM
      b.  Installing the GM driver      
      c.  Running the GM Mapper
      d.  Testing the GM installation
III.  Verifying the GM performance
 IV.  Running IP over GM
  V.  Improving IP Performance
 VI.  Cautions/Caveats
VII.  Miscellaneous
      a.  Uninstallation of the GM driver 
************************************************************************
If difficulties are encountered, please consult the FAQ
         http://www.myri.com/scs/GM_FAQ.html
and all technical support questions should be directed to help@myri.com.

If you have problems on any architecture, please send the output from
/var/log/system.log to <help@myri.com>.
************************************************************************

======================================
I. GM Binary Installation Instructions
======================================

Current cautions and common problems:

   * If the host is rebooted, you must rerun the GM Mapper. 
   * The GM Mapper must be run before any communication over Myrinet can occur. 
   * In most cases, the GM Mapper should not be run continuously. It
     is intrusive and can interfere with network traffic intensive jobs
     (e.g, MPI applications). 
   * Incorrectly written programs which attempt to free() or munmap() 
     memory prior to  deregistering it may "hang" indefinitely.  
     For details see the Cautions/Caveats below

GM installation is performed in three easy steps:

1. Unpacking the GM driver. 
--------------------------------------

When reading the instructions below, please replace VERSION with
your operating system version (10.1 or 10.2):

        tar zxvf gm-1.6.3_MacOSX-VERSION-new-features-8ports.tar.gz


2.  Installing the GM driver:
--------------------------------------------- 

        Select an installation directory path <install_path>.  It is
        usually best for <install_path> to be the path to an NFS
        directory available on all machines that are to share this GM
        installation.  The directory must be accessible using
        <install_path> on all machines that are to share the
        installation.  <install_path> must be an absolute path; it
        must start with "/".  However, <install_path> may contain
        symbolic links.

        cd gm-1.6.3_MacOSX-VERSION-new-features-8ports
        ./GM_INSTALL <install_dir>

        You may omit <install_dir> to install the driver in /opt/gm/.

	Next, you must run

	sudo <install_path>/sbin/gm_install_drivers

	on each machine to install the drivers on that machine.

	If you wish for the driver to auto-load an boot, you must copy
	StartupParameters.plist to /System/Library/StartupItems/GM
	
	sudo cp <install_path>/etc/StartupParameters.plist /System/Library/StartupItems/GM/

	Alternatively, 	you may start and stop the drivers manually using:

	sudo /System/Library/StartupItems/GM/gm start
	sudo /System/Library/StartupItems/GM/gm stop

	or

	sudo /System/Library/StartupItems/GM/gm restart

	to start, stop, or restart the driver, respectively.

	For directions on how to uninstall the GM driver, refer to the
	"Miscellaneous" section.

        Note:  If the host is rebooted, you must reload the GM driver (and
               rerun the GM mapper).



3. Running the GM Mapper.
------------------------

        cd {GM_HOME}/binary/sbin
        sudo ./mapper ../etc/gm/map_once.args

There are 3 ways in which the GM Mapper can be run. Refer to section 2
'Running the Mapper' of the README-linux for details.

3. Testing the GM Installation.
------------------------------

A variety of test scripts are available in binary/bin/ to test your GM
installation. We recommend the five tests described in tests/README.

=======================================
II. GM Source Installation Instructions 
=======================================

To compile GM on Mac OS X platforms, you will need GNU make, and the
Apple C/C++ compiler.  These tools are provided by Apple as part of
the Developer Tools package available through the Apple Developer
Connection (http://www.apple.com/developer).  The Developer Tools
package is also included in the Mac OS X 10.2 retail box.

Current cautions and common problems:

   * If the host is rebooted, you must rerun the GM Mapper. 
   * The GM Mapper must be run before any communication over Myrinet can occur. 
   * In most cases, the GM Mapper should not be run continuously. It
     is intrusive and can interfere with network traffic intensive jobs
     (e.g, MPI applications). 
   * Incorrectly written programs which attempt to free() or munmap() 
     memory prior to  deregistering it may "hang" indefinitely.  
     For details see the Cautions/Caveats below.

GM installation is performed in three easy steps:

1. Configuring and compiling the GM driver:
----------------------------------------------------

        tar zxvf gm-1.6.3_MacOSX.tar.gz
        cd gm-1.6.3_MacOSX
        ./configure
        make

2.  Installing the GM driver:
--------------------------------------------- 

        Select an installation directory path <install_path>.  It is
        usually best for <install_path> to be the path to an NFS
        directory available on all machines that are to share this GM
        installation.  The directory must be accessible using
        <install_path> on all machines that are to share the
        installation.  <install_path> must be an absolute path; it
        must start with "/".  However, <install_path> may contain
        symbolic links.

        cd binary
        ./GM_INSTALL <install_dir>

        You may omit <install_dir> to install the driver in /opt/gm/.

	Next, you must run

	sudo <install_path>/sbin/gm_install_drivers

	on each machine to install the drivers on that machine.

	If you wish for the driver to auto-load an boot, you must copy
	StartupParameters.plist to /System/Library/StartupItems/GM
	
	sudo cp <install_path>/etc/StartupParameters.plist /System/Library/StartupItems/GM/

	Alternatively, 	you may start and stop the drivers manually using:

	sudo /System/Library/StartupItems/GM/gm start
	sudo /System/Library/StartupItems/GM/gm stop

	or

	sudo /System/Library/StartupItems/GM/gm restart

	to start, stop, or restart the driver, respectively.

	For directions on how to uninstall the GM driver, refer to the
	"Miscellaneous" section.

        Note:  If the host is rebooted, you must reload the GM driver (and
               rerun the GM mapper).


3. Running the GM Mapper.
------------------------

        cd {GM_HOME}/binary/sbin
        sudo ./mapper ../etc/gm/map_once.args

There are 3 ways in which the GM Mapper can be run. Refer to section 2
'Running the Mapper' of the README-linux for details.

3. Testing the GM Installation.
------------------------------

A variety of test scripts are available in binary/bin/ to test your GM
installation. We recommend the five tests described in tests/README.

=================================
III. Verifying the GM Performance
=================================
 
   We recommend the following test to verify the GM performance.  View
   the results of the hardware benchmark test of the PCI bus with the
   DMA engine of the Myrinet adapter.
 
       cd {GM_HOME}/binary/bin
       ./gm_debug --no-counters
 
   Note: The output of this command gives the maximum sustained bandwidth
         that can be obtained from the PCI bus.  Refer to the section
         entitled "GM Performance" in the {GM_HOME}/README for complete
         details on expected GM performance.

======================
IV. Running IP over GM
======================

The IP device is accessed via
        ifconfig enX <address> netmask <netmask> broadcast <bcast> up

where you must replace 'enX with the appropriate name (en1, en2, etc.)
Note that this is different than GM on other platforms, where the
device name is myriX.  This is due to quirks in the Mac OS X network
stack, over which we have no control.

The first ethernet device number used by GM is obtained via sysctl:
% sysctl net.gm.gm_base_en_unit
net.gm.gm_base_en_unit: 1

This means that the first myrinet interface on this host is named
"en1", and subsequent interfaces will be named "en2", "en3", etc.

Note that the GM network interface is not visible or configurable
through the Network control panel.  You must use ifconfig to configure
it.
 
===========================
V. Improving IP performance
===========================

If you don't see the speed that you expect, you might consider adjusting
the TCP send and receive window sizes.

To adjust the window sizes, modify the kern.ipc.maxsockbuf,
net.inet.tcp.sendspace and net.inet.tcp.recvspace like this:

% sudo sysctl -w kern.ipc.maxsockbuf=2097152
% sudo sysctl -w net.inet.tcp.sendspace=262144
% sudo sysctl -w net.inet.tcp.recvspace=262144

====================
VI. Cautions/Caveats
====================

*************************************************************************

You need to run the mapper at least once after loading the driver
on all nodes in your network.
cd {GM_HOME}/binary/sbin
sudo ./mapper ../etc/gm/map_once.args

COMPILING from SOURCE:
---------------------

You may not use the --enable-maintainer-mode flag on Mac OS X.  This is
because the lanai cross compiler contains a bug which causes it to
crash when run on PowerPC.  You must use the prebuilt firmware, or
build your firmware on another platform.


INCORRECTLY WRITTEN PROGRAMS "HANG"
-----------------------------------

Incorrectly written programs which attempt to free() or munmap() 
memory prior to  deregistering it may "hang" indefinitely.  

This happens because the MacOSX/darwin kernel does not expect drivers
to leave memory regions "wired" in memory for long periods of time.
When the memory region is free'ed, the VM system blocks
uninterruptibly waiting for the GM driver to unwire the memory.  The
GM driver has no way of knowing that the memory system is waiting for it
to unwire a memory region.

To recover from this problem, you can unload & reload the module (cd
binary; sudo ./GM_INSTALL).  At unload time, the GM driver will free
all memory associated with all ports and will send kill -9 signals to
all processes with open ports.  This will allow the hung program to
exit.

To diagnose this problem, run the program under sc_usage.  If the last
system call the program made before "hanging" was vm_deallocate or
munmap, then your application has a bug which must fixed before it can
run on MacOSX/darwin.  See the sc_usage(1) man page for instructions
on how to run sc_usage.

Ethernet Unit Number "Stealing"
------------------------------

Due to a quirk in the way the GM driver interacts with the
MacOSX/darwin network stack, users planning to use ethernet emulation
must load the GM module *after* loading all other network drivers.  If
a network driver is loaded after GM, the newly loaded network driver
has no way of knowing what unit number the GM ethernet (en) interface
is using and it may "steal" the GM unit number.  This could make one or
both devices unusable until the machine is rebooted.

DARWIN on INTEL
---------------

Darwin on Intel has not been tested and is not supported.  We would
not be terribly surprised to hear that it worked, but we offer no
support.


*************************************************************************
===================
VIII. Miscellaneous
===================

------------------------------------
a.  Uninstallation of the GM driver
------------------------------------

The gm_install_drivers script generates the script /sbin/gm_uninstall_drivers,
which can be used to uninstall the drivers.

The GM_INSTALL script generates the script <install_path>/sbin/GM_UNINSTALL,
which can be used to uninstall GM.
