************************************************************************
*           Myricom GM networking software and documentation           *
*                 Copyright (c) 2001, 2002 by Myricom, Inc.            *
* All rights reserved.  See the file `COPYING' for copyright notice.   *
************************************************************************

README-FreeBSD for gm-1.6.3

    Supported platforms:   4.x and 5.x for ia32 and alpha.

    Supported interfaces:  LANai7 (PCI64, PCI64A), and LANai9 (PCI64B, PCI64C)
 
                          If you have LANai4, you will need to upgrade your
                          interface, or use a previous version of GM 
			  such as gm-1.2.3 for 256K and gm-1.5.2 for larger 
			  memory sizes.

                          For installation instructions of an earlier GM 
			  version please refer to the respective  README 
			  and README-<arch> files.

Table of Contents:
-----------------

  I.  GM Binary Installation
      a.  Unpacking the GM driver
      b.  Installing the GM driver
      b.  Running the GM Mapper
      c.  Testing the GM installation
 II.  GM Source Installation
      a.  Configuring and compiling GM
      b.  Installing the GM driver      
      c.  Running the GM Mapper
      d.  Testing the GM installation
III.  Verifying the GM performance
 IV.  Running IP over GM
  V.  Improving IP Performance
 VI.  Fork() and System() Support
VII.  Caveats
      a.  Loading/unloading the GM driver
      b.  Difficulties to unload the GM module?
VIII. Miscellaneous
      a.  Uninstallation of the GM driver

************************************************************************
If difficulties are encountered, please consult the FAQ
         http://www.myri.com/scs/GM_FAQ.html
and all technical support questions should be directed to help@myri.com.
************************************************************************

==========================
I.  GM Binary Installation
==========================

GM Binary installation is supported for FreeBSD/x86 versions
4.1.1-RELEASE through 4.7-RELEASE.  If you're running an older or
newer version of FreeBSD (4.0 through 4.1-RELEASE, or 5.x), or if you
are running FreeBSD on an alpha, you must build the driver from
source.  Please skip to section II.( GM Source Installation
Instructions).

Current cautions and common problems:
 
   * If the host is rebooted, you must reload the GM driver (and rerun the
     GM Mapper).
   * The GM Mapper must be run before any communication over Myrinet can occur.
   * In most cases, the GM Mapper should not be run continuously. It
     is intrusive and can interfere with network traffic intensive jobs
     (e.g, MPI applications).

GM installation is performed in the following three steps. 

1. Unpacking the GM driver. 
--------------------------------------

	tar zxvf gm-1.6.3_FreeBSD-x86-FreeBSD-4.1.1-RELEASE-new-features-8ports.tar.gz

2.  Installing the GM driver:
--------------------------------------------- 

        Select an installation directory path <install_path>.  It is
        usually best for <install_path> to be the path to an NFS
        directory available on all machines that are to share this GM
        installation.  The directory must be accessible using
        <install_path> on all machines that are to share the
        installation.  <install_path> must be an absolute path; it
        must start with "/".  However, <install_path> may contain
        symbolic links.

        cd gm-1.6.3_FreeBSD-x86-FreeBSD-4.1.1-RELEASE
        ./GM_INSTALL <install_dir>

        You may omit <install_dir> to install the driver in /opt/gm/.

	Next, you must run

        su root
        <install_path>/sbin/gm_install_drivers

	on each machine to install the drivers on that machine.

	If you wish for the driver to auto-load an boot, you must make
	the appropriate symlink in the /usr/local/etc/rc.d directory:
	
	su root
	ln -s /usr/local/etc/rc.d/gm /usr/local/etc/rc.d/gm.sh

	Alternatively, 	you may start and stop the drivers manually using:

        su root
        /usr/local/etc/rc.d/gm start
        /usr/local/etc/rc.d/gm stop

	or

        su root
        /usr/local/etc/rc.d/gm restart

	to start, stop, or restart the driver, respectively.

	For directions on how to uninstall the GM driver, refer to the
	"Miscellaneous" section.

        Note:  If the host is rebooted, you must reload the GM driver (and
               rerun the GM mapper).


3. Running the GM Mapper
------------------------

	cd {GM_HOME}/binary/sbin/
        su root
        ./mapper ../etc/gm/map_once.args
 
There are 3 ways in which the GM Mapper can be run. Refer to section 2
'Running the Mapper' of the README-linux for details.

4. Testing the GM Installation
------------------------------

   A variety of test scripts are available in {GM_HOME}/binary/bin to test your
   GM installation. We recommend the five tests described in tests/README.


===========================
II.  GM Source Installation
===========================

Current cautions and common problems:
 
   * If the host is rebooted, you must reload the GM driver (and rerun the
     GM Mapper).
   * The GM Mapper must be run before any communication over Myrinet can occur.
   * In most cases, the GM Mapper should not be run continuously. It
     is intrusive and can interfere with network traffic intensive jobs
     (e.g, MPI applications).

GM installation is performed in the following three steps. 

1. Configuring and compiling GM:
---------------------------------------------

	Install GNU make from the /usr/ports/devel/gmake port or
	the gmake package.

        gunzip -c gm-1.6.3_FreeBSD.tar.gz | tar xvf -
	cd {GM_HOME}
        ./configure
	gmake

        By default, we assume that your FreeBSD kernel build directory is
        located in /sys/compile/GENERIC.  If your FreeBSD kernel build
        directory is not located in /sys/compile/GENERIC, you must configure
        with the following option:

	./configure --with-freebsd-kernel=<freebsd-source-dir>

        where <freebsd-source-dir> specifies the directory for the FreeBSD
        kernel source.  The kernel header files MUST match the running kernel
        exactly: not only should they both be from the same version, but they
        should also contain the same kernel configuration options.

        If you are running the binary "GENERIC" kernel, you must
        install the kernel sources, configure a "GENERIC" kernel:
         # cd /sys/<arch>/conf; config -g GENERIC
        and run "make depend" in the GENERIC kernel build directory
        (/sys/compile/GENERIC)
         # cd /sys/compile/GENERIC; make depend
      
        Note:  If you have a mixture of hosts with LANai4 and LANai7
        (or LANai9) interfaces that need to talk to each other, you
        must configure with --disable-new-features on all of the hosts.

        For a complete listing of all options to configure, type:

        ./configure --help

        Note:  Do not use the configure flag --enable-directcopy.  This
        flag is not a valid option to GM 1.5.  It will be re-enabled
        in a future release.

        The GM_INSTALL script will unload any existing GM device driver, 
        load the current device driver and create /dev/gm<X> device 
        i-nodes. It does not configure the IP device, nor does it set up
        any scripts to load the GM driver at boot time.  During the
        GM_INSTALL phase, GM prints messages to the kernel log (dmesg).
        If the running kernel and the kernel header used for compilation
        are mismatched, GM will print a warning message to the kernel
        log.

        Please be sure to read the {GM_HOME}/README and the
        {GM_HOME}/README-FreeBSD for further details of 
        operating-system-specific caveats.

2.  Installing the GM driver:
--------------------------------------------- 

        Select an installation directory path <install_path>.  It is
        usually best for <install_path> to be the path to an NFS
        directory available on all machines that are to share this GM
        installation.  The directory must be accessible using
        <install_path> on all machines that are to share the
        installation.  <install_path> must be an absolute path; it
        must start with "/".  However, <install_path> may contain
        symbolic links.

        cd binary
        ./GM_INSTALL <install_dir>

        You may omit <install_dir> to install the driver in /opt/gm/.

	Next, you must run

        su root
        <install_path>/sbin/gm_install_drivers

	on each machine to install the drivers on that machine.

	If you wish for the driver to auto-load an boot, you must make
	the appropriate symlink in the /usr/local/etc/rc.d directory:
	
	su root
	ln -s /usr/local/etc/rc.d/gm /usr/local/etc/rc.d/gm.sh

	Alternatively, 	you may start and stop the drivers manually using:

        su root
        /usr/local/etc/rc.d/gm start
        /usr/local/etc/rc.d/gm stop

	or

        su root
        /usr/local/etc/rc.d/gm restart

	to start, stop, or restart the driver, respectively.

	For directions on how to uninstall the GM driver, refer to the
	"Miscellaneous" section.

        Note:  If the host is rebooted, you must reload the GM driver (and
               rerun the GM mapper).


3. Running the GM Mapper
------------------------

        cd {GM_HOME}/binary/sbin
        su root
        ./mapper ../etc/gm/map_once.args
 
There are 3 ways in which the GM Mapper can be run. Refer to section 2
'Running the Mapper' of the README-linux for details.

4. Testing the GM Installation
------------------------------

   A variety of test scripts are available in {GM_HOME}/binary/bin to test your
   GM installation. We recommend the five tests described in tests/README.

=================================
III. Verifying the GM Performance
=================================

   We recommend the following test to verify the GM performance.  View
   the results of the hardware benchmark test of the PCI bus with the 
   DMA engine of the Myrinet adapter.

       cd {GM_HOME}/binary/bin
       ./gm_debug --no-counters

   Note: The output of this command gives the maximum sustained bandwidth
         that can be obtained from the PCI bus.  Refer to the section
         entitled "GM Performance" in the {GM_HOME}/README for complete 
         details on expected GM performance.

 
======================
IV. Running IP over GM
======================

The FreeBSD command to enable IP over GM is as follows:

    /sbin/ifconfig myri0 <IP_address> up

where you must replace 'myri0' with the appropriate name (myri1, myr2, etc.)
if you have more than one Myrinet interface per host.  For more information,
please refer to the FAQ (http://www.myri.com/scs/GM_FAQ.html). 


===========================
V. Improving IP performance
===========================

IP tuning consists of the following:

sudo sysctl -w net.inet.tcp.sendspace=65536
sudo sysctl -w net.inet.tcp.recvspace=65536

Some additional gain can sometimes be seen by increasing these values
to 262144.  If one increases these values to 262144, one must also increase
kern.ipc.maxsockbuf to something larger, such as 524288.

If you would like to obtain the maximum performance out of IP,
you can increase the size of your mbuf clusters to PAGE_SIZE by changing
MCLSHIFT in sys/{i386,alpha}/include/param.h to 12 on i386 and 13 on alpha.
(eg, PAGE_SIZE == 1 << MCLSHIFT).  This requires a kernel rebuild and a GM
rebuild. 

================================
VI.  Fork() and System() Support
================================

By default, GM supports the function vfork().  As the function popen()
calls vfork(), it is also safe to use at any time with GM.

However, GM does not support fork() when a GM port is open, as it is 
incompatible with some GM OS manipulations.

Please contact help@myri.com for further details.

=============
VII.  Caveats
=============

-----------------------------------
a.  Loading/unloading the GM driver
-----------------------------------

To load the driver, run 'kldload binary/sbin/gm.ko'
To unload the driver, run 'kldunload -n gm.ko'

-----------------------------------------
b.  Difficulties to unload the GM module?
-----------------------------------------

>>Let's say a user left a mapper running and forgot about it.  She then
>>attempts to unload the module, and receives the following message:

% sudo kldunload gm
kldunload: can't unload file: Device busy

>> At this point, this message will appear on the system console:


WARNING: GM: 1 refs to minor unit 8. module cannot be unloaded until all ports are closed

>> Now the user has enough information to track down the process &
>> kill it.  The easiest way to do this is to use fstat (like lsof,
>> but in base system) to look for processes with /dev/gm* open:


% fstat | grep gm
root     mapper       601    3 /          8484 crw-------     gm8 rw
% sudo kill -TERM 601
% sudo kldunload gm
%

===================
VIII. Miscellaneous
===================

------------------------------------
a.  Uninstallation of the GM driver
------------------------------------

The gm_install_drivers script generates the script /sbin/gm_uninstall_drivers,
which can be used to uninstall the drivers.

The GM_INSTALL script generates the script <install_path>/sbin/GM_UNINSTALL,
which can be used to uninstall GM.
