************************************************************************
*           Myricom GM networking software and documentation           *
*                 Copyright (c) 2003 by Myricom, Inc.                  *
* All rights reserved.  See the file `COPYING' for copyright notice.   *
************************************************************************

README-FreeBSD for gm-2.0

    Supported platforms:   4.1.1-RELEASE - 4.7-RELEASE for ia32 and alpha,
			   and 4.9-RELEASE for ia32 and alpha.
			   Note: 4.8-RELEASE requires a patch for
			         proper operation.  See section VI.c below.

    Supported interfaces: PCI64, PCI64A, PCI64B, PCI64C, PCIXD

                          Note:  gm-2.0 does not interoperate with gm-1.x.
			  A mixture of hosts with gm-1.x and gm-2.0 cannot
			  talk to each other.

                          If you have PCI32{A,B,C} interfaces, you will need
			  to upgrade your interface, or use a previous version
			  of GM (gm-1.2.3 for PCI32A and gm-1.5.2.1 for
			  PCI32B and PCI32C).

                          For installation instructions of an earlier GM 
			  version please refer to the respective  README 
			  and README-<arch> files.

Table of Contents:
-----------------

  I.  GM Binary Installation
      a.  Unpacking the GM driver
      b.  Installing the GM driver
      c.  Enabling IP over Myrinet (Ethernet emulation)
      d.  Testing the GM installation
 II.  GM Source Installation
      a.  Configuring and compiling GM
      b.  Installing the GM driver      
      c.  Enabling IP over Myrinet (Ethernet emulation)
      d.  Testing the GM installation
III.  Verifying the GM performance
 IV.  Improving IP Performance
  V.  Fork() and System() Support
 VI.  Caveats
      a.  Loading/unloading the GM driver
      b.  Difficulties to unload the GM module?
      c.  Running GM on FreeBSD 4.8-RELEASE
 VII. Miscellaneous
      a.  Uninstallation of the GM driver

************************************************************************
If difficulties are encountered, please consult the FAQ
         http://www.myri.com/scs/FAQ/
and all technical support questions should be directed to help@myri.com.
************************************************************************

==========================
I.  GM Binary Installation
==========================

GM Binary installation is supported for FreeBSD/x86 versions
4.1.1-RELEASE through 4.9-RELEASE.  If you're running an older or
newer version of FreeBSD (4.0 through 4.1-RELEASE, or 5.x), or if you
are running FreeBSD on an alpha, you must build the driver from
source.  Please skip to section II.( GM Source Installation
Instructions).

Please note that while the GM driver works with 5.1-CURRENT as of
August 20th 2003, the 5.x branch of FreeBSD is still under active
development.  Therefore interfaces may change between the date of the
gm-2.0_FreeBSD release and the time when FreeBSD 5.x becomes the
stable branch.  Consequently, we recommend Myrinet users who are
interesting in running FreeBSD choose FreeBSD 4.x.

GM installation is performed in the following four steps. 

1. Unpacking the GM driver. 
--------------------------------------

	tar zxvf gm-2.0_FreeBSD-x86-FreeBSD-4.1.1-RELEASE-16ports.tar.gz

2.  Installing the GM driver:
--------------------------------------------- 

        Select an installation directory path <install_path>.  It is
        usually best for <install_path> to be the path to an NFS
        directory available on all machines that are to share this GM
        installation.  The directory must be accessible using
        <install_path> on all machines that are to share the
        installation.  <install_path> must be an absolute path; it
        must start with "/".  However, <install_path> may contain
        symbolic links.

        cd gm-2.0_FreeBSD-x86-FreeBSD-4.1.1-RELEASE
        ./GM_INSTALL <install_path>

        If you omit <install_path>, the driver will be installed in
        the default directory, /opt/gm/.

	Next, you must run

        su root
        <install_path>/sbin/gm_install_drivers
        /usr/local/etc/rc.d/gm start

	on each machine to install the drivers on that machine.

        Important note:  The gm_install_drivers script does not configure
        the IP device.  If you wish to run IP over GM/Myrinet (ethernet
        emulation), you must configure the device.  Refer to step 3.

	If you wish for the driver to auto-load at boot, you must make
	the appropriate symlink in the /usr/local/etc/rc.d directory:
	
	su root
	ln -s /usr/local/etc/rc.d/gm /usr/local/etc/rc.d/gm.sh

	Alternatively, 	you may start and stop the drivers manually using:

        su root
        /usr/local/etc/rc.d/gm start
        /usr/local/etc/rc.d/gm stop

	or

        su root
        /usr/local/etc/rc.d/gm restart

	to start, stop, or restart the driver, respectively.

	For directions on how to uninstall the GM driver, refer to the
	"Miscellaneous" section.

        Note:  If the host is rebooted, you must reload the GM driver.


3. Enabling IP over Myrinet (Ethernet Emulation) (OPTIONAL)
-----------------------------------------------------------

   If you wish to run IP over Myrinet (ethernet emulation), the FreeBSD
   command to enable IP over GM is as follows:

    /sbin/ifconfig myri0 <IP_address> up

   where you must replace 'myri0' with the appropriate name (myri1, myri2, etc.)
   if you have more than one Myrinet interface per host.

   For suggestions on improving performance, please refer to section
   "IV. Improving IP Performance".

   Consult the "Running IP" section of the FAQ (http://www.myri.com/scs/FAQ/)
   for other related questions.


4. Testing the GM Installation
------------------------------

   Once the GM software has been properly installed on all of the
   hosts in your cluster, you are ready to validate your Myrinet  
   installation by performing the following sequence of tests.

      * Check the LEDs on each switch port and interface port
      * Run gm_board_info on one host
      * Run gm_debug to test the PCI bandwidth
      * Run gm_allsize to test the links in the network
      * Run gm_stress to test the network

   Each of these steps is detailed in the Troubleshooting section of the FAQ
     http://www.myri.com/scs/FAQ/

   The test scripts (gm_board_info, gm_debug, gm_allsize, gm_stress) are
   available in <install_path>/bin in your GM installation. A README
   describing each of these tests can be found in <install_path>/bin/README.

===========================
II.  GM Source Installation
===========================

GM installation is performed in the following four steps. 

1. Configuring and compiling GM:
---------------------------------------------

	Install GNU make from the /usr/ports/devel/gmake port or
	the gmake package.

        gunzip -c gm-2.0_FreeBSD.tar.gz | tar xvf -
	cd {GM_HOME}
        ./configure
	gmake

        By default, we assume that your FreeBSD kernel build directory is
        located in /sys/compile/GENERIC.  If your FreeBSD kernel build
        directory is not located in /sys/compile/GENERIC, you must configure
        with the following option:

	./configure --with-freebsd-kernel=<freebsd-source-dir>

        where <freebsd-source-dir> specifies the directory for the FreeBSD
        kernel source.  The kernel header files MUST match the running kernel
        exactly: not only should they both be from the same version, but they
        should also contain the same kernel configuration options.

        If you are running the binary "GENERIC" kernel, you must
        install the kernel sources, configure a "GENERIC" kernel:
         # cd /sys/<arch>/conf; config -g GENERIC
        and run "make depend" in the GENERIC kernel build directory
        (/sys/compile/GENERIC)
         # cd /sys/compile/GENERIC; make depend
      
        For a complete listing of all options to configure, type:

        ./configure --help

        Note:  Do not use the configure flag --enable-directcopy.  This
        flag is not a valid option to GM 2.0.  It will be re-enabled
        in a future release.

        The GM_INSTALL script will unload any existing GM device driver, 
        load the current device driver and create /dev/gm<X> device 
        i-nodes. It does not configure the IP device, nor does it set up
        any scripts to load the GM driver at boot time.  During the
        GM_INSTALL phase, GM prints messages to the kernel log (dmesg).
        If the running kernel and the kernel header used for compilation
        are mismatched, GM will print a warning message to the kernel
        log.

        Please be sure to read the {GM_HOME}/README and the
        {GM_HOME}/README-FreeBSD for further details of 
        operating-system-specific caveats.

2.  Installing the GM driver:
--------------------------------------------- 

        Select an installation directory path <install_path>.  It is
        usually best for <install_path> to be the path to an NFS
        directory available on all machines that are to share this GM
        installation.  The directory must be accessible using
        <install_path> on all machines that are to share the
        installation.  <install_path> must be an absolute path; it
        must start with "/".  However, <install_path> may contain
        symbolic links.

        cd binary
        ./GM_INSTALL <install_path>

        If you omit <install_path>, the driver will be installed in
        the default directory, /opt/gm/.

	Next, you must run

        su root
        <install_path>/sbin/gm_install_drivers
        /usr/local/etc/rc.d/gm start

	on each machine to install the drivers on that machine.

	If you wish for the driver to auto-load at boot, you must make
	the appropriate symlink in the /usr/local/etc/rc.d directory:
	
	su root
	ln -s /usr/local/etc/rc.d/gm /usr/local/etc/rc.d/gm.sh

	Alternatively, 	you may start and stop the drivers manually using:

        su root
        /usr/local/etc/rc.d/gm start
        /usr/local/etc/rc.d/gm stop

	or

        su root
        /usr/local/etc/rc.d/gm restart

	to start, stop, or restart the driver, respectively.

	For directions on how to uninstall the GM driver, refer to the
	"Miscellaneous" section.

        Note:  If the host is rebooted, you must reload the GM driver.


3. Enabling IP over Myrinet (Ethernet Emulation) (OPTIONAL)
-----------------------------------------------------------

   If you wish to run IP over Myrinet (ethernet emulation), the FreeBSD
   command to enable IP over GM is as follows:

    /sbin/ifconfig myri0 <IP_address> up

   where you must replace 'myri0' with the appropriate name (myri1, myri2, etc.)
   if you have more than one Myrinet interface per host.

   For suggestions on improving performance, please refer to section
   "IV. Improving IP Performance".

   Consult the "Running IP" section of the FAQ (http://www.myri.com/scs/FAQ/)
   for other related questions.

4. Testing the GM Installation
------------------------------

   Once the GM software has been properly installed on all of the
   hosts in your cluster, you are ready to validate your Myrinet  
   installation by performing the following sequence of tests.

      * Check the LEDs on each switch port and interface port
      * Run gm_board_info on one host
      * Run gm_debug to test the PCI bandwidth
      * Run gm_allsize to test the links in the network
      * Run gm_stress to test the network

   Each of these steps is detailed in the Troubleshooting section of the FAQ
     http://www.myri.com/scs/FAQ/

   The test scripts (gm_board_info, gm_debug, gm_allsize, gm_stress) are
   available in <install_path>/bin in your GM installation. A README
   describing each of these tests can be found in <install_path>/bin/README.

=================================
III. Verifying the GM Performance
=================================

   We recommend the following test to verify your GM performance.

       cd <install_path>/bin
       ./gm_debug --no-counters

   This gm_debug test displays the results of the hardware benchmark test of
   the PCI bus with the DMA engine of the Myrinet interface. The output of this
   command indicates the maximum sustained bandwidth that can be obtained from
   the PCI bus, and thus provides an upper bound on GM performance. A detailed
   description of this benchmark can be found in the FAQ
   (http://www.myri.com/scs/FAQ/).

   The output of this command also tells you if the Myrinet interface was
   correctly detected as 64-bit / 66 MHz, for example. If the interface was
   not correctly detected by the BIOS, you should suspect a riser card problem
   or a PCI slot problem.

   Performance graphs (http://www.myri.com/myrinet/performance) for GM on
   Linux are available. The performance measurements were obtained by running
   gm_allsize tests for latency and bandwidth as described in the FAQ entry
   ("What are the run-time options to gm_allsize?"). Refer to the section 
   entitled "GM Performance" in the <GM_source_path>/README for complete 
   details on expected GM performance.
 
============================
IV. Improving IP performance
============================

IP tuning consists of the following:

sudo sysctl -w net.inet.tcp.sendspace=65536
sudo sysctl -w net.inet.tcp.recvspace=65536

Some additional gain can sometimes be seen by increasing these values
to 262144.  If one increases these values to 262144, one must also increase
kern.ipc.maxsockbuf to something larger, such as 524288.

If you would like to obtain the maximum performance out of IP,
you can increase the size of your mbuf clusters to PAGE_SIZE by changing
MCLSHIFT in sys/{i386,alpha}/include/param.h to 12 on i386 and 13 on alpha.
(eg, PAGE_SIZE == 1 << MCLSHIFT).  This requires a kernel rebuild and a GM
rebuild. 

===============================
V.  Fork() and System() Support
===============================

By default, GM supports the function vfork().  As the function popen()
calls vfork(), it is also safe to use at any time with GM.

However, GM does not support fork() when a GM port is open, as it is 
incompatible with some GM OS manipulations.

Please contact help@myri.com for further details.

============
VI.  Caveats
============

-----------------------------------
a.  Loading/unloading the GM driver
-----------------------------------

To load the driver, run 'kldload <install_path>/sbin/gm.ko'
To unload the driver, run 'kldunload -n gm.ko'

-----------------------------------------
b.  Difficulties to unload the GM module?
-----------------------------------------

>>Let's say a user left a gm_allsize running and forgot about it.  She then
>>attempts to unload the module, and receives the following message:

% sudo kldunload gm
kldunload: can't unload file: Device busy

>> At this point, this message will appear on the system console:


WARNING: GM: 1 refs to minor unit 8. module cannot be unloaded until all ports are closed

>> Now the user has enough information to track down the process &
>> kill it.  The easiest way to do this is to use fstat (like lsof,
>> but in base system) to look for processes with /dev/gm* open:


% fstat | grep gm
root     gm_allsize       601    3 /          8484 crw-------     gm8 rw
% sudo kill -TERM 601
% sudo kldunload gm
%

------------------------------------
c.  Running GM on FreeBSD 4.8-RELEASE
------------------------------------

FreeBSD 4.8 contains a minor bug in the VM system which will cause our
driver to leak wired memory when GM applications exit uncleanly.  This
can happen if you ^C an MPICH-GM job.

This bug was introduced in version 1.187.2.16 of sys/vm/vm_map.c prior
to 4.8-RELEASE, and was corrected in version 1.187.2.21 of
sys/vm/vm_map.c, prior to 4.9-RELEASE.

To work around this problem, you can apply the patch below.
Alternatively, we suggest that you upgrade to FreeBSD 4.9 if possible.

Index: vm/vm_map.c
===================================================================
RCS file: /home/ncvs/src/sys/vm/vm_map.c,v
retrieving revision 1.187.2.20
retrieving revision 1.187.2.21
diff -u -r1.187.2.20 -r1.187.2.21
--- vm/vm_map.c	10 Aug 2003 18:14:11 -0000	1.187.2.20
+++ vm/vm_map.c	1 Oct 2003 01:21:37 -0000	1.187.2.21
@@ -1659,6 +1659,7 @@
 		 * becomes completely unwired, unwire its physical pages and
 		 * mappings.
 		 */
+		entry = start_entry;
 		while ((entry != &map->header) && (entry->start < end)) {
 			KASSERT(entry->eflags & MAP_ENTRY_USER_WIRED, ("expected USER_WIRED on entry %p", entry));
 			entry->eflags &= ~MAP_ENTRY_USER_WIRED;

==================
VII. Miscellaneous
==================

------------------------------------
a.  Uninstallation of the GM driver
------------------------------------

The gm_install_drivers script generates the script /sbin/gm_uninstall_drivers,
which can be used to uninstall the drivers.

The GM_INSTALL script generates the script <install_path>/sbin/GM_UNINSTALL,
which can be used to uninstall GM.

