Sun Cluster 3.1 10/03 Data Services Developer's Guide

Evaluating Restart Versus Failover

If the probefail variable is something other than 0 (success), it means the nslookup command timed out or that the reply came from a server other than the sample service's DNS. In either case, the DNS server is not functioning as expected and the fault monitor calls the decide_restart_or_failover() function to determine whether to restart the data service locally or request that the RGM relocate the data service to a different node. If the probefail variable is 0, then a message is generated that the probe was successful.
if [ $probefail -ne 0 ]; then decide_restart_or_failover else logger -p ${SYSLOG_FACILITY}.err\ -t [$SYSLOG_TAG]\ "${ARGV0} Probe for resource HA-DNS successful" fi

The decide_restart_or_failover() function uses a time window (Retry_interval) and a failure count (Retry_count) to determine whether to restart DNS locally or request that the RGM relocate the data service to a different node. It implements the following conditional code (see the code listing for decide_restart_or_failover() in PROBE Program).

If this is the first failure, restart the data service. Log an error message and bump the counter in the retries variable.
If this is not the first failure, but the window has been exceeded, restart the data service. Log an error message, reset the counter, and slide the window.
If the time is still within the window and the retry counter has been exceeded, then fail over to another node. If the fail over does not succeed, log an error and exit the probe program with status 1 (failure).
If time is still within the window but the retry counter has not been exceeded, restart the data service. Log an error message and bump the counter in the retries variable.

If the number of restarts reaches the limit during the time interval, the function requests that the RGM relocate the data service to a different node. If the number of restarts is under the limit, or the interval has been exceeded so the count begins again, the function attempts to restart DNS on the same node. Note the following about this function:

The gettime utility is used to track the time between restarts. This is a C program residing in the (Rt_basedir) directory.
The Retry_count and Retry_interval system-defined resource properties determine the number of restart attempts and the interval over which to count. These properties default to 2 attempts in a period of 5 minutes (300 seconds) in the RTR file, though the cluster administrator could change them.
The restart_service() function is called to attempt to restart the data service on the same node. See the next section, Restarting the Data Service, for information about this function.
The scha_control() API function, with the GIVEOVER option, brings the resource group containing the sample data service offline and back online on a different node.

Restarting the Data Service

The restart_service() function is called by decide_restart_or_failover() to attempt to restart the data service on the same node. This function does the following.

It determines if the data service is still registered under PMF. If the service is still registered, the function:
- Obtains the Stop method name and the Stop_timeout value for the data service.
- Uses hatimerun to launch the Stop method for the data service, passing the Stop_timeout value.
- (If the data service is successfully stopped) obtains the Start method name and the Start_timeout value for the data service.
- Uses hatimerun to launch the Start method for the data service, passing the Start_timeout value.
If the data service is no longer registered under PMF, the implication is that the data service has exceeded the maximum number of allowable retries under PMF, so the scha_control() function is called with the GIVEOVER option to fail the data service over to a different node.

function restart_service
{

        # To restart the data service, first verify that the 
        # data service itself is still registered under PMF.
        pmfadm -q $PMF_TAG
        if [[ $? -eq 0 ]]; then
                # Since the TAG for the data service is still registered under
                # PMF, first stop the data service and start it back up again.

                # Obtain the Stop method name and the STOP_TIMEOUT value for
                # this resource.
                STOP_TIMEOUT=`scha_resource_get -O STOP_TIMEOUT \
                        -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ
                STOP_METHOD=`scha_resource_get -O STOP \
                        -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ
                hatimerun -t $STOP_TIMEOUT $RT_BASEDIR/$STOP_METHOD \
                        -R $RESOURCE_NAME -G $RESOURCEGROUP_NAME \
                        -T $RESOURCETYPE_NAME

                if [[ $? -ne 0 ]]; then
                        logger-p ${SYSLOG_FACILITY}.err -t [$SYSLOG_TAG] \
                                "${ARGV0} Stop method failed."
                        return 1
                fi

                # Obtain the START method name and the START_TIMEOUT value for
                # this resource.
                START_TIMEOUT=`scha_resource_get -O START_TIMEOUT \
                        -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ
                START_METHOD=`scha_resource_get -O START \
                        -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ
                hatimerun -t $START_TIMEOUT $RT_BASEDIR/$START_METHOD \
                        -R $RESOURCE_NAME -G $RESOURCEGROUP_NAME \
                        -T $RESOURCETYPE_NAME

                if [[ $? -ne 0 ]]; then
                        logger-p ${SYSLOG_FACILITY}.err -t [$SYSLOG_TAG] \
                                "${ARGV0} Start method failed."
                        return 1
                fi


        else
                # The absence of the TAG for the dataservice 
                # implies that the data service has already
                # exceeded the maximum retries allowed under PMF. 
                # Therefore, do not attempt to restart the
                # data service again, but try to failover
                # to another node in the cluster.
                scha_control -O GIVEOVER -G $RESOURCEGROUP_NAME \
                        -R $RESOURCE_NAME
        fi

        return 0
}


5. Sample Data Service Defining a Fault Monitor Probe Program