Providing Process Management
The RMAPI and the DSDL provide process management facilities to implement resource monitors and resource control callbacks. The RMAPI defines the following facilities (see the man pages for details about each of these commands and programs):
The DSDL provides the scds_hatimerun function to implement the hatimerun functionality.
The DSDL provides a set of functions (scds_pmf_*) to implement the PMF functionality. See PMF Functions for an overview of the DSDL PMF functionality and for a list of the individual functions.
Providing Administrative Support for a Resource
Administrative actions on resources include setting and changing resource properties. The API defines the Validate and Update callback methods so you can hook into these administrative actions.
The RGM calls the optional Validate method when a resource is created and when administrative action updates the properties of the resource or its containing group. The RGM passes the property values for the resource and its resource group to the Validate method. The RGM calls Validate on the set of cluster nodes indicated by the Init_nodes property of the resource's type (see Resource Type Properties, or the rt_properties(5) man page, for information about Init_nodes. The RGM calls Validate before the creation or update is applied, and a failure exit code from the method on any node causes the creation or update to fail.
The RGM calls Validate only when resource or group properties are changed through administrative action, not when the RGM sets properties, or when a monitor sets the resource properties Status and Status_msg.
The RGM calls the optional Update method to notify a running resource that properties have been changed. The RGM invokes Update after an administrative action succeeds in setting properties of a resource or its group. The RGM calls this method on nodes where the resource is online. This method can use the API access functions to read property values that might affect an active resource and adjust the running resource accordingly.
Implementing a Failover Resource
A failover resource group contains network addresses such as the built in resource types logical hostname and shared address, and failover resources such as the data service application resources for a failover data service. The network address resources, along with their dependent data service resources move between cluster nodes when data services fail over or are switched over. The RGM provides a number of properties that support implementation of a failover resource.
Set the boolean resource type property Failover to TRUE, to restrict the resource from being configured in a resource group that can be online on more than one node at a time. This property defaults to FALSE, so you must declare it as TRUE in the RTR file for a failover resource.
The Scalable resource property determines if the resource uses the cluster shared-address facility. For a failover resource, set Scalable to FALSE because a failover resource does not use shared addresses.
The RG_mode resource group property allows the cluster administrator to identify a resource group as failover or scalable. If RG_mode is FAILOVER, the RGM sets the Maximum_primaries property of the group to 1 and restricts the resource group to being mastered by a single node. The RGM does not allow a resource whose Failover property is TRUE to be created in a resource group whose RG_mode is SCALABLE.
The Implicit_network_dependencies resource group property specifies that the RGM should enforce implicit strong dependencies of non-network-address resources on all network-address resources (logical hostname and shared address) within the group. This means that the non-network address (data service) resources in the group will not have their Start methods called until the network addresses in the group are configured up. The Implicit_network_dependencies property defaults to TRUE.
Implementing a Scalable Resource
A scalable resource can be online on more than one node simultaneously. Scalable resources include data services such as Sun Cluster HA for Sun One Web Server and HA-Apache.
The RGM provides a number of properties that support implementation of a scalable resource.
Set the boolean resource type property, Failover, to FALSE, to allow the resource to be configured in a resource group that can be online on more than one node at a time.
The Scalable resource property determines if the resource uses the cluster shared-address facility. Set this property to TRUE because a scalable service uses a shared-address resource to make the multiple instances of the scalable service appear as a single service to the client.
The RG_mode property enables the cluster administrator to identify a resource group as failover or scalable. If RG_mode is SCALABLE, the RGM allows Maximum_primaries to have a value greater than 1, meaning the group can be mastered by multiple nodes simultaneously. The RGM allows a resource whose Failover property is FALSE to be instantiated in a resource group whose RG_mode is SCALABLE.
The cluster administrator creates a scalable resource group to contain scalable service resources, and a separate failover resource group to contain the shared-address resources upon which the scalable resource depends.
The cluster administrator uses the RG_dependencies resource group property to specify the order in which resource groups are brought online and offline on a node. This ordering is important for a scalable service because the scalable resources and the shared address resources upon which they depend are in different resource groups. A scalable data service requires that its network address (shared address) resources be configured up before it is started. Therefore, the administrator must set the RG_dependencies property (of the resource group containing the scalable service) to include the resource group containing the shared address resources.
When you declare the Scalable property in the RTR file for a resource, the RGM automatically creates the following set of scalable properties for the resource:
| Network_resources_used | Identifies the shared address resources used by this resource. This property defaults to the empty string so the cluster administrator must provide the actual list of shared addresses the scalable service uses when creating the resource. The scsetup command and SunPlex Manager provide features to automatically set up the necessary resources and groups for scalable services. | ||||||
| Load_balancing_policy | Specifies the load balancing policy for the resource. You can explicitly set the policy in the RTR file (or allow the default, LB_WEIGHTED). In either case, the cluster administrator can change the value when creating the resource (unless you set Tunable for Load_balancing_policy to NONE or FALSE in the RTR file). Legal values are:
For a scalable service with Load_balancing_policy LB_STICKY or LB_STICKY_WILD, changing Load_balancing_weights while the service is online can cause existing client affinities to be reset. In that case, a different node might service a subsequent client request even if the client had been previously serviced by another node in the cluster. Similarly, starting a new instance of the service on a cluster, might reset existing client affinities. | ||||||
| Load_balancing_weights | Specifies the load to be sent to each node. The format is weight@node,weight@node, where weight is an integer reflecting the relative portion of load distributed to the specified node. The fraction of load distributed to a node is the weight for this node divided by the sum of all weights of active instances. For example, 1@1,3@2 specifies that node 1 receives 1/4 of the load and node 2 receives 3/4. | ||||||
| Port_list | Identifies the ports on which the server is listening. This property defaults to the empty string. You can provide a list of ports in the RTR file. Otherwise, the cluster administrator must provide the actual list of ports when creating the resource. |
You can create a data service that can be configured by the administrator to be either scalable or failover. To do so, declare both the Failover resource type property and the Scalable resource property as FALSE in the data service's RTR file. Specify the Scalable property to be tunable at creation.
The Failover property value (FALSE) allows the resource to be configured into a scalable resource group. The administrator can enable shared addresses by changing the value of Scalable to TRUE when creating the resource, and thusly create a scalable service.
On the other hand, even though Failover is set to FALSE, the administrator can configure the resource into a failover resource group to implement a failover service. The administrator does not change the value of Scalable, which is FALSE. To support this contingency, you should provide a check in the Validate method on the Scalable property. If Scalable is FALSE, verify that the resource is configured into a failover resource group.
The Sun Cluster 3.1 10/03 Concepts Guide contains additional information about scalable resources.
Validation Checks for Scalable Services
Whenever a resource is created or updated with the scalable property set to TRUE, the RGM validates various resource properties. If the properties are not configured correctly, the RGM rejects the attempted update or creation. The RGM performs the following checks:
The Network_resources_used property must be non-empty and contain the names of existing shared address resources. Every node in the Nodelist of the resource group containing the scalable resource must appear in either the NetIfList property or AuxNodeList property of each of the named shared address resources.
The RG_dependencies property of the resource group that contains the scalable resource must include the resource groups of all shared address resources listed in the scalable resource's Network_resources_used property.
The Port_list property must be non-empty and contain a list of port-protocol pairs such that protocol is either tcp or udp. For example,
Port_list=80/tcp,40/udp
Writing and Testing Data Services
This section provides some information about writing and testing data services.
Using Keep-Alives
On the server side, using TCP keep-alives protects the server from wasting system resources for a down (or network-partitioned) client. If these resources are not cleaned up (in a server that stays up long enough), eventually the wasted resources grow without bound as clients crash and reboot.
If the client-server communication uses a TCP stream, then both the client and the server should enable the TCP keep-alive mechanism. This provision applies even in the non-HA, single-server case.
Other connection-oriented protocols might also have a keep-alive mechanism.
On the client side, using TCP keep-alives enables the client to be notified when a network address resource has failed over or switched over from one physical host to another. That transfer of the network address resource breaks the TCP connection. However, unless the client has enabled the keep-alive, it does not necessarily learn of the connection break if the connection happens to be quiescent at the time.
For example, suppose the client is waiting for a response from the server to a long-running request, and the client's request message has already arrived at the server and has been acknowledged at the TCP layer. In this situation, the client's TCP module has no need to keep retransmitting the request, and the client application is blocked, waiting for a response to the request.
Where possible, in addition to using the TCP keep-alive mechanism, the client application also must perform its own periodic keep-alive at its level, because the TCP keep-alive mechanism is not perfect in all possible boundary cases. Using an application-level keep-alive typically requires that the client-server protocol supports a null operation or at least an efficient read-only operation such as a status operation.