SUNCLUSTER 3.2 & Above Versions- Just i feel important for an Sys Admin -Anand
Its suggested to move UCLM port range to 7000-7032 for
Suncluster 3.2 and higher
To check the port UDLM is running on
#scrgadm -pvv |grep udlm:port
To check if something is already running on that port
#netstat -an|egrep '\.60-[0-3][0-9]'
also check if new port range is
clear
#netstat -an|egrep '\.70-[0-3][0-9]'
Now lets disable and unmanage the RAC Framework
#scswitch -F -g rac-framework-rg
#scswitch -n -j rac_new
#scswitch -n -j rac_udlm
#scswitch -n -j rac-framework-rg
Now we have to reboot the cluster so it comes up with rac_udlm offline to allow
the port change.
Remember you have to reboot the
cluster using scshutdown on master node, so we have consistent reboot as per
the quorum assigned.
#scshutdown -y -g0
Once the nodes are up, from the master node then make the port change
#scrgadm -c -j rac_udlm-rs -x
port=7000 to set the port range.
Now bring the rac-framework offline
#scswitch -Z -g rac-framework-rg
Verify the cluster is using the new UDLM port 7000.
#scrgadm -pvv |grep udlm:port
If you have any pending resources after port change:
clresource disable -R -g rac-framework-rg +
clresource enable -R -g
rac-framework-rg +
What is Split-Brain?
The term "Split-Brain" is
often used to describe the scenario when two or more co-operating processes in
a distributed system, typically a high availability cluster, lose connectivity
with one another but then continue to operate independently of each other,
including acquiring logical or physical resources, under the incorrect
assumption that the other process(es) are no longer operational or using the
said resources.
What does "co-operating"
mean?
Co-operating processes are those
that use shared or otherwise related resources, including accessing or
modifying shared system state, during the process of performing some
coordinated action, typically at the request of a client.
What's at risk?
The biggest risk following a
Split-Brain event is the potential for corrupting system state. There are
three typical causes of corruption:
The processes that were once
co-operating prior to the Split-Brain event occurring independently modify the
same logically shared state, thus leading to conflicting views of system state.
This is often called the "multi-master problem".
New requests are accepted after the
Split-Brain event and then performed on potentially corrupted system state
(thus potentially corrupting system state even further).
When the processes of the distributed
system "rejoin" together it is possible that they have conflicting
views of system state or resource ownerships. During the process of
resolving conflicts, information may be lost or become corrupted.
Examples of potential corruption
include, creating multiple copies of the same information, updating the same
information multiple times, deleting information, creating multiple events for
a single operation, processing an event multiple times, starting duplicate
services or suspending existing services.
What if the processes aren't
co-operating?
If the processes in a distributed
system aren't co-operating in any way (ie: they don't use any shared
resources), Split-Brain, or it's effects, may not occur.
I thought Split-Brain was all about
physical network infrastructure and connectivity failures. Can it really occur
at the process level, on the same physical server?
While network infrastructure failure
is one of the more common causes of Split-Brain, the loss of communication or
connectivity between two or more processes on a single physical server, even
running on a single processor, may also cause a Split-Brain event.
For example; if one of two
co-operating processes on a server are swapped out for a long period of time,
longer than the configured network or connectivity time-out between the
processes, a Split-Brain may occur if each process continues to operate
independently, especially when the swapped out process returns to normal
operation. ie: The swapped process does not take into account that it has been
unavailable for a long period of time.
Similarly if a process is
interrupted for a long period of time, say due to an unusually long Garbage
Collection or when a physical processor is unavailable for a process to due
heavy contention virtualized infrastructure, the said process may not respond
to communication requests from another process, and thus a Split-Brain may
occur.
Split-Brain does not require
physical network infrastructure to occur.
Garbage Collection can't cause a
Split-Brain. Right?
Unfortunately not. As
explained above, excessively long Garbage Collection or regular back-to-back
Garbage Collections may make a process seen unavailable to other processes in a
distributed system and thus not be in a position to respond to communication
requests.
Split-Brain will only ever occur
when a system has two processes. With three or more processes it can
easily be detected and resolved. Right?
Unfortunately not. Split-Brain
scenarios may just as easily occur when there are n processes (where n > 2)
in a distributed system. For example: it's possible that all n processes
in a distributed system, especially on heavily loaded hardware, could be
swapped out at the same point in time, thus effectively losing connectivity
with each other, thus potentially creating an n-way Split-Brain.
As Split-Brains occur as the result
of incorrectly performing some action due to an observed communication failure,
the number of "pieces" of "brains" may be greater than two.
Splits always occur
"down-the-middle". Right?
Unfortunately not. Following
on from above, as most distributed systems consist of large numbers of
processes (large values of n), splits rarely occur "down-the-middle",
especially when n is odd number! In fact, even when n is an even number, there
is absolutely no guarantee that a split will contain two equally sized
collections of processes. For example: A system consisting of five
processes may be split such that one side of the system (ie: brain) may have
three processes and the other side may have two processes. Alternatively
it may be split such that one side has four processes and the other has just
one process. In a system with six processes, a split may occur with four
processes on one side and just two on another.
Stateless architectures don't suffer
from Split-Brain. Right?
Correct. Systems that don't
have shared state or use shared resources typically don't suffer from
Split-Brain.
Client+Server architectures don't
suffer from Split-Brain. Right?
Correct - if and only if the Server
component of the architecture operates as single process. However if the
Server component of an architecture operates as a collection of processes, it's
possible that such an architecture will suffer from Split-Brain.
All Stateful architectures suffer
from Split-Brain. Right?
The "statefulness" of an
architecture does not imply that it will suffer from Split-Brain. It is
very possible to define an architecture that is stateful and yet avoids the
possibility of Split-Brain (as defined above), by ensuring no shared resources
are accessed across processes.
The solution is simple - completely
avoid Split-Brains by waiting longer for communication to recover, doing more
checks and avoiding assumptions. Right?
Unfortunately in most Split-Brain
scenarios all that can be observed is an inability to send and/or receive
information. It is from these observations that systems must make assumptions
about a failure. When these assumptions are incorrect, Split-Brain may
occur.
While waiting longer for a communication
response may seem like a reasonable solution, the challenge is not in waiting.
The waiting part is easy. The challenge is determining "how long to
wait" or "how many times to retry".
Unfortunately to determine "how
long to wait" or "retry" we need to make some assumptions about
process connectivity and ultimately process availability. The challenge
here is that those assumptions may quickly become invalid, especially in a
dynamically or arbitrarily loaded distributed system. Alternatively if there
is a sudden spike in the number of requests, processes may pause more
frequently (especially in the case of Garbage Collection or in virtualized
environments) and thus increase the potential for a Split-Brain scenario to
form.
Split-Brain only occurs in systems
that use unreliable network protocols (ie: protocols other than TCP/IP). Right?
The network protocol used by a
distributed system, whether a reliable protocol like TCP/IP or unreliable
protocol like UDP (unicast or multicast) does not preclude a Split-Brain from
occurring. As discussed above, a physical network is not required for a
Split-Brain scenario to develop.
TCP/IP decreases the chances of a
Split-Brain occurring. Right?
Unfortunately not. Following
on from above, the operational challenge with the use of TCP/IP with in a
distributed system is that the protocol does not directly expose internal
communication failures to the system, like for example the ability to send from
a process and not receive (often called "deafness") or the ability to
receive but not send (often called "muteness"). Instead TCP/IP
protocol failures are only notified after a fixed, typically operating system
designated time-out (usually configurable but typically set to seconds or even
minutes by default). However, the use of unreliable protocols such as
UDP, may highlight the communication problems sooner, including the ability to
detect "deafness" or "muteness" and thus allow a system to
take corrective action soon.
Having all processes in a
distributed system connected via a single physical switch will help prevent
Split-Brain. Right?
While deploying a distributed system
such that all processes are interconnected via a single physical switch may
seem to reduce the chances of a Split-Brain occurring, the possibility of a
switch failing atomically at once is extremely low. Typically when a
switch fails, it does so in an unreliable and degrading manner. That is,
some components of a switch will continue to remain operational where as others
may be intermittent. Thus in their entirety, switches become
intermittent, before they fail completely (or are shutdown completely for
maintenance).
What are the best practices for
dealing with Split-Brain?
While the problem of Split-Brains
can't be solved using a generalized approach, "prevention" and
"cure" are possible.
There are essentially six approaches
that may be used to prevent a Split-Brain from occurring;
· Use high quality and reliable network infrastructure.
· Provide multiple paths of communication between processes,
so that a single observed communication failure does not trigger a Split-Brain.
· Avoid overloading physical resources so that processes are
not swapped out for long periods of time.
· Avoid unexpectedly long Garbage Collection pauses.
· Ensure communication time-outs are suitably long enough to
prevent a Split-Brain occurring "too early" due to (3) or (4)
· Architect an application so that it uses few shared
resources (this is rarely possible).
However even implementing all of the
approaches, Split-Brain may still occur. In which case we need to focus
on "cure".
There are essentially four
approaches to "curing" a Split-Brain:
Fail-Fast: As soon as a Split-Brain scenario is detected, the entire
system or suitable processes in the system are immediately shutdown to avoid
the possibility of corruption.
Isolation is weak form of Fail-Fast. Instead of shutting down processes of a system,
they are simply isolated. When the Split-Brain is recovered, the isolated
processes are re-introduced into the system as if they were new (ie: they drop
any previously held information/assumptions to avoid corruption). After the
Split-Brain event has occurred, the Isolated processes may continue to perform
work, but on re-introduction to the system, currently information / processing
will be lost.
Fencing is a stronger form of isolation, but still a weaker form of
Fail-Fast. Fencing requires an additional constraint over Isolation in
that fenced-processes must stop execution immediately (and release resources),
rather than continue to operate after a Split-Brain has occurred.
Resolve Conflicts (assumes the above two approaches have not be used)
When the communication channels have
been recovered, it's highly possible that there are conflicts at the resource
level - including conflicts in system state. By providing a
"Conflict Resolving" interface, developers may provide an application
specific mechanism for resolving said conflicts, thus allowing a system to
continue to operation. Of course, this is completely application
dependent and development intensive, but provides the best way to recover.
How can a Split-Brain event be
detected? How are they defined?
Unfortunately there is no simple way
to define or detect when a Split-Brain has occurred. While it's fairly
obvious and perhaps trivial in a system composed of only two processes, these
are increasingly rare. Most distributed systems have 10's if not 1000's
of processes.
For example; if four processes in a
five process system collectively lose contact with a single process, does that
mean a four-to-one Split-Brain has occurred, or simply that a single process
has failed?
The common solution to this problem
to define what is called a Quorum, the idea being, those processes not
belonging to the quorum should Fail-Fast or be Isolated. The
typically way to define a quorum is to specify the minimum number of
co-operating processes must be collectively available to continue operating.
Hence for the above example, with a
quorum of "three processes", the system would treat a failure of a
single process not as a Split-Brain, but simply as a lost process.
However if the quorum were incorrectly defined to be "a single
process", a Split-Brain would occur - with a four to one split.
Obviously depending on size-based
quorums may be problematic as it's yet another assumption we need to make -
"how big should a quorum be?". Often however the definition of
a quorum is less about the number of processes that are collectively available,
but instead more about the roles or locality of the said processes.
For example; if three processes in a
five process system collectively lose contact with two other processes, but
those two processes remain in contact, we essentially have a three-to-two
Split-Brain. With a quorum that is defined using a "size-based"
approach, the two processes may be failed-fast/isolated. However let's
consider that the three processes have also lost connectivity with a valuable
resource (say a database or network attached storage device), but the other two
processes have not. In this situation it's often preferable that the two
surviving processes should remain available and the other three processes
should be failed-fast/isolated.
Further, and as previously
mentioned, it's important to remember that failures in communication between
processes are rarely observed to occur at the same time. Rather they are
observed over a period of time, perhaps seconds, minutes or even hours.
Thus when discussing
"when" a Split-Brain occurs, we usually need to consider the entire
period of time, during which there may be multiple failure and recovery events,
to fully conclude that a Split-Brain has occurred.
How does Oracle Coherence deal with
Split-Brain?
Internally Oracle Coherence uses a
variety of both proprietary (UDP uni-cast and multi-cast based) and standard
(TCP/IP) network technologies for inter-process communication and maintaining
system health. These technologies are combined in multiple ways to enable
highly scalable and high-performance one-to-one and one-to-many communication
channels to be established and reliably observed, across hundreds (even
thousands if you really like) of processes, with little CPU or network
overhead.
For example, through the combined
use of these technologies Coherence can easily detect and appropriately deal
with remote-garbage-collection across a system sub-second (using commodity 1Gb
switch infrastructure).
The major philosophy and delivered
advantages of this approach is to ideally "prevent" a Split-Brain
occurring as much as possible. Should a Split-Brain occur, say due to a
switch failure, Coherence does the following:
Uses Isolation to ensure reliable
system communication between the individual fragments of the Split-Brain.
Without reliable communication, data integrity within the fragments cannot be
guaranteed (or recovered)
Uses Fencing around processes that
are deemed unresponsive (ie: deaf), but continue to communicate with other
processes in the system. This is a form of dynamic blacklisting to ensure
system-wide communication reliability and prevent Fenced processes corrupting
state.
Raises real-time programmatic events
concerning process health. This enables developers to provide custom Fail-Fast
algorithms based on system health changes. In Coherence these events are
handled using MemberListeners.
Raises real-time programmatic events
concerning the locality and availability of partitions (ie: information). This
enables developers to provide custom Fail-Fast algorithms based on partitions
being move or becoming unavailable due to catastrophic system failure. In
Coherence these events may be handled using Backing Map Listeners and/or
Partition Listeners.
Uses a simple "largest side
wins" size-based quorum rule to resolve resource (and state) ownership
when a Split-Brain is recovered.
Amnesia Scenario:
Node node-1 is shut down.
Node node-2 crashes and will not
boot due to hardware failure.
Node node-1 is rebooted but stops
and prints out the messages:
Booting as part
of a cluster
NOTICE: CMM: Node
node-1 (nodeid = 1) with votecount = 1 added.
NOTICE: CMM: Node
node-2 (nodeid = 2) with votecount = 1 added.
NOTICE: CMM:
Quorum device 1 (/dev/did/rdsk/d4s2) added; votecount = 1, bitmask of nodes
with configured paths = 0x3.
NOTICE: CMM: Node
node-1: attempting to join cluster.
NOTICE: CMM:
Quorum device 1 (gdevname /dev/did/rdsk/d4s2) can not be acquired by the
current cluster members. This quorum device is held by node 2.
NOTICE: CMM:
Cluster doesn't have operational quorum yet; waiting for quorum.
Node node-1 cannot boot completely
because it cannot achieve the needed quorum vote count.
NOTE: for Oracle Solaris 3.2 update
1 and above with Solaris 10 : the boot continue, in non cluster mode, after a
timeout.
In the above case, node node-1
cannot start the cluster due to the amnesia protection of Oracle Solaris
Cluster. Since node node-1 was not a member of the cluster when it was shut
down (when node-2 crashed) there is a possibility it has an outdated CCR and
should not be allowed to automatically start up the cluster on its own.
The general rule is that a node can
only start the cluster if it was part of the cluster when the cluster was last
shut down. In a multi node cluster it is possible for more than one node to
become "the last" leaving the cluster.
Resolution:
If you have a 3 node cluster, start
with a node that is suitable for starting the cluster. Eg a node connected to
the majority of the storage. In this example node-1 represents this first node.
1. Stop node-1 and reboot in
non-cluster mode. (Single user not necessary, only faster)
ok boot -sx
2. Make a backup of
/etc/cluster/ccr/infrastructure file or /etc/cluster/ccr/global/infrastructure
depending upon cluster and patch revisions noted in UPDATE_NOTE #1 below.
# cd /etc/cluster/ccr
# /usr/bin/cp infrastructure
infrastructure.old
Or if UPDATE_NOTE #1 applies
# cd /etc/cluster/ccr/global
# /usr/bin/cp infrastructure
infrastructure.old
3. Get this node's id.
# cat /etc/cluster/nodeid
4. Edit the
/etc/cluster/ccr/infrastructure file or /etc/cluster/ccr/global/infrastructure
depending upon cluster and patch revisions noted in UPDATE_NOTE #1 below.
Change the quorum_vote to 1 for the
node that is up (node-1, nodeid = 1).
cluster.nodes.1.name node-1
cluster.nodes.1.state enabled
cluster.nodes.1.properties.quorum_vote 1
For all other nodes and any Quorum
Device, set the votecount to zero.
Other nodes, where is any node id
but the one edited above:
cluster.nodes..properties.quorum_vote
0
Quorum Device(s), where is the
quorum device id which is internal to the cluster code:
cluster.quorum_devices..properties.votecount
0
5. Regenerate the checksum of the
infrastructure file by running:
# /usr/cluster/lib/sc/ccradm -i
/etc/cluster/ccr/infrastructure -o
Or if UPDATE_NOTE #1 applies
# /usr/cluster/lib/sc/ccradm -i
/etc/cluster/ccr/global/infrastructure -o
NOTE: If running SC 3.3, SC 3.2u3 or
3.2 Cluster core patch equal to or greater then 126105-36 (5.9) or 126106-36
(5.10) or 126107-36 (5.10 x86) the ccradm command would be
# /usr/cluster/lib/sc/ccradm recover
-o infrastructure
6. Boot node node-1 into the
cluster.
# /usr/sbin/reboot
7. The cluster is now started, so as
long as other nodes have been mended they can be booted up and join the cluster
again. When these nodes joins the cluster their votecount will be reset to its
original value, and if a node is connected to any quorum device its voteqount
will also be reset.
UPDATE_NOTE #1: If running Solaris
Cluster 3.2u2 or higher, the directory path /etc/cluster/ccr is replaced with
/etc/cluster/ccr/global. The same applies if running Cluster core patch equal
to or greater then 126105-27 (5.9) or 126106-27 (5.10) or 126107-27 (5.10 x86)
Troubleshooting Steps Outline for
node not booting into cluster
1. Verify booting from the right
system disk
2. Verify actual cluster doesn't
start issue
3. Verify correct boot syntax
4. Verify not “invalid CCR table”
5. Verify not “waiting for
operational quorum” amnesia
6. Confirm private interfaces have
started
7. Verify that the node did not
already join the cluster with a date/time in the future.
8. Verify if other errors are
reported on the console.
Troubleshooting Steps in Detail:
1) Verify booting from the right
system disk
In many cases, there are several
system disks attached to a system: the current disk,
the mirror of the current disk,
backup of system disk, system disk with a previous solaris version, ...
Be sure that you are booting from
the right disk.
If you are using an alias, check
that the devalias points to the right system disk.
2) Verify actual cluster doesn't
start issue
Ensure the system will boot in
non-cluster mode.
This step will verify the hardware
and Solaris operating system are functional.
A system may be booted in
non-cluster mode by adding the '-x' boot flag.
3) Verify correct boot syntax
Check that you used the right boot syntax
to boot the node.
If you boot using an alias
In order to boot a node in cluster
mode, you must not use the -x flag.
And you must see the following
message on the console
Booting as part of a cluster
4) Verify not “invalid CCR table”
The cluster uses several tables
stored as files in the CCR repository.
If one of is table is corrupted, it
could prevent the node to boot in the cluster.
You may see errors like the
following ones on the console:
* UNRECOVERABLE ERROR:
/etc/cluster/ccr/infrastructure file is corrupted Please reboot in noncluster
mode(boot -x) and Repair
* WARNING: CCR: Invalid CCR table :
infrastructure
* WARNING: CCR: Invalid CCR table :
dcs_service_xx.
* WARNING: CCR: Table
/etc/cluster/ccr/directory has invalid checksum field. .....
UNRECOVERABLE ERROR: Sun Cluster
boot: Could not initialize cluster framework Please reboot in non cluster
mode(boot -x) and Repair
* ...
In such case, further
troubleshooting is required to understand the failure. For additional support
contact Oracle Support.
5) Verify not “waiting for
operational quorum” amnesia
A cluster must reach an operational
quorum in order to start.
It means that we must have enough
votes (more than half of all configured votes) to be able to start.
A node will report the "waiting
for operational quorum” message during boot if that quorum cannot be reached.
NOTICE: CMM: Node ita-v240c:
attempting to join cluster. NOTICE: CMM: Cluster doesn't have operational
quorum yet; waiting for quorum.
This happens when the node is not
able to talk to the other cluster nodes through the private links and is not
able to access the quorum device(s).
This usually happens when the other
cluster node has been the last one being stopped and we are rebooting another
node first.
In that case check that you are
booting the nodes in the right order.
If the other node owning the quorum
device(s) cannot be booted due to other problems, then you need to recover from
amnesia.
6) Verify that the private
interfaces have started.
A node will not be able to join the
cluster if the private links do not start.
You'll then see messages like the
following ones:
NOTICE: clcomm: Adapter bge3
constructed
NOTICE: clcomm: Adapter bge2
constructed
NOTICE: CMM: Node ita-v240c:
attempting to join cluster.
NOTICE: CMM: Cluster doesn't have
operational quorum yet; waiting for quorum.
NOTICE: clcomm: Path ita-v240c:bge3
- ita-v240b:bge3 errors during initiation
NOTICE: clcomm: Path ita-v240c:bge2
- ita-v240b:bge2 errors during initiation
WARNING: Path ita-v240c:bge3 -
ita-v240b:bge3 initiation encountered errors, errno = 62. Remote node may be
down or unreachable through this path.
WARNING: Path ita-v240c:bge2 -
ita-v240b:bge2 initiation encountered errors, errno = 62. Remote node may be
down or unreachable through this path.
The node won't be able to talk to
the other node(s) and continue its boot until it prints:
NOTICE: clcomm: Path ita-v240c:bge3
- ita-v240b:bge3 online
NOTICE: clcomm: Path ita-v240c:bge2
- ita-v240b:bge2 online
7) Verify that the node did not
already join the cluster with a date/time in the future.
When node C joins node B in a
cluster, node B saves the boot date/time of node C.
It is used by node B as node C's
incarnation number.
May 19 03:12:05 ita-v240b
cl_runtime: [ID 537175 kern.notice] NOTICE: CMM: Node ita-v240c (nodeid: 1,
incarnation #: 1242748936) has become reachable.
May 19 03:12:05 ita-v240b
cl_runtime: [ID 377347 kern.notice] NOTICE: CMM: Node ita-v240c (nodeid = 1) is
up; new incarnation number = 1242748936.
Now assume that node C joined the
cluster with the wrong date/time being set, say a few hours in the future.
The administrator realizes that and
decides to reboot node C with a fixed date/time.
Node B will then see node C trying
to join the cluster with an incarnation number before the one it already knows.
Node B then think that this is a
stale version of node C trying to join the cluster and it will refuse that, and
report the following message:
May 19 03:21:08 ita-v240b
cl_runtime: [ID 182413 kern.warning] WARNING: clcomm: Rejecting communication
attempt from a stale incarnation of node ita-v240c;
reported boot time Tue May 19
08:21:01 GMT 2009, expected boot time Tue May 19 16:02:16 GMT 2009 or later.
In older SunCluster version (3.0 for
example), this message is not printed at all.
You may find more information by
dumping a cluster debug buffer on node B.
ita-v240b# mdb -k
Loading modules: [ unix krtld
genunix dtrace specfs ufs sd pcisch ip hook neti sctp arp usba fcp fctl qlc nca
ssd lofs zfs random logindmux ptm cpc sppp crypto fcip nfs ]
>*udp_debug/s
.....
th 3000305f6c0 tm 964861043: PE
60012f10440 peer 1 laid 2 udp_putq stale remote incarnation 1242721261
.....
Node C comes with an incarnation of
1242721261 which is before the already known incarnation of 1242748936.
Node C will only be allowed to join
the cluster once its date/time goes beyond the date/time when it first joined
the cluster.
To fix this, wait for the time to go
beyond the first date/time and reboot node C.
This is not always possible if the
date was too far in the future (days, weeks, months or years!).
The only other fix is to reboot the
whole cluster.
May 19 03:46:14 ita-v240b
cl_runtime: [ID 537175 kern.notice] NOTICE: CMM: Node ita-v240c (nodeid: 1,
incarnation #: 1242722767) has become reachable.
May 19 03:46:14 ita-v240b
cl_runtime: [ID 377347 kern.notice] NOTICE: CMM: Node ita-v240c (nodeid = 1) is
up; new incarnation number = 1242722767.
May 19 03:46:14 ita-v240b
cl_runtime: [ID 377347 kern.notice] NOTICE: CMM: Node ita-v240b (nodeid = 2) is
up; new incarnation number = 1242722750.
8) Verify if other errors are
reported on the console.
The node can fail to join the
cluster due to other errors seen during the boot sequence in the various
scripts being run.
This can block the boot sequence
until the issue is fixed.
This includes: - failure to mount
the global devices filesystem. - mount point missing. - invalid /etc/vfstab
entry.
Fix the error seen and reboot the
node in the cluster.
Cluster Panics due to Reservation
Conflict:
A major issue for clusters is a
failure that causes the cluster to become partitioned (called split brain).
When split brain occurs, not all nodes can communicate, so individual nodes or
subsets of nodes might try to form individual or subset clusters. Each subset
or partition might believe it has sole access and ownership to the multihost
devices. When multiple nodes attempt to write to the disks, data corruption can
occur. In order to avoid data corruption Solaris Cluster software uses SCSI
Reservation method to implement failure fencing and to limit the nodes leaving
the cluster from accessing shared devices.
When the hosts are in cluster they are able to write their keys on shared
devices and access the devices. However, when a nodes becomes unreachable, the
cluster framework will drop the node from the cluster and have the dropped
node's keys removed. Therefore, cluster framework fences that node out and
limits its access to the shared devices.
There are three types of SCSI reservation methods: SCSI2, SCSI3 and
SCSI2/SCSI3 Persistent Group Reservation Emulation (PGRE).
In an SCSI2 type reservation, only one host at a time can reserve and
access the disk whereas in SCSI3 and SCSI3 PGRE every host connected to the
host can write its keys and access the disk. Also SCSI2 reservation is cleared
by reboots whereas the other two methods are permanent and survive
reboots.
Solaris Cluster framework uses SCSI3
PGRE for the storage devices with two physical paths to the hosts and SCSI3 for
the devices with more than 3 paths to the hosts.
Reservation conflict arises when a
node, not in communication with the surviving node(s) in the cluster, either a
fenced out member of the cluster or a non-member host from another cluster
tries to access a shared device it has been fenced off from, i.e. its key is
not present on the device. In such a case, if failfast mechanism is enabled on
the offending node (e.g. when the node is running in the cluster but unaware
that it is fenced off due to unforeseen circumstances such as a hung status)
will panic by design in order to limit its access to the shared device.
Solaris Cluster 3.x node panics with "SCSI Reservation
Conflict" with the following messages:
Nov 14 15:14:28 node2 cl_dlpitrans:
[ID 624622 kern.notice] Notifying cluster that this node is panicking
Nov 14 15:14:28 node2 unix: [ID
836849 kern.notice]
Nov 14 15:14:28 node2
^Mpanic[cpu8]/thread=2a100849ca0:
Nov 14 15:14:28 node2 unix: [ID
747640 kern.notice] Reservation Conflict
Nov 14 15:14:28 node2 unix: [ID
100000 kern.notice]
To troubleshoot this we have do few verifications as given below:
1. Verify that no disk is accessible
by hosts that are not part of this cluster:
If a disk is used by another host(s)
of another cluster or host(s) using SCSI reservation methods, even if it is not
being used or mounted, the keys on the disk might be altered without the
knowledge of any of the nodes in our cluster.
2. Check EMC Gate Keeper or VCMD disks
are accessible by cluster nodes:
If EMC Gate Keeper or VCMDB disks are exposed to Sun Cluster, the
cluster nodes might panic with SCSI Reservation Conflict panics. To avoid such
circumstances, one might "blacklist" VCMDB disks. Please consult EMC
for appropriate procedure.
We have to unmask or blacklist them
in /etc/fp.conf file .
3. Check the time stamps of the messages on all the nodes during and prior to
the panic:
When the nodes booted into a cluster stops receiving heart beats from
other node(s) via private interconnects, they have no way of telling if the the
problem was due to:
problem with the private interconnect paths, e.g. cable problems,
the other node(s) going down as a
part of a normal shutdown, i.e. "init 0",
the other nodes(s) paniced
the other node(s) being hung.
Removing SCSI Reservations
SCSI-2 Reservations and SCSI-3
Persistent Reservations. Solaris Cluster uses only one SCSI reservation type
for a given shared storage device. However, both SCSI reservation types may be
used at the same time in the same cluster.
To remove SCSI-2 Reservations use
the 'scsi' command with the 'release' option as shown below. This
command, however, must be executed from the system that owns the reservation in
order to successfully remove the reservation.
# /usr/cluster/lib/sc/scsi -c disfailfast -d /dev/did/rdsk/d#s2
# /usr/cluster/lib/sc/scsi -c
release -d /dev/did/rdsk/d#s2
The '#' character is a reference to
the DID device number of the disk from which you wish to remove a SCSI-2
Reservation. The standard Solaris disk device reference, /dev/rdsk/c#t#d#s2,
can also be used in place of the DID device reference.
Note: Sun Microsystems recommends
that you first disable the failfast mechanism using the 'scsi' command with the
'disfailfast' option, as shown above, before removing any SCSI reservations.
The system will panic if the system is running in cluster mode and you do
not disable the failfast mechanism and you mistakenly attempt a release of a
SCSI-2 Reservation when the SCSI reservation was in fact a SCSI-3 Persistent
Reservation.
An alternative is to use the 'reserve' command with the 'release' option to
remove a SCSI-2 Reservation as shown below.
# /usr/cluster/lib/sc/reserve
-c disfailfast -z /dev/did/rdsk/d#s2
# /usr/cluster/lib/sc/reserve
-c release -z /dev/did/rdsk/d#s2
Note: SCSI-2 Reservations will be
removed automatically if the system that owns the SCSI-2 Reservations is shut
down or power cycled. Likewise, SCSI-2 Reservations will be removed
automatically if the storage devices that have SCSI-2 Reservations on them are
shut down or power cycled. There are other methods that also remove
SCSI-2 Reservations without resorting to the commands presented above to do so.
For example, a SCSI bus reset will remove SCSI-2 Reservations from the
storage devices affected by the reset.
Example 1:If DID device d15 has a SCSI-2 Reservation you wish to remove, use
the following commands.
# /usr/cluster/lib/sc/scsi -c
disfailfast -d /dev/did/rdsk/d15s2
do_enfailfast returned 0
# /usr/cluster/lib/sc/scsi -c
release -d /dev/did/rdsk/d15s2
do_scsi2_release returned 0
Alternatively, if DID device d15
corresponds to disk device c3t4d0 on the cluster node where the commands will
be executed, you could use the following commands instead.
# /usr/cluster/lib/sc/reserve
-c disfailfast -z /dev/rdsk/c3t4d0s2
do_enfailfast returned 0
# /usr/cluster/lib/sc/reserve
-c release -z /dev/rdsk/c3t4d0s2
do_scsi2_release 0
Note, again, that you can use either
the DID device reference or the Solaris disk device reference with both the
'scsi' and the 'reserve' commands.
Example 2:If DID device d15 has a SCSI-3 Persistent Reservation and you
mistakenly execute the 'scsi' command with the 'release' option without
disabling the failfast mechanism, the system will panic if it is running in
cluster mode. So, if you execute the following command against a device
that has a SCSI-3 Persistent Reservation and the command prompt does not
return, then your system has probably experienced a 'Reservation Conflict'
panic.
# /usr/cluster/lib/sc/scsi -c
release -d /dev/did/rdsk/d15s2
Check the system console for a panic
message like the following.
panic[cpu0]/thread=2a1003e5d20: Reservation Conflict
If, however, DID device d15 has a
SCSI-3 Persistent Reservation and you mistakenly use both commands presented at
the beginning of this section, you should see the following results.
# /usr/cluster/lib/sc/scsi -c disfailfast -d /dev/did/rdsk/d15s2
do_enfailfast returned 0
# /usr/cluster/lib/sc/scsi -c
release -d /dev/did/rdsk/d15s2
do_scsi2_release returned -1
The failfast mechanism will be
disabled for DID device d15, but the SCSI-3 Persistent Reservation will remain
intact. The 'release' option has no effect on a SCSI-3 Persistent
Reservation.
Removing SCSI-3 Persistent
Reservations
To remove SCSI-3 Persistent
Reservations and all the reservation keys registered with a device use the
'scsi' command with the 'scrub' option as shown below. This command does
not need to be executed from the system that owns the reservation or from a
system running in the cluster.
# /usr/cluster/lib/sc/scsi -c disfailfast -d /dev/did/rdsk/d#s2
# /usr/cluster/lib/sc/scsi -c
scrub -d /dev/did/rdsk/d#s2
The '#' character is a reference to
the DID device number of the disk from which you wish to remove a SCSI-3 Persistent
Reservation. The standard Solaris disk device reference,
/dev/rdsk/c#t#d#s2, can also be used in place of the DID device reference.
Note: Sun Microsystems recommends that you first disable the failfast mechanism
using the 'scsi' command with the 'disfailfast' option, as shown above, before
removing any SCSI reservations.
Before executing the scrub operation, it is recommended that you confirm there
are reservation keys registered with the device. To confirm there are
reservation keys registered with the device execute the 'scsi' command with the
'inkeys' option and then with the 'inresv' option as shown below.
# /usr/cluster/lib/sc/scsi -c
inkeys -d /dev/did/rdsk/d#s2
# /usr/cluster/lib/sc/scsi -c
inresv -d /dev/did/rdsk/d#s2
Note: SCSI-3 Persistent Reservations are 'persistent' by design which means
that any reservation keys registered with a storage device or any reservation
placed on a storage device are to be retained by the storage device, even if
powered off, and the only way to remove them is by issuing a specific SCSI
command to do so. In other words, there are no automatic methods whereby
a SCSI-3 Persistent Reservation will be removed, such as a reset or power cycle
of the device. Sun Cluster uses the Persistent Reserve Out (PROUT) SCSI
command with the PREEMPT AND ABORT service action to remove SCSI-3 Persistent
Reservations. This PROUT SCSI command is also programmed into the 'scsi'
command through the 'scrub' option for your use when absolutely necessary.
Note: Most storage systems also provide a method to manually remove SCSI-3
Persistent Reservations by accessing or logging into the storage device
controller and executing commands defined by the storage vendor for this
purpose.
Example 3:If DID device d27 has a SCSI-3 Persistent Reservation you wish to
remove, use the following commands.
The first command displays the
reservation keys registered with the device.
# /usr/cluster/lib/sc/scsi -c
inkeys -d /dev/did/rdsk/d27s2
Reservation keys(3):
0x44441b0800000003
0x44441b0800000001
0x44441b0800000002
The second command displays the
SCSI-3 Persistent Reservation owner and the reservation type.
# /usr/cluster/lib/sc/scsi -c inresv -d /dev/did/rdsk/d27s2
Reservations(1):
0x44441b0800000001
type ---> 5
Note: Type 5 corresponds to a
Write-Exclusive Registrants-Only type of SCSI-3 Persistent Reservation.
This is the SCSI-3 Persistent Reservation type used in Sun Cluster.
The third command removes the SCSI-3 Persistent Reservation and all the
reservation keys registered with the device.
# /usr/cluster/lib/sc/scsi -c
scrub -d /dev/did/rdsk/d27s2
Reservation keys currently on
disk:
0x44441b0800000003
0x44441b0800000001
0x44441b0800000002
Attempting to remove all keys
from the disk...
Scrubbing complete, use
'/usr/cluster/lib/sc/scsi -c inkeys -d /dev/did/rdsk/d27s2' to verify success
When executing the 'scsi' command
with the 'scrub' option, the output first displays the keys currently
registered with the device and then recommends that you check the device again
to confirm that the registered keys have been removed. This command and
the result is shown below.
# /usr/cluster/lib/sc/scsi -c
inkeys -d /dev/did/rdsk/d27s2
Reservation keys(0):
You should also confirm that the
SCSI-3 Persistent Reservation has also been removed.
# /usr/cluster/lib/sc/scsi -c
inresv -d /dev/did/rdsk/d27s2
Reservations(0):
Removing PGRe Keys
Sun Cluster uses the SCSI-3
Persistent Reservation keys that are registered with the quorum device when the
Cluster Membership Monitor (CMM) determines the quorum vote tally. How
these keys are used by the CMM is beyond the scope of this document, however
when SCSI-2 Reservations are used there are no reservation keys.
Therefore, when SCSI-2 Reservations are used with a quorum device, Sun
Cluster uses an emulation mode to store SCSI-3 Persistent Reservation keys on
the quorum device for when the CMM needs to use them.
This mechanism is referred to
as Persistent Group Reservation emulation, or PGRe, and even though the reservation
keys are the same ones used with SCSI-3 compliant devices they will be referred
to in the remainder of this document as emulation keys because of the special,
non-SCSI, way they are handled and stored in this case.
Note: Emulation keys are stored in a vendor defined location on the disk and do
not interfere with or take away from the storage of user data on the quorum
device.
Note: The PGRe mechanism is used only by Sun Cluster and the way emulation keys
are handled and stored are defined only by Sun Cluster. The PGRe
mechanism and the emulation keys are not a part of the SCSI Specification.
In other words, the PGRe mechanism and the way emulation keys are handled
and stored do not have any operational effect as it relates to what is defined
in the SCSI Specification documents. Therefore, any desire or need to
remove emulation keys must be associated only with Sun Cluster itself and
should have nothing whatsoever to do with the storage devices used as quorum
devices
To remove emulation keys use the 'pgre' command with the 'pgre_scrub' option as
shown below.
# /usr/cluster/lib/sc/pgre -c
pgre_scrub -d /dev/did/rdsk/d#s2
The '#' character is a reference to
the DID device number of the disk from which you wish to remove emulation keys.
The standard Solaris disk device reference, /dev/rdsk/c#t#d#s2, can also
be used in place of the DID device reference.
Before executing the pgre_scrub
operation, it is recommended that you confirm there are reservation keys
registered with the device. To confirm there are reservation keys
registered with the device execute the 'pgre' command with the 'pgre_inkeys'
option and then with the 'pgre_inresv' option as shown below.
# /usr/cluster/lib/sc/pgre -c
pgre_inkeys -d /dev/did/rdsk/d#s2
# /usr/cluster/lib/sc/pgre -c
pgre_inresv -d /dev/did/rdsk/d#s2
Example 4:If DID device d4 has emulation keys you wish to remove, use the
following commands.
The first command displays the
emulation keys that have been registered with the device.
# /usr/cluster/lib/sc/pgre -c
pgre_inkeys -d /dev/did/rdsk/d4s2
key[0]=0x447f129700000001.
The second command displays which
emulation key is the reservation owner.
# /usr/cluster/lib/sc/pgre -c
pgre_inresv -d /dev/did/rdsk/d4s2
resv[0]:
key=0x447f129700000001.
The third command removes the
emulation keys from the device.
# /usr/cluster/lib/sc/pgre -c
pgre_scrub -d /dev/did/rdsk/d4s2
Scrubbing complete. Use
'/usr/cluster/lib/sc/pgre -c pgre_inkeys -d /dev/did/rdsk/d4s2'
to verify success.
When executing the 'pgre' command with
the 'pgre_scrub' option, the output displays whether the operation completed
successfully and then recommends that you check the device again to confirm
that the emulation keys have been removed. This command and the result is
shown below.
# /usr/cluster/lib/sc/pgre -c
pgre_inkeys -d /dev/did/rdsk/d4s2
No keys registered.
You should also confirm that the
reservation owner has also been removed.
# /usr/cluster/lib/sc/pgre -c
pgre_inresv -d /dev/did/rdsk/d4s2
No reservations on the
device.
Example 5:If the DID device has never been used as a quorum device you will
receive the following error when you use the 'pgre' command. This error
means that the storage area on the disk used by Sun Cluster to store the
emulation keys has not been initialized. Sun Cluster initializes the area
on the disk to store emulation keys when the quorum device is created.
# /usr/cluster/lib/sc/pgre -c
pgre_inkeys -d /dev/did/rdsk/d7s2
quorum_scsi2_sector_read:
pgre id mismatch. The sector id is .
scsi2 read returned error
(22).
/usr/cluster/lib/sc/pgre -c
pgre_inkeys -d /dev/did/rdsk/d7s2 command failed errno = 22.
Sun cluster software is unable to set any scsi-3 reservations on the array even
though array is functioning:
The reservations needed to be cleared
at the array level.
From any host connected to the
array, unmount the volumes.Confirm that you have a host with serial access to
the array.
On the host managing the array, run sccli:
# /usr/sbin/sccli
Choose the correct array if multiple
are connected to the host. Run the command to shutdown the controller, it is
necessary to sync the cache to the disks so that the LDs remain consistent.
sccli> shutdown
controller
Reset the controller
sccli> reset
controller
From the host with serial port access
Watch the boot up until you find message
Restoring saved persistent reservations. Preparing to restore saved persistent
reservations.
Type skip which will skip the
loading of any persistent reservation keys stored on the array, essential
scrubbing them from the array.
Allow the array to continue to boot
normally. At this time, retry the cluster command to initiate scsi-3
reservations.
The reset on the array needs only be
run once. If any reservations on any hosts exist prior to this procedure, the
host will no longer have access to the lds, and could result in a panic of that
host, or an ejection from the cluster.
Sun
Cluster 3.2 has the following features and limitations:
· Support for 2-16 nodes.
· Global device capability--devices can be shared across the
cluster.
· Global file system --allows a file system to be accessed
simultaneously by all cluster nodes.
· Tight implementation with Solaris--The cluster framework
services have been implemented in the kernel.
· Application agent support.
· Tight integration with zones.
· Each node must run the same revision and update of the
Solaris OS.
· Two node clusters must have at least one quorum device.
· Each cluster needs at least two separate private networks.
(Supported hardware, such as ce and bge may use tagged VLANs to run private and
public networks on the same physical connection.)
· Each node's boot disk should include a 500M partition mounted
at /globaldevices
· Attached storage must be multiply connected to the nodes.
· ZFS is a supported file system and volume manager. Veritas
Volume Manager (VxVM) and Solaris Volume Manager (SVM) are also supported
volume managers.
· Veritas multipathing (vxdmp) is not supported. Since vxdmp
must be enabled for current VxVM versions, it must be used in conjunction with
mpxio or another similar solution like EMC's Powerpath.
· SMF services can be integrated into the cluster, and all
framework daemons are defined as SMF services
· PCI and SBus based systems cannot be mixed in the same
cluster.
·
Boot devices cannot be on a disk
that is shared with other cluster nodes. Doing this may lead to a locked-up
cluster due to data fencing.
The
overall health of the cluster may be monitored using the cluster status or
scstat -v commands. Other useful options include:
scstat
-g: Resource group status
scstat
-D: Device group status
scstat
-W: Heartbeat status
scstat
-i: IPMP status
scstat
-n: Node status
Failover
applications (also known as "cluster-unaware" applications in the Sun
Cluster documentation) are controlled by rgmd (the resource group manager
daemon). Each application has a data service agent, which is the way that the
cluster controls application startups, shutdowns, and monitoring. Each
application is typically paired with an IP address, which will follow the
application to the new node when a failover occurs.
"Scalable"
applications are able to run on several nodes concurrently. The clustering
software provides load balancing and makes a single service IP address
available for outside entities to query the application.
"Cluster
aware" applications take this one step further, and have cluster awareness
programmed into the application. Oracle RAC is a good example of such an
application.
All
the nodes in the cluster may be shut down with cluster shutdown -y -g0. To boot
a node outside of the cluster (for troubleshooting or recovery operations, run
boot -x
clsetup
is a menu-based utility that can be used to perform a broad variety of
configuration tasks, including configuration of resources and resource groups.
Cluster Configuration
The
cluster's configuration information is stored in global files known as the
"cluster configuration repository" (CCR). The cluster framework files
in /etc/cluster/ccr should not be edited manually; they should be managed via
the administrative commands.
The
cluster show command displays the cluster configuration in a nicely-formatted
report.
The
CCR contains:
· Names of the cluster and the nodes.
· The configuration of the cluster transport.
· Device group configuration.
· Nodes that can master each device group.
· NAS device information (if relevant).
· Data service parameter values and callback method paths.
· Disk ID (DID) configuration.
·
Cluster status.
Some
commands to directly maintain the CCR are:
ccradm:
Allows (among other things) a checksum re-configuration of files in
/etc/cluster/ccr after manual edits. (Do NOT edit these files manually unless
there is no other option. Even then, back up the original files.) ccradm -i /etc/cluster/ccr/filename
-o
scgdefs:
Brings new devices under cluster control after they have been discovered by
devfsadm.
The
scinstall and clsetup commands may
We
have observed that the installation process may disrupt a previously installed
NTP configuration (even though the installation notes promise that this will
not happen). It may be worth using ntpq to verify that NTP is still working
properly after a cluster installation.
Resource Groups
Resource
groups are collections of resources, including data services. Examples of
resources include disk sets, virtual IP addresses, or server processes like
httpd.
Resource
groups may either be failover or scalable resource groups. Failover resource
groups allow groups of services to be started on a node together if the active
node fails. Scalable resource groups run on several nodes at once.
The
rgmd is the Resource Group Management Daemon. It is responsible for monitoring,
stopping, and starting the resources within the different resource groups.
Some
common resource types are:
· SUNW.LogicalHostname: Logical IP address associated with a
failover service.
· SUNW.SharedAddress: Logical IP address shared between nodes
running a scalable resource group.
·
SUNW.HAStoragePlus: Manages global
raw devices, global file systems, non-ZFS failover file systems, and failover
ZFS zpools.
Resource
groups also handle resource and resource group dependencies. Sun Cluster allows
services to start or stop in a particular order. Dependencies are a particular
type of resource property. The r_properties man page contains a list of
resource properties and their meanings. The rg_properties man page has similar
information for resource groups. In particular, the Resource_dependencies
property specifies something on which the resource is dependent.
Some
resource group cluster commands are:
# clrt register resource-type: Register a resource type.
# clrt register -n node1name,node2name resource-type: Register
a resource type to specific nodes.
# clrt unregister resource-type: Unregister a resource type.
# clrt list -v: List all resource types and their associated
node lists.
# clrt show resource-type: Display all information for a
resource type.
# clrg create -n node1name,node2name rgname: Create a resource
group.
# clrg delete rgname: Delete a resource group.
# clrg set -p property-name rgname: Set a property.
# clrg show -v rgname: Show resource group information.
# clrs create -t HAStoragePlus -g rgname -p AffinityOn=true -p
FilesystemMountPoints=/mountpoint resource-name
# clrg online -M rgname
# clrg switch -M -n nodename rgname
# clrg offline rgname: Offline the resource, but leave it in a
managed state.
# clrg restart rgname
# clrs disable resource-name: Disable a resource and its fault
monitor.
# clrs enable resource-name: Re-enable a resource and its
fault monitor.
# clrs clear -n nodename -f STOP_FAILED resource-name
# clrs unmonitor resource-name: Disable the fault monitor, but
leave resource running.
# clrs monitor resource-name: Re-enable the fault monitor for
a resource that is currently enabled.
# clrg suspend rgname: Preserves online status of group, but
does not continue monitoring.
# clrg resume rgname: Resumes monitoring of a suspended group
# clrg status: List status of resource groups.
#
clrs status -g rgname
Data Services
A
data service agent is a set of components that allow a data service to be
monitored and fail over within the cluster. The agent includes methods for
starting, stopping, monitoring, or failing the data service. It also includes a
registration information file allowing the CCR to store the information about
these methods in the CCR. This information is encapsulated as a resource type.
The
fault monitors for a data sevice place the daemons under the control of the
process monitoring facility (rpc.pmfd), and the service, using client commands.
Public Network
The
public network uses pnmd (Public Network Management Daemon) and the IPMP
in.mpathd daemon to monitor and control the public network addresses.
IPMP
should be used to provide failovers for the public network paths. The health of
the IPMP elements can be monitored with scstat -i
The
clrslh and clrssa commands are used to configure logical and shared hostnames,
respectively.
#
clrslh create -g rgname
logical-hostname
Private Network
The
"private," or "cluster transport" network is used to
provide a heartbeat between the nodes so that they can determine which nodes
are available. The cluster transport network is also used for traffic related
to global devices.
While
a 2-node cluster may use crossover cables to construct a private network,
switches should be used for anything more than two nodes. (Ideally, separate
switching equipment should be used for each path so that there is no single
point of failure.)
The
default base IP address is 172.16.0.0, and private networks are assigned
subnets based on the results of the cluster setup. Available network interfaces
can be identified by using a combination of dladm show-dev and ifconfig.
Private
networks should be installed and configured using the scinstall command during
cluster configuration. Make sure that the interfaces in question are connected,
but down and unplumbed before configuration. The clsetup command also has menu
options to guide you through the private network setup process.
Alternatively,
something like the following command string can be used to establish a private
network:
# clintr add nodename1:ifname
# clintr add nodename2:ifname2
# clintr add switchname
# clintr add nodename1:ifname1,switchname
# clintr add nodename2:ifname2,switchname
#
clintr status
The
health of the heartbeat networks can be checked with the scstat -W command. The
physical paths may be checked with clintr status or cluster status -t intr.
Quorum
Sun
Cluster uses a quorum voting system to prevent split-brain and cluster amnesia.
The Sun Cluster documentation refers to "failure fencing" as the
mechanism to prevent split-brain (where two nodes run the same service at the
same time, leading to potential data corruption).
"Amnesia"
occurs when a change is made to the cluster while a node is down, then that
node attempts to bring up the cluster. This can result in the changes being
forgotten, hence the use of the word "amnesia."
One
result of this is that the last node to leave a cluster when it is shut down
must be the first node to re-enter the cluster. Later in this section, we will
discuss ways of circumventing this protection.
Quorum
voting is defined by allowing each device one vote. A quorum device may be a
cluster node, a specified external server running quorum software, or a disk or
NAS device. A majority of all defined quorum votes is required in order to form
a cluster. At least half of the quorum votes must be present in order for
cluster services to remain in operation. (If a node cannot contact at least
half of the quorum votes, it will panic. During the reboot, if a majority
cannot be contacted, the boot process will be frozen. Nodes that are removed
from the cluster due to a quorum problem also lose access to any shared file
systems. This is called "data fencing" in the Sun Cluster
documentation.)
Quorum
devices must be available to at least two nodes in the cluster.
Disk
quorum devices may also contain user data. (Note that if a ZFS disk is used as
a quorum device, it should be brought into the zpool before being specified as
a quorum device.)
Sun
recommends configuring n-1 quorum devices (the number of nodes minus 1). Two
node clusters must contain at least one quorum device.
Disk
quorum devices must be specified using the DID names.
Quorum
disk devices should be at least as available as the storage underlying the
cluster resource groups.
Quorum
status and configuration may be investigating using:
# scstat -q
#
clq status
These
commands report on the configured quorum votes, whether they are present, and
how many are required for a majority.
Quorum
devices can be manipulated through the following commands:
# clq add did-device-name
# clq remove did-device-name: (Only removes the device from
the quorum configuration. No data on the device is affected.)
# clq enable did-device-name
# clq disable did-device-name: (Removes the quorum device from
the total list of available quorum votes. This might be valuable if the device
is down for maintenance.)
#
clq reset: (Resets the configuration
to the default.)
By
default, doubly-connected disk quorum devices use SCSI-2 locking. Devices
connected to more than two nodes use SCSI-3 locking. SCSI-3 offers persistent
reservations, but SCSI-2 requires the use of emulation software. The emulation
software uses a 64-bit reservation key written to a private area on the disk.
In
either case, the cluster node that wins a race to the quorum device attempts to
remove the keys of any node that it is unable to contact, which cuts that node
off from the quorum device. As noted before, any group of nodes that cannot
communicate with at least half of the quorum devices will panic, which prevents
a cluster partition (split-brain).
In
order to add nodes to a 2-node cluster, it may be necessary to change the
default fencing with scdidadm -G prefer3 or cluster set -p
global_fencing=prefer3, create a SCSI-3 quorum device with clq add, then remove
the SCSI-2 quorum device with clq remove.
NetApp
filers and systems running the scqsd daemon may also be selected as quorum
devices. NetApp filers use SCSI-3 locking over the iSCSI protocol to perform
their quorum functions.
The
claccess deny-all command may be used to deny all other nodes access to the
cluster. claccess allow nodename re-enables access for a node.
Purging Quorum Keys
CAUTION:
Purging the keys from a quorum device may result in amnesia. It should only be
done after careful diagnostics have been done to verify why the cluster is not
coming up. This should never be done as long as the cluster is able to come up.
It may need to be done if the last node to leave the cluster is unable to boot,
leaving everyone else fenced out. In that case, boot one of the other nodes to
single-user mode, identify the quorum device, and:
For
SCSI 2 disk reservations, the relevant command is pgre, which is located in
/usr/cluster/lib/sc:
# pgre -c pgre_inkeys -d /dev/did/rdks/d#s2 (List the keys in
the quorum device.)
#
pgre -c pgre_scrub -d
/dev/did/rdks/d#s2 (Remove the keys from the quorum device.)
Similarly,
for SCSI 3 disk reservations, the relevant command is scsi:
# scsi -c inkeys -d /dev/did/rdks/d#s2 (List the keys in the
quorum device.)
#
scsi -c scrub -d /dev/did/rdks/d#s2
(Remove the keys from the quorum device.)
Global Storage
Sun
Cluster provides a unique global device name for every disk, CD, and tape drive
in the cluster. The format of these global device names is
/dev/did/device-type. (eg /dev/did/dsk/d2s3) (Note that the DIDs are a global
naming system, which is separate from the global device or global file system
functionality.)
DIDs
are componentsof SVM volumes, though VxVM does not recognize DID device names
as components of VxVM volumes.
DID
disk devices, CD-ROM drives, tape drives, SVM volumes, and VxVM volumes may be
used as global devices. A global device is physically accessed by just one node
at a time, but all other nodes may access the device by communicating across
the global transport network.
The
file systems in /global/.devices store the device files for global devices on
each node. These are mounted on mount points of the form /global/.devices/node@nodeid,
where nodeid is the identification number assigned to the node. These are
visible on all nodes. Symbolic links may be set up to the contents of these
file systems, if they are desired. Sun Cluster sets up some such links in the /dev/global
directory.
Global
file systems may be ufs, VxFS, or hsfs. To mount a file system as a global file
system, add a "global" mount option to the file system's vfstab entry
and remount. Alternatively, run a mount -o global... command.
(Note
that all nodes in the cluster should have the same vfstab entry for all cluster
file systems. This is true for both global and failover file systems, though
ZFS file systems do not use the vfstab at all.)
In
the Sun Cluster documentation, global file systems are also known as
"cluster file systems" or "proxy file systems."
Note
that global file systems are different from failover file systems. The former
are accessible from all nodes; the latter are only accessible from the active
node.
Maintaining Devices
New
devices need to be read into the cluster configuration as well as the OS. As
usual, we should run something like devfsadm or drvconfig; disks to create the
/device and /dev links across the cluster. Then we use the scgdevs or scdidadm
command to add more disk devices to the cluster configuration.
Some
useful options for scdidadm are:
# scdidadm -l: Show local DIDs
# scdidadm -L: Show all cluster DIDs
#
scdidadm -r: Rebuild DIDs
We
should also clean up unused links from time to time with devfsadm -C and
scdidadm -C
The
status of device groups can be checked with scstat -D. Devices may be listed
with cldev list -v. They can be switched to a different node via a cldg switch
-n target-node dgname command.
Monitoring
for devices can be enabled and disabled by using commands like:
# cldev monitor all
# cldev unmonitor d#
# cldev unmonitor -n nodename d#
#
cldev status -s Unmonitored
Parameters
may be set on device groups using the cldg set command, for example:
#
cldg set -p failback=false dgname
A
device group can be taken offline or placed online with:
# cldg offline dgname
#
cldg online dgname
VxVM-Specific Issues
Since
vxdmp cannot be disabled, we need to make sure that VxVM can only see one path
to each disk. This is usually done by implementing mpxio or a third party
product like Powerpath. The order of installation for such an environment would
be:
· Install Solaris and patches.
· Install and configure multipathing software.
· Install and configure Sun Cluster.
·
Install and configure VxVM
If
VxVM disk groups are used by the cluster, all nodes attached to the shared
storage must have VxVM installed. Each vxio number in /etc/name_to_major must
also be the same on each node. This can be checked (and fixed, if necessary)
with the clvxvm initialize command. (A reboot may be necessary if the
/etc/name_to_major file is changed.)
The
clvxvm encapsulate command should be used if the boot drive is encapsulated
(and mirrored) by VxVM. That way the /global/.devices information is set up
properly.
The
clsetup "Device Groups" menu contains items to register a VxVM disk
group, unregister a device group, or synchronize volume information for a disk
group. We can also re-synchronize with the cldg sync dgname command.
Solaris Volume
Manager-Specific Issues
Sun
Cluster allows us to add metadb or partition information in the /dev/did format
or in the usual format. In general:
Use
local format for boot drive mirroring in case we need to boot outside the
cluster framework.
Use
cluster format for shared disksets because otherwise we will need to assume the
same controller numbers on each node.
Configuration
information is kept in the metadatabase replicas. At least three local replicas
are required to boot a node; these should be put on their own partitions on the
local disks. They should be spread across controllers and disks to the degree
possible. Multiple replicas may be placed on each partition; they should be
spread out so that if any one disk fails, there will still be at least three
replicas left over, constituting at least half of the total local replicas.
When
disks are added to a shared diskset, database replicas are automatically added.
These will always be added to slice 7, where they need to remain. If a disk
containing replicas is removed, the replicas must be removed using metadb.
If
fewer than 50% of the replicas in a diskset are available, the diskset ceases
to operate. If exactly 50% of the replicas are available, the diskset will
continue to operate, but will not be able to be enabled or switched on another
node.
A
mediator can be assigned to a shared diskset. The mediator data is contained
within a Solaris process on each node and counts for two votes in the diskset
quorum voting.
Standard
c#t#d#s# naming should be used when creating local metadb replicas, since it
will make recovery easier if we need to boot the node outside of a cluster
context. On the other hand, /dev/did/rdsk/d#s# naming should be used for shared
disksets, since otherwise the paths will need to be identical on all nodes.
Creating a new shared
diskset involves the following steps:
(Create
an empty diskset.) # metaset -s set-name -a -h node1-name node2-name
(Create
a mediator.) # metaset -s set-name -a -m node1-name node2-name
(Add
disks to the diskset.) # metaset -s set-name -a /dev/did/rdsk/d# /dev/did/rdsk/d#
(Check
that the diskset is present in the cluster configuration.)
# cldev list -v
# cldg status
#
cldg show set-name
