Timothy Parker Consulting Incorporated

Clustering For Protection

If you’ve been reading the high-performance Unix and Windows NT media lately, you can’t help but come across discussions of clustering. Microsoft has developed Wolfpack to allow clustering of Windows NT servers. A plethora of software products are available for Unix servers, such as 1776’s Fault-Freedom II and Qualix Group’s QualixHA+. If you run servers that must remain available at all times, it’s time you considered clustering, too.

It really doesn’t matter what size company you run: availability is the issue here. A single-person company that depends on the Web for sales realizes that every minute the system is down is lost sales. That’s no different than a 1,000 person corporation that requires access to databases for all employees with no down-time. Clustering can solve both these issues in exactly the same way.

Clustering is not a huge market at the moment. According to Strategic Research Corporation (Santa Barbaba, CA) 1997 clustering sales were about $160 million. However, the market is growing exponentially, especially as hardware prices drop and more companies rely on smaller Intel-powered servers instead of minis and mainframes. Strategic Research Corporation is projecting sales by 2000 of $1.15 billion. Clustering is becoming important to practically every serious operating system. Windows NT has it and practically every version of Unix has it, including SCO UnixWare and SCO OpenServer.

What is clustering?

High-availability subsystems have been available for most operating systems for years. For example, RAID (Redundant Array of Inexpensive Disks) is designed specifically to allow your system continue servicing disk requests even when one or more disk drives in a RAID array fail. SCSI controller cards like DPT’s SmartRAID IV and ICP Vortex’ Disk Array Controller allow you to set up complete redundancy for disk drives. These even go as far as automatic failover capabilities to a second or third SCSI chain in case one chain develops critical faults. Complete disk subsystems like DPT’s ServerOnCall and RAIDStation allow two servers to access the same RAID subsystem, so when one machine fails the second can keep data service going.

Other examples of redundant subsystems abound. It’s not unusual to find two power supplies in a server. ALR’s Revolution 2XL, for example, keeps two power supplies running at all times. In case of a failure of one power supply, the second is capable of maintaining the entire system. Multiple processor motherboards can function similarly. A failure of one CPU does not have to shut down the entire motherboard, with the other CPUs taking up the load.

All of this subsystem protection covers you for all but one major failure: the entire system. System failures can be for any reason, such as memory chip or motherboard failures, network card failure, or peripheral problems. While some cases can be handled with redundant subsystems, some subsystem failures will crash the entire machine. It’s exactly for this reason, complete system failure, that clustering was developed.

The easiest way to think of clustering is as a hot standby. In the older days of minicomputers, hot stand-bys were common. If you were running a larger organization that had dozens of minicomputers deployed, it was common to find one or two warmed up and ready to go in case one of the primary servers failed. A quick software load may have been necessary, but the hot standby

The simplest type of clustering uses a twin machine, called a mirror, of a primary server. The mirror contains exactly the same software (and usually the same hardware) as the primary, and is kept up to date at all times through network transfers. The only difference between the two machines is that the mirror does not respond to network requests that are directed at the server. If the primary fails, software on the mirror detects the failure and starts up all the functions on the mirror. The network IP address of the mirror becomes the same as that of the failed primary, so any network requests are handled by the mirror exactly as though it was the primary. As far as users and applications on the network are concerned, there is no difference between the primary and mirror, and there’s no way for anyone to tell which is servicing network requests. Often the only sign there is a fail-over to the mirror is a very short wait for some services to respond to network requests. It’s special software on the primary server and the mirror that handle the fail-over and ensure the two machines are identical.

A slight variation on this simplest cluster theme is to have the two servers mirror each other. If you have a Web server and an application server on your network, for example, each can mirror the other. If the Web server dies, the application server handles its own load as well as the Web servers’. This can be extended to allow almost every machine on the network to have a mirror.

Instead of dedicating a machine to act as a mirror and not process any requests except when a fail-over occurs, most clustering software allows any machine on the network to mirror any other. When a Web server fails, for example, a user’s workstation may take up the slack. Depending on the clustering software, more than one workstation may spread the load between themselves to prevent any one machine getting bogged down. The better clustering software can detect a failure on any monitored machine then decide which machines are best able to take up the failed machine’s tasks, and whether to spread the load across more than one machine. All the basic tasks of changing IP addresses, starting daemons, and ensuring that every application that was running on a failed machine is started elsewhere is handled by the clustering software.

Why use clustering?

Clustering is a scary concept to some system administrators because or the perceived front-end setup and configuration problems, and the load that can be imposed on a network. It is true that clustering imposes some overhead on a network, as primary and mirrors keep up to date, but this can often be done on a dedicated separate network if you are assigning specific mirrors. As for the heavy front-end work, that’s a myth that has to be put to rest. Commercial software today makes setting up a cluster simple and relatively fast.

Let’s take a small company network. Assume there are twenty employees all using Unix workstations or Windows 95 machines. There’s a primary server used by the company to handle application requests and perhaps to act as a fileserver. If one of the user’s workstations has the horsepower to act as a fail-over for the server, the clustering software can be installed and configured in a matter of two hours. For larger networks with multiple servers and flexible failover systems, time involvement becomes higher, but not by much. Even for high-end networks which use very flexible clustering and high-availability software, an entire network can be configured in half a day. Surely that’s worth the effort to ensure servers are always available.

The cost of setting up a cluster is primarily in the clustering software. 1776’s Fault-Freedom II, for example, costs about $4,000 for a simple two-machine cluster, with prices rising as more machines are handled. Even though that’s an appreciable amount of money, it’s cheaper than purchasing two hot standby machines, and has a lot more flexibility.

Choosing clustering software

So you’ve decided to buy clustering software to protect your servers. Smart move. Now, the hard question is which clustering software do you choose. There are quite a few clustering products available, depending on the type of hardware that will be clustered. You must decide if you want to mirror entire machines, or just services. If you only want to ensure that a Web server daemon is always running, that involves different software than most clustering systems offer. For service and machine protection, expensive software like Qualix Group’s QualixHA+ (the HA stands for High Availability) protects specific services and drives, as well as complete machines, through clustering.

There are a few things you should look for when choosing clustering software. The first is one that has been designed specifically for clustering, and not as an add-on thought to some other networking product. Chances are the latter approach won’t offer the same reliability as a dedicated cluster software package. Of course, the software should be designed specifically for your operating system. Buying UnixWare clustering software for your OpenServer system just doesn’t work right. Neither does buying generic clustering software intended for "any Unix system". Find something that has been tested and approved on your hardware and software combination. Some clustering software requires hardware for the primary and mirror to be identical, which other packages are not particular about the peripherals and CPU.

Equally important is performance. There is no point installing clustering software if your network and server drags down because of it. Ideally, clustering software will have a minimal impact on your server and network. Some clustering software help boost performance by using primary and mirror as a dual read system, so that network requests can be handled by either machine, eliminating some server load. A mirror is only good if the mirror contains and exact duplicate of the server at failure time. If the clustering software doesn’t keep both machines identical with no delay, then a mirror won’t do you much good. The key limitation in maintaining synchronization between primary and mirrors is often TCP/IP, although a dedicated connection can help (even over a serial port), and some software uses TCP to its advantage to speed throughput.

You need software that will let you mirror all or part of an entire server, with the mirror taking over all functions of that server in case of problems. It’s not good enough to require you to manually change IP addresses, or tell the mirror that the time has come for it to start up. This should be automatic. If you need the ability to mirror more than one server, make sure that the same package will perform those functions. Don’t buy multiple copies of a two-machine cluster package to mirror more than one server. It doesn’t work that way!

Make sure that in case of a failure the mirror kicks in quickly. A five minute wait might as well be hours, as far as most users are concerned. Typically, clustering software should allow fail-over to a mirror in seconds. When a failure does occur, your software should let you know instantly. Better packages provide not only workstation window notification, but can also page you or call your cell phone!

Ideally, your software will have simple installation and configuration routines. Setting up clustering is not easy if you have to do it all manually. Scripts should be available to perform all the tasks for you, leaving you to worry about which machines and drives are to be mirrored. Check out both the documentation and support service. If you’re spending big bucks on clustering software, you should get a decent manual and telephone support thrown in.

For most SCO users, the choice is pretty simple: 1776’s Fault-Freedom II or SCO UnixWare Reliant HA. SCO does not have any particular plans to introduce clustering software for SCO OpenServer, but does have software for UnixWare. Introduced in 1997 as SCO ReliantHA, the product was aimed at OEMs. SCO UnixWare Reliant HA provides application high-availability failover clustering for two, three, or four nodes. SCO UnixWare Reliant HA includes applets which allow you to describe an application, its start-up behavior, and any dependencies on other applications, resources, and hardware. You also define what should occur when the application (or one of its dependencies) fails. Typically, you can restart the application or resource, switch paths to another disk or application on the same server, or in case of catastrophic server failures, rely on a fail-over to another node. SCO UnixWare Reliant HA is complex to set up and configure properly, but there are sample configurations available for commonly used applications such as Oracle, NetScape Web servers, NFS, and so on. SCO UnixWare Reliant HA has been updated for UnixWare 7 and is to be released with the name UnixWare 7 ReliantHA Release 1.1.

SCO has been working with Tandem (now a division of Compaq) on the release of Tandem Non-Stop Clusters for SCO UnixWare 2.1. Tandem's Non-Stop Clusters for UnixWare provide for multiple node Single System Image clustering for system high-availability and scalability using Tandem’s ServerNet hardware. At least for now the system is aimed almost exclusively at the telecom market, although wider marketing may be used later to include all SCO UnixWare customers.

For SCO OpenServer customers, 1776’s Fault-Freedom II is the most widely used clustering software. Fault-Freedom II was the first clustering package available for OpenServer, too. Fault-Freedom II allows true mirroring of data from a server to a dedicated mirror, or to a machine that is a user workstation. Data on the two machines are kept identical either through the normal network or through a dedicate connection between the two. A standard serial line or a dedicated local network can be used for this task. Fault-Freedom II allows all or part of a server to be mirrored. You may decide not to include all the disks in the server in the mirror, only a few partitions or filesystems. This will often be handy if the mirror machine is a user workstation as well, where a complete image of the server is undesirable and conflicts with the user’s requirements. In these cases, mirroring just the server partitions that provide particular functions, such as user data storage or Web service, is enough.

Fault-Freedom II allows the hardware and software loads of the primary and mirror to be different. It’s possible that a primary that consists of state-of-the-art hardware with fast CPUs and Ultra SCSI drives, while the mirror is an older Pentium with IDE drives. Fault-Freedom II doesn’t care about the hardware, and to a large extent the software, as long as the operating system is the same. Fault-Freedom II allows mirrors to be physically removed from the primary, including elsewhere in the country as long as a data link is maintained between the two. Fault-Freedom II provides flexibility to meet almost any requirement a SCO system administrator could wish for.

The competition

SCO isn’t the only operating system that offers clustering, of course. The primary competitor for SCO’s UnixWare and OpenServer platforms is Microsoft’s Windows NT. In addition, Novell is expected to announce clustering support built into their Modesto 64-bit operating system designed for the Merced CPU to be released by Intel in 1999.

Microsoft’s Wolfpack is the clustering package Microsoft offers for NT, and it performs the same tasks. In case a primary goes down, a mirror machine picks up the slack. So far, Wolfpack is largely untested, coming out of beta only lately. Preliminary tests show it does work well, though, and does offer the benefits of clustering to users of Windows NT. It is not possible to mirror Windows NT machines to other platforms, and data drives can’t be mirrored to other platforms, either, which is possible with NFS under Unix.

Other Unix versions have clustering software, such as QualixHA+, although they tend to be very expensive packages when compared to Fault-Freedom II. These are designed for Sun and HP departmental servers, and provide mirrored failover of entire machines or services to one of more mirrors. Mirrors with packages like QualixHA+ can be any machine, with the software polling continually to find the best way to spread a failed machine’s load. QualixHA+ is clever software, but is out of the league of most SCO system administrators.

Setting up a cluster

To show the procedure for setting up a cluster for a SCO OpenServer network, we created a ten machine network that used two SCO OpenServer servers, one for Web services and the other as an application and NFS server, four Windows 95 clients, and four SCO OpenServer workstations running on older Pentium machines. We chose 1776’s Fault-Freedom II as our clustering software. After the network was installed and working properly, we started the installation of Fault-Freedom II. Fault-Freedom II allows two servers to be mirrored, with up to 32 slices (partitions) of disk space to be mirrored.

The Fault-Freedom II software installs in about five minutes using scoadmin. From there, the longer process of configuration begins. The routine is simplified enormously by Fault-Freedom II’s scripts, which guide you through most parts of the process. After specifying the slices of disk space and the services that are to be mirrored, the process is complete except for setting up the mirrors. Each server disk slice is mirrored to a particular target machine, and we mixed the targets so the other server got most of the load but two user workstations acted as mirrors for the user data partitions. The second server was set up in the same way, so we had two servers each acting as mirrors of each other, and some slices of disk from each server mirrored to user workstations. Total configuration time, including the learning process, was about three and a half hours.

We started up the system and network and noticed no appreciable effect on the server performance itself because of the Fault-Freedom II software. We did notice an impact on the network when we stress-tested the servers with repeated disk writes from the eight clients. Because the two servers need to write each data file to its mirror right away, the network took a major hit in performance. While this was not noticeable in normal use, as soon as we started to force the servers with multiple writes, network performance suffered. This could be solved by using a faster network, but we took the easy approach and installed another two network cards and connected the two servers to each other. Telling Fault-Freedom II to use the dedicated network was simple, and the network load dropped back to normal. All in all, a simple setup, made easier by excellent scripts.

When we failed either of the servers, a user working on a workstation (either Windows 95 or Unix) noticed no delays in service of requests, other than perhaps the slightest hesitation. When properly failed over, the mirror kicked in with full service request take-up, and paged us with a message about the failure. We’re not sure what else you could want from the software.

Is clustering for you?

Whether you choose to implement clustering depends on how tolerant your network is of server failures, and how important high-availability is to your system. Balance the need for high availability against the cost of the clustering software, and you can decide on that basis. In most cases, clustering requires no special hardware, only the clustering software itself. For larger networks, a powerful enough mirror should be available to prevent the system crashing in a failover because the mirror server can’t keep up, but other than that, hardware requirements are simple. A dedicated network between server and mirror is a good idea, and quite inexpensive to create.

For our test network, the total cost of setting up our two clustered servers was $3,950 for Fault-Freedom II and $200 for two network cards and a run of coax. Time commitment was about six hours total. No bad for the peace of mind clustering brings. It won’t replace our RAID arrays, but we do now know that we can offer services to the network day and night, barring a complete failure of both servers. Great insurance.

Sidebar: 1776’s Fault-Freedom II and DPT’s ServerOnCall

Although both products accomplish the same end result of providing clustering capabilities to handle hardware failure, both 1776’s Fault Freedom II and DPT’s ServerOnCall go about it in different ways.

DPT couples two machines through SCSI interface cards to a separate hard drive subsystem (which itself uses RAID for redundancy). Both machines can use the subsystem at the same time, although there may be conflicts if both have the same IP address configuration. This is usually solved by having separate boot drives for each server, and the data or application filesystems on the RAIDstation subsystem. If the ServerOnCall software detects a failure of one machine, the other can take over all the serving requests (such as HTTP requests) from the failed machine.

1776’s Fault-Freedom II is entirely software based and relies on the software to notify a backup server when the primary experiences problems. In that case, the backup can take over all the tasks of the primary.

There is quite a pricing difference between the two software products with ServerOnCall selling for $540 and Fault-Freedom II selling for $3950. Of course, if you factor in the thousands of dollars that the DPT RAIDstation subsystem adds to the DPT package, there’s no comparison. However, when a full configuration of two servers is compared head-to-head with RAID arrays implemented on both, the two products are fairly closely priced.

Configuration of the Fault-Freedom II system is more complicated and involved than ServerOnCall, but there is more flexibility with Fault-Freedom II. Instead of physically tying two machines together with SCSI cables, Fault-Freedom II allows an unlimited number of machines to be clustered for backup purposes. If there is a problem with Fault-Freedom II it’s that synchronization between primary and backup servers can increase network traffic considerably, while there’s almost no network traffic with ServerOnCall and RAIDStation because of the shared disk subsystem. In testing, both software packages worked without a hitch so there’s no reliability issue involved in choosing between the two. We’d be happy with either package as neither is a compromise, just two different ways to get the same overall result.

Sidebar: Setting up a cluster with 1776’s Fault Freedom II