







                                 LOADBALANCED v1.2


                             (c) 2001-2002 Thor Newman
             












































Introduction
===============================================================================

Loadbalanced (load balance daemon) is a Java application which provides
layer 3 TCP service load balancing over any IP network -- including of course
the Internet. 

This method of balancing TCP services is extremely common in
the enterprise for reasons of scalability and redundancy, and solutions range 
from crude round-robining of DNS to very expensive and sophisticated multilayer 
switches and dedicated load balancing hardware such as Cisco LocalDirectors.

This software is not intended as a replacement to any of them.

Loadbalanced is freeware. You may freely redistribute and use this software for
whatever conceivable use you can find for it, however difficult that may be ;)

Unlike many loadbalancing solutions I have seen, loadbalanced is not limited to
a single Ethernet segment with which to load-balance services to, although its
performance is correspondingly atrocious compared to dedicated hardware 
products. 

Personally I have found loadbalanced useful professionally as a quick and easy
testbed for evaluating applications in a distributed environment before actually 
setting the hardware distributed environment up. Its performance is 
hardly outstanding being Java based, but it is quick and simple to set up and has 
proved stable. 

I should emphasize that this software is hardly a replacement for an 
enterprise-grade hardware product, and is not intended for use in 
high-volume production environments.

If anything loadbalanced should be called 'hobbyware' because it's something I
wrote in my free time as a Java learning exercise. I'd be very interested in 
feedback from anyone who finds this software even remotely useful. Please send
all feedback to:

flubdgub@hotmail.com


Disclaimer / Distribution
===============================================================================

This software is freeware, meaning it is made available for use free of
charge by the author. This software is copyrighted and is the property of the
author, Thor Newman, who reserves and retains all rights to this software.
It may be used and redistributed freely. It may not be resold for profit.

This software is offered as is. No guarantee is made with respect to the
performance or reliability of this product. The author is not responsible
for any and all damages that occur to anyone for any reason as a direct or 
indirect result of using this software. 

By using this software, you implicitly agree to this disclaimer.



Installation
===============================================================================

File List
---------

Loadbalanced should be distributed with the following files:

 - loadbalanced.jar 		[ JAR file of application classes ]
 - loadbalanced.conf 		[ Default configuration file / template ]
 - loadbalanced.txt 		[ This document ]



This software requires a Sun Java runtime environment version 1.3 or later in 
order to run. 

This platform can be downloaded free of charge from http://java.sun.com/ for 
Solaris SPARC and x86, Win32, and Linux.

This software has been tested in JRE 1.3.X and the new 1.4.X.

This software has not been tested with earlier version of Java, and may not
perform reliably, or may not execute at all, with versions earlier than
1.3.

Assuming you have the JRE correctly installed on your platform,
installation of loadbalanced simply requires the following files to be in a
single directory.


Necessary Files
---------------

 - loadbalanced.jar
 - loadbalanced.conf 	[ or other configuration file ]


Place these files in their own directory, such as:

 - /usr/local/bin/loadbalanced 

or,

 - C:\Program Files\loadbalanced

Place them wherever you wish. You can separate the configuration file from the
jar file and pass the path and name of the configuration file to loadbalanced
when you launch it from the command line.

All the loadbalanced classes are contained within the "loadbalanced.jar" file
that should have accompanied this document.





Launching
===============================================================================

Loadbalanced should be run as a daemon. The configuration file
loadbalanced.conf should be configured appropriately for your hardware
(see Configuration).

To launch in Win32:

  [Command Prompt/MSDOS Prompt] - javaw -jar loadbalanced.jar

To launch in UNIX:

  [Shell] - java -jar loadbalanced.jar &

Optionally, you may specify an alternate path and configuration file as a 
command line parameter, and manually pass install directory to the CLASSPATH
if appropriate. For example:

[Assume installed to C:\loadbalanced]
Win32:  

javaw -cp "%CLASSPATH%;C:\loadbalanced" -jar C:\loadbalanced\loadbalanced.jar 
	c:\loadbalanced\lb1.cfg

[Assume installed to /bin/loadbalanced/]
UNIX :  

java -cp "$CLASSPATH;/bin/loadbalanced" -jar /bin/loadbalanced/loadbalanced.jar 
	/etc/lb_server_a.conf &







Output
===============================================================================

If logging has been enabled (it is by default) in the configuration file,
loadbalanced generates a logfile called loadbalanced.log in the directory
loadbalanced is running in. Disabling logging may improve performance but at
the loss of the audit trail that logging provides.

The logging is quite detailed and (hopefully) self-explanatory, I am not going
to attempt to describe all the possible log lines in this document. I should
note however that in addition to logging connections, user commands, service
state transitions, etc, most exceptions are logged here as well, which was
originally put into place as an aid to debugging. The information contained in
the logs is very useful for debugging purposes, as well as providing your 
organization with a quite thorough audit trail of activity involving 
loadbalanced.

Control of logging granularity is done with the LOGGING and THREAD_LOGGING 
configuration entries of loadbalanced.conf.



Input
===============================================================================

Interaction with the daemon is via the command processor. This is a listening
thread that is by default on port 1024 of the local host that loadbalanced is 
running on that can be connected to using any standard telnet application. The
command processor understands various commands for interacting with the daemon,
these commands are listed in the COMMAND PROCESSOR COMMANDS section.

Among other things, the command processor allows for the cleanest possible
shutdown of the daemon once launched, by issuing a 'halt' command.

Note: If you are using the horrible stock pre-2000 Windows Telnet, be sure to 
enable "Local Echo" in it's configuration before using the command processor. 
No carriage returns will happen if you don't enable this, and the output looks
a tad messy. Enabling this feature does not seem to be necessary on any other 
telnet client I have used.

Note 2: The command processor address _cannot_ currently be specified, it 
essentially grabs onto the 'most prominent' address, typically corresponding
to the hostname of the local machine it is running on.



Overview
===============================================================================

There are two main components involved in loadbalancing connections. First,
you must have a pool of remote services defined. These might be standard
TCP services such as SMTP, HTTP, etc, or custom ones listening on custom
ports. Loadbalanced monitors active services, sending keepalives to determine
if the service appears to be available and functioning. Services which fail
keepalives will be marked as down, and connections will not be sent to them.
The service will automatically change back to active state when it is
running properly again.

Next you must define virtual servers (virtuals) which run on the loadbalanced
server bound to IP addresses attached to the underlying operating system and
configured appropriately in loadbalanced. Virtuals accept connections and
round-robin them to the services which are added to the virtual server in
the configuration file. Typically, the virtual server becomes the remote
client target; in the case of a website, the website DNS entry would resolve
to the IP address defined in the virtual server for that service. The client
connects to the virtual server and it's connection is automatically balanced
to an active web server defined in the virtual server configuration.
Obviously the content served up by all the loadbalanced webservers has to be
100% identical, otherwise the client will get different content depending on
which server it is directed to.

All in all, this is pretty basic stuff. If it's not making any sense to you
so far then probably this isn't something you should be tackling, or maybe
I'm just horrible at writing docs.

This concludes the general overview. Specifics of setting up a hypothetical
load balanced server farm are presented below.


Configuration
===============================================================================

Configuring loadbalanced to loadbalance a service is quite easy. All
configuration is done in the configuration file loadbalanced.conf, or
whichever configuration file you wish to use. The loadbalanced.conf file
which accompanies the distribution is the template you should work from,
modify it appropriately. It has all the configurable system variables set
to defaults and has some sample service and virtual configurations.

There are three sections to loadbalanced.conf, and the order the sections
appear in the file is important. All commands are case sensitive. The
configuration file parser will generate an exception for commands it does not
understand and halt execution. Text is evaluated on a line by line basis, not
character by character; an entry cannot be split across multiple lines.


The first section covers APPLICATION VARIABLES, which configures parameters
of loadbalanced itself.

The second section defines SERVICES, which are remote services you intend
to direct loadbalancing to.

The third section defines VIRTUALS (virtual servers) which are services that
reside on loadbalanced. Remote clients connect to the virtuals, and are then
automatically loadbalanced to appropriate services.

The specifics of all commands available in the configuration file are in
section CONFIGURATION COMMANDS. This section will present a hypothetical
scenario and walk through configuration of loadbalanced to accomplish it.

For this scenario we have three servers, one in London, one in New York, and
one in San Jose. Servers A, B, and C. Servers A & B are standard web servers
listening on port 80 and serving up completely identical content for the
website www.blah.com. Server C runs loadbalanced.

First you must define two services, one for server A and one for server B.
These services should point to their respective server IP addresses and on
the appropriate port, which in this case is 80.

The syntax for a service definition is:

service <name> {
        address <ip address>
        port <service port>
}

The name must be a string with no spaces. Commands are case sensitive. The
opening and closing curly braces are very important.


Assume server A has IP address 1.1.1.1 and server B has address 2.2.2.2.
These services would be entered in the configuration file as follows.

Example:

service server_a {
        address 1.1.1.1
        port 80
}

service server_b {
        address 2.2.2.2
        port 80
}

Next you must define a virtual server. The virtual server definition syntax
is as follows:

virtual <name> {
        address <local IP address>
        port <port>
        add <service 1>
        add <service 2>
        .
        .
        .
}

The name must be a string with no spaces, and is case sensitive. The name
can contain special characters such as '.' or '-', and can be any 
reasonable length. A good convention would be to descriptively name the
virtual to associate it with the service it is for. In the case of a
website, it makes sense to name the virtual the same as FQDN of the
website.

It is the responsibility of the underlying operating system to provide the
appropriate IP address for the virtual server to bind to. So assuming you
the hardware running loadbalanced has IP address 3.3.3.3, and it has
TCP/IP reachability via the Internet to server_a and server_b,
you would configure the virtual server as follows:

Example: 

virtual www.blah.com {
        address 3.3.3.3
        port 80
        add server_a
        add server_b
}


Launching loadbalanced with this configuration in your configuration file
will generate a virtual server listening on 3.3.3.3 port 80. If you direct
a web browser to this IP address, the connection will automatically be
round-robin loadbalanced to server_a and to server_b. If one server becomes
unavailable (taking the web server down for maintenance for example, or
network reachability problems) the service will be flagged as down and
traffic will be directed to the remaining server. Of course you may add
as little as one or as many as a hundred defined services to a virtual
server configuration, there are no hard limits set in loadbalanced. If you run
out of RAM I guess you've configured too much!

It should be noted that you may re-use a single IP address many times in a
virtual definition -- so long as each virtual has a different port setting,.

Example:

virtual www.blah.com {
	address 3.3.3.3
	port 80
	add server_a
	add server_b
}

virtual secure.blah.com {
	address 3.3.3.3
	port 443
	add server_c
	add server_d
	sticky 15
}

In the example above, regular http connections to port 80 at address 3.3.3.3
will be directed to the services defined in server_a and server_b. SSL 
connections to port 443 at address 3.3.3.3 will be directed to the services
defined in server_c and server_d. 



CONFIGURATION COMMANDS
======================

This is a list of all commands available in the configuration file, with a
description of their meaning. 

* Remember commands are case sensitive.

Application Variables Commands
------------------------------

- LOGGING [true or false]
  The main application will log information to the file loadbalanced.log
  in the main running directory. The default value is 'true'.

- THREAD_LOGGING [true or false]
  Threads will log information to the file loadbalanced.log. Turning off
  THREAD_LOGGING may improve performance. The default value is 'true'.

- COMMAND_PORT [Integer TCP/IP port number]
  This is the port the command processor listens on. The command processor
  allows you to interact with loadbalanced while it is running. See
  the COMMAND PROCESSOR COMMANDS section for more information. The default
  port is 1024.

- COMMAND_TIMEOUT [milliseconds]
  An idle connection to the command processor will automatically terminate
  after the amount of time specified here, in milliseconds. The default
  value is 300000, or 5 minutes.

- PASSWORD [ string ]
  This is the password you use to login to the command processor. This may be
  any string, although it may not contain spaces. The default password is
  'loadbalanced'.

- PROMPT [ string ]
  This is the prompt you will be presented with while logged into the command
  processor. This may be any string, although you cannot use spaces. The
  default value is '[loadbalanced:]'



Services Commands
-----------------

This is a list of all the commands available in a service configuration
block. The syntax for this block is:

service <name> {
        <commands>
}

 - address < remote ip address >
   This is the IP address of the remote server you wish to direct traffic to.
   This can be any reachable IP address on a TCP/IP network.

 - port < remote port >
   This is the port the remote service is listening on, for example 80 for
   standard http.

 - shutdown
   Put this command in the configuration if you wish this service to be
   administratively shutdown when loadbalanced launches.

 - keepalive_timeout < seconds >
   This command can be used to modify the default keepalive connection timeout 
   of 2 seconds

 - keepalive_frequency < seconds >
   This command can be used to modify the default keepalive frequency of
   5 seconds

 - keepalive_maxfails < number >
   This command can be used to modify the number of consecutive failed checks
   required before a service will be marked as down. The default is 2.

A note on configuring keepalives
================================
A service which is not in 'shutdown' state while loadbalanced is running will
automatically be checked to see if there is a service listening at the
defined IP address on the defined port. This keepalive is very simple, it
simply establishes a TCP connection to the remote service. If a connection
is successful than all is well; if a connection cannot be established, this is
noted. If 2 consecutive connections fail, the service will be flagged as
down, meaning loadbalanced will not direct connections toward this service.
It will continue to monitor the service with keepalives however, and bring
the service backup should the remote service begin responding to keepalives.

The keepalive logic is controlled by three system variables which can be
configured individually in each service.

These variables are:

 - keepalive_timeout <seconds>
   The amount of time (in seconds) that an attempt to establish the
   keepalive connection will go on for; the default is 2 seconds.

 - keepalive_frequency <seconds>
   The amount of time (in seconds) that separates checks . The default is
   15.

 - keepalive_maxfails
   The number of consecutive failed keepalives that must occur before a
   service will be marked as down. The default is 2.
 

Virtual Server Commands
-----------------------

This is a list of all the commands available in a virtual server
configuration block. The syntax for this block is:

virtual <name> {
        <commands>
}

 - address < ip address >
   This is the local IP address that the virtual server binds to. This
   address should be configured and available on the underlying operating
   system loadbalanced is running on. This is important, loadbalanced does
   not provision configured IP addresses, it just assumes they are there. A
   BindException will be thrown and noted in the log if a virtualserver is unable
   to bind to it's configured IP address -- the affected virtual server will be put    
   into shutdown state automatically.

 - port < remote port >
   This is the local port that the virtual server listens on.

 - add < service name >
   This adds a remote service (defined previously) to the virtual server to
   direct traffic to. The name of the service is case sensitive, and is the
   same string as is used to in the <name> part of the service configuration.

 - sticky < # of mins >
   This causes connections to be 'sticky' for the number of minutes
   specified. A sticky connection means that after a connection has been
   established and sent to a particular service, additional connections from
   the same source IP address will be directed to that same service rather
   than being directed to other services added to the virtual server. This
   behavior persists for the length of time specified by the 'sticky' 
   statement. When the sticky time expires, the next connection will be 
   round-robined, and then that destination will become sticky.

   This is essential in particular for loadbalancing SSL connections (secure 
   web servers).


COMMAND PROCESSOR COMMANDS
===============================================================================

The command processor allows monitoring and interaction with the loadbalanced
daemon while it is running. It allows connections monitoring, allows you to
view the configuration and status of services and virtual servers configured
on the current running iteration of loadbalanced. It also allows the cleanest
method of shutting down loadbalanced.

The command processor is interacted with via telnet, to the port defined by
COMMAND_PORT in the configuration file. The default port is 1024. You will
be greeted with a Password prompt, and you must enter the string defined by
the PASSWORD variable in the configuration file. The default password is
'loadbalanced'. Only one connection to the command processor at a time is
possible. There is no encryption.

 - show version 
   This command displays version, stats, platform, and runtime information.

 - show virtual
   This command shows the configuration of virtual servers running on
   loadbalanced. It also shows their state ( active or shutdown ).

 - show services
   This command shows the configuration of services configured on
   loadbalanced. It also shows their state ( active, shutdown, or down )

 - show service summary
   This command displays summary information of currently configured services
   on loadbalanced.

 - show connections
   This command displays information on current connections to loadbalanced.

 - show sticky
   This command will list all active 'sticky' caches (see Virtual Server Config)
   displaying name / remote ip address / time to live (in seconds)

 - logout
   This command logs out of the command processor, closing the connection.

 - halt
   This command will cleanly shutdown the loadbalanced daemon.

 - clear sticky <remote ip address>
   This command will clear a sticky cache for the specified remote ip address.
   The affected sticky object is immediately expired and a subsequent connection
   from the client will follow normal round-robin logic, and generate a new 
   sticky connection as per normal.

 - clear connection <remote ip address>
   This command is not yet implemented ;)

 - active service <name>
   This command will change a service which is administratively shutdown to
   active; the service is then available for use, assuming it passes
   keepalive checks. The name string is case sensitive.

 - active virtual <name>
   This command will change a virtual server which is administratively
   shutdown to active. This means the virtual server will start accepting
   connections and load balancing them. The name string is case sensitive.

 - shutdown virtual <name>
   This command will change the state of an active virtual server to shutdown.
   A shutdown virtual server halts it's listening thread and will not accept
   connections.

 - shutdown service <name>
   This command will change the state of an active or down service to
   administratively shutdown. A shutdown service will not have traffic
   directed to it, and will not generate keepalives. This is a 'soft'
   shutdown in that existing connections to the service are not disconnected
   when the service is shutdown, but are allowed to gracefully exit.




Revision History
===============================================================================

January 6th 2001      1.0     	Released (ha ha)

April 4th 2001	      1.1	Fixed multiple keepalive instance bug

Feb 19 2002	      1.2	This minor release addresses a number of small 
				bugs, including an annoying bug related to 
				Sticky connections to services that fail after 
				the Sticky cache is created. While was at it I 
				added the 'show sticky' and 'clear sticky 
				<addr>' statements	


Development & Platform Notes 
===============================================================================

Loadbalanced was developed using Sun's Java SDK1.3 for Win32, on a PentiumII
workstation running Windows 98 (on my gaming PC to be exact). Microsoft Notepad 
was the 'IDE'. You might think I'm crazy for developing in something as limited 
as Notepad but at least it doesn't crash.

Development to 1.0 took approximately four months, including one month where I 
ignored it completely. Development was done sporadically in my spare time. I'd 
put in a few good nights then ignore it for a week. This was a programming 
challenge/activity, not something I put together to make a buck or change the 
world. My decision to release the software freeware is in the hopes that it will 
perform well enough in the real world to be a useful tool for small to medium 
sized IT environments. As noted in the introduction, I would be very interested 
to hear the sort of situations people are using this software in, and how well 
it works for them, and how it performs on different types of hardware and how 
well it holds up to high use. I was striving for stability, not speed, and was
determined to avoid thread deadlocks. Loadbalanced is a very thread intensive
application.

Subsequent to 1.0 I have only looked at this code occasionally, fixed bugs
which I found or were reported to me, tinkered, etc.

I don't pretend or claim that this software serves any useful purpose, however
it has been a useful testbed tool for me from time to time professionally, and
writing it was certainly an interesting learning experience in multi-threaded
server programming.

The one major improvement that is needed is an overhaul of the command 
processor code, which basically grew as a debugging tool during development.
As a result the code is functional but crude and disorganized.
 
Development of a completely new CommandProcessor.class, to follow IOS-style
conventions, is in the works but it's hobbyware and isn't happening all that 
quickly ;)

Bug reports and thoughts on future additions to the code would be appreciated 
too but I am not promising to break my neck investigating/implementing them. 
Again, send such feedback to:

flubdgub@hotmail.com

With regard to platform recommendations, loadbalanced has been tested/used by 
me on the following platforms. 

 + Solaris SPARC 8 / JRE 1.3.X , 1.4.0  
 + Solaris SPARC 2.6 / JRE 1.3.X
 + Solaris x86 8 / JRE 1.3.X

 + Windows XP / JRE 1.3.X , 1.4.0
 + Windows 2000 / JRE 1.3.X
 + Windows NT4 / JRE 1.3.X
 
 + Red Hat Linux 2.2.X / JRE 1.3.X

My opinion is that Solaris by far provides the most efficient Java environment.


Thor Newman
flubdgub@hotmail.com



