[consulting] Distributed Loads, Failover and Dynamic Content

[Boerland, Bert]

This is a very interesting thread. Let me chip in my experience for Global Loadbalancing.

First lets defined Global Loadbalancing; it is not a failover between two datacenters, nor is it load /sharing/ or balancing the load over a cluster at one location. Global Loadbalancing is /balancing/  over more nodes between multiple geographical separated areas, preferably in more autonomous systems (AS-es) and preferably giving the client the /closest/ available node.

Basically there are three problems to overcome:
*	Frontend: give the client an available node that is least busy and is closest to the enduser 
*	Middleware: having the static content synced over multiple geographical separated nodes 
*	Backend: having the dynamic content synced over multiple geographical separated nodes 

The second one is the one where people focus on, since this one is most easy solved. You might sync you can use rsync but the more static dynamic content one has, the bigger a problem this will become. With static dynamic content I am referring to uploaded documents in the CMS it self, for example pictures in a gallery. If one wants a real solution here, one might try GFS. In fact, if you don't want SAN replication and your datacenter are more then 10 kilometers apart from each other, GFS is one of the only ways to sync this since all GFS nodes in the cluster are peers, it is a shared storage journaled cluster file system and really rocks your socks of. Rsyncing is just not up to this IMHO.

The same holds true for the backend, database replication as suggested doesn't do it. You need database synchronization / clustering. And this is by no means simple (http://www.mysql.com/products/database/cluster/ ), or cheap (think Oracle RAC).

The frontend is something that some people will do with Round Robin DNS. However, this doenst do any checking if the site is available, it doenst balance (it does sharing) and it doenst deliver AS wise the shortest path. For this to do you need either a global loadbalncer (appliances like BigIP) or go wild with Linux HA, Ultramonkey and custom BIND.

However, you really should aks your self, what is my problem and what am I trying to solve. If you just want high availability and vertical scaling, you /might/ need multiple frontend/backend in a clustered form at one location. Do you really need multiple datacentra? I think you will find out that the chance that one good (BGP connected to the internet and having multiple no breaks) datacenter will fail is next to null. Compare this with the extra complexity of having Global Loadbalancing, including all operational cost and you will most likely just use one datacenter with multiple nodes in a cluster in it. Unless you are google...