[development] Scalability, Load Balancing, and High Availability - Possible SoC Proposal

[Kieran Lal]

On 3/19/07, Scott Hadfield <hadsie at gmail.com> wrote:
>
> Hi All,
>
> I've been thinking about a possible summer of code idea, but I'm not sure
> it will fly so I wanted to get some feedback.
>
> The basic idea is to look into solutions for running Drupal simultaneously
> across multiple servers for the purposes of running extremely large/busy
> sites, for load balancing on multiple servers (both database and web
> servers), or just for high availability (if one server dies, another will
> kick in). The project would consist of developing and documenting ways that
> this can be accomplished and developing tools or other Drupal
> patches/modules that are needed to accomplish the various tasks.
>
> One of the main problems with this as a SoC proposal is that a large
> majority of what needs to be done here is OS/web server/database server
> specific. There would probably be a lot of Drupal specific stuff, but I'm
> not sure exactly how much.
>
> Even if this won't fly as a SoC project I am still be interested on
> working on this. What's already been done in this area? Maybe some of this
> has already been solved?


I think this is a good idea for a project.  Jeremy will be presenting many
learnings from KernelTrap and the CivicSpace ASP at the Drupal Scalability
workshop on Saturday.  The Drupal community has learned a lot, in particular
Gerhard has captured and tested significant performance and scalability
improvements in building the Drupal.org infrastructure.

Having an intern document all these learnings would be great asset for the
community and help in the adoption of Drupal by businesses that ultimately
ask the question, "Will Drupal scale?".

CivicSpace has learned a lot about high availability and scalability in
building our ASP and we'd be interested in sharing what we've learned, which
we have documented fairly well.  I'd be happy to make much of this available
if we believed that work would receive good stewardship.

Topics to consider:

Scaling
1) Scaling web servers horizontally.  Shared file systems for horizontally
scaling web servers.
2) Scaling databases.
3) Performance tuning queries.  Configuring databases for load
characteristics.  Managing IO bottlenecks.
4) Logging - managing logs for scalability and IO tuning

High Availability
4) Network availability - Bonding, CARP
5) Database fail over - replication with master and slaves
6) Restoration - text and binary database restoration, remote recovery

I don't think there is anything with wrong with doing lots of research and
then writing small amounts of code for tools where appropriate.  My
suspicion is that a well prepared project would win wide support and Google
will support a project that has support of the mentoring organizations.

Cheers,
Kieran



Thanks for any feedback,
> Scott




-- 
To strive, to seek, to find, and not to yield.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.drupal.org/pipermail/development/attachments/20070320/bedce0a0/attachment.htm