On 3/19/07, Scott Hadfield <hadsie@gmail.com> wrote:

Hi All,

I've been thinking about a possible summer of code idea, but I'm not sure it will fly so I wanted to get some feedback.

The basic idea is to look into solutions for running Drupal simultaneously across multiple servers for the purposes of running extremely large/busy sites, for load balancing on multiple servers (both database and web servers), or just for high availability (if one server dies, another will kick in). The project would consist of developing and documenting ways that this can be accomplished and developing tools or other Drupal patches/modules that are needed to accomplish the various tasks.

One of the main problems with this as a SoC proposal is that a large majority of what needs to be done here is OS/web server/database server specific. There would probably be a lot of Drupal specific stuff, but I'm not sure exactly how much.

Even if this won't fly as a SoC project I am still be interested on working on this. What's already been done in this area? Maybe some of this has already been solved?

I think this is a good idea for a project. Jeremy will be presenting many learnings from KernelTrap and the CivicSpace ASP at the Drupal Scalability workshop on Saturday. The Drupal community has learned a lot, in particular Gerhard has captured and tested significant performance and scalability improvements in building the Drupal.org infrastructure.

Having an intern document all these learnings would be great asset for the community and help in the adoption of Drupal by businesses that ultimately ask the question, "Will Drupal scale?".

CivicSpace has learned a lot about high availability and scalability in building our ASP and we'd be interested in sharing what we've learned, which we have documented fairly well. I'd be happy to make much of this available if we believed that work would receive good stewardship.

Topics to consider:

Scaling
1) Scaling web servers horizontally. Shared file systems for horizontally scaling web servers.
2) Scaling databases.
3) Performance tuning queries. Configuring databases for load characteristics. Managing IO bottlenecks.
4) Logging - managing logs for scalability and IO tuning

High Availability
4) Network availability - Bonding, CARP
5) Database fail over - replication with master and slaves
6) Restoration - text and binary database restoration, remote recovery

I don't think there is anything with wrong with doing lots of research and then writing small amounts of code for tools where appropriate. My suspicion is that a well prepared project would win wide support and Google will support a project that has support of the mentoring organizations.

Cheers,
Kieran

Thanks for any feedback,
Scott