http://drupal.org/node/22218

Thanks. I had hoped there was some way to emulate the new 4.7 infrastructure in 4.6. I guess I will wait on the 4.7 issues until we upgrade everything.

...

Also, does anyone have advice on how to calculate how many Drupal sites a server can handle?

Bryght runs over a thousand. It's totally traffic dependent.

Over a thousand sites on a single server? Wow, that is much more scalable that I would have guessed. I am coming from the JSP world, which is more or a resource hog. Is there a more useful metric, like total simultaneous authorized users or total hits per hour? How does Bryght make the call, "time to pop another server in the rack"?

In the end all this stuff comes down to a process of screwing with everything until it stops being broken, then leaving it the hell alone. That's not a Drupal thing. That applies to everything I ever worked on.

spoken like someone who's done it before. performance optimization really is an art, and a messy one at that.

Hi Dan, Good point. Right now our infrastructure consists of a single dell poweredge (3GHz Xeno, 2GB ram, 70 GB scsi raid), running FreeBSD. We can pretty much dedicate this box to serving Drupal (apache and mysql on the same box). We are using the engineering department's network, so I am not worried about running out of bandwidth. We hope to be able to serve as many sites as there is interest. The first stage is to set up sites for about half-a-dozen projects and centers and see how they use them. My feeling is that we could end up filling a very important niche on campus, and demand could grow quite large, once the word gets out. And CITRIS is not just UCB. Our charter is to encourage inter-departmental & multi-campus collaboration at UCB, UC Santa Cruz, UC David & UC Merced. My guess is that the typical site will have a few dozen, or less, registered users producing content that is consumed by a few thousand, or less, viewers. This is my initial target as I pick the supported modules and default configurations. But that is just my guess, and the groups might end up using the system very differently. That is why Drupal's combination of features and costomizabilty seems like such a great fit. -tao Dan Robinson wrote:

Hi Tao,

Glad to hear things are working for you. Rather than ask a very open ended question about how Drupal scales - for which the only reasonable answer is "it depends" - you might try outlining your requirements. How many sites, how many users, how are the sites going to be used, administered etc. Also some information on your current IT infrastructure - types of servers, network etc. would be helpful.

Dan

...

Hi Drupal devs,

First, thank you all for an amazingly useful tool.

I am a programmer at the CITRIS center at UC Berkeley. We recently finished moving our own web site (http://www.citris-uc.org) over to Drupal and I am now working on the infrastructure to allow us to start providing Drupal-based sites for more of the campus community. So far I have been able to do everything I need with the available modules and a bit of hacking, but I am now starting to create custom modules. I was wondering if it is possible to create modules that will be compatible with both 4.6 and 4.7?

Also, does anyone have advice on how to calculate how many Drupal sites a server can handle? Installing new hardware can take months around here, so I need to stay well ahead of the curve.

thanks, Tao Starbow CITRIS

The main limiting number with the old drupal.org server was the number of concurrent, active registered users. Other factors were the fact that due to a lot of content being added the caching was less effective than we thought (has been worked on) and that MySQL used quite a lot of memory (we only had 1G).

also it was getting slammed by misbehaving web crawlers. if we had an automatic ban module, it might have extended the life of that server.

So you are saying that Drupal acts like one big app, independent of the number of sites (I am assuming you mean 10,000 visitors vs 100 sites with 100 visitors)? That's really good to know. Is there a rule of thumb for 1 registered user = X anonymous users? -tao Bèr Kessels wrote:

On factor I found, and one that we used to estimate our business model is that it hardly matters if you run one site with 1000 visitors or 100 with 100.

Hope that helps you finetuning your model.

Ber Op woensdag 23 november 2005 22:43, schreef Tao Starbow:

...

Hi Dan,

Good point. Right now our infrastructure consists of a single dell poweredge (3GHz Xeno, 2GB ram, 70 GB scsi raid), running FreeBSD. We can pretty much dedicate this box to serving Drupal (apache and mysql on the same box). We are using the engineering department's network, so I am not worried about running out of bandwidth.

We hope to be able to serve as many sites as there is interest. The first stage is to set up sites for about half-a-dozen projects and centers and see how they use them. My feeling is that we could end up filling a very important niche on campus, and demand could grow quite large, once the word gets out. And CITRIS is not just UCB. Our charter is to encourage inter-departmental & multi-campus collaboration at UCB, UC Santa Cruz, UC David & UC Merced.

My guess is that the typical site will have a few dozen, or less, registered users producing content that is consumed by a few thousand, or less, viewers. This is my initial target as I pick the supported modules and default configurations. But that is just my guess, and the groups might end up using the system very differently. That is why Drupal's combination of features and costomizabilty seems like such a great fit.

-tao

Dan Robinson wrote:

...

Hi Tao,

Glad to hear things are working for you. Rather than ask a very open ended question about how Drupal scales - for which the only reasonable answer is "it depends" - you might try outlining your requirements. How many sites, how many users, how are the sites going to be used, administered etc. Also some information on your current IT infrastructure - types of servers, network etc. would be helpful.

Dan

...

Hi Drupal devs,

First, thank you all for an amazingly useful tool.

I am a programmer at the CITRIS center at UC Berkeley. We recently finished moving our own web site (http://www.citris-uc.org) over to Drupal and I am now working on the infrastructure to allow us to start providing Drupal-based sites for more of the campus community. So far I have been able to do everything I need with the available modules and a bit of hacking, but I am now starting to create custom modules. I was wondering if it is possible to create modules that will be compatible with both 4.6 and 4.7?

Also, does anyone have advice on how to calculate how many Drupal sites a server can handle? Installing new hardware can take months around here, so I need to stay well ahead of the curve.

thanks, Tao Starbow CITRIS

When one site was very active, I saw the same issues / problems then when several sites had slightly higher load.

I don't agree with Ber's conclusion. Ber, what test methodology did you use to draw that conclusion? Because each site has its own cache, multiple sites should scale better than one big site. No? -- Dries Buytaert :: http://www.buytaert.net/

On 23 Nov 2005, at 22:43, Tao Starbow wrote:

Good point. Right now our infrastructure consists of a single dell poweredge (3GHz Xeno, 2GB ram, 70 GB scsi raid), running FreeBSD. We can pretty much dedicate this box to serving Drupal (apache and mysql on the same box). We are using the engineering department's network, so I am not worried about running out of bandwidth.

Looks like that machine should be able to host quite a few Drupal sites; I'm thinking 250 sites should be possible, but it obviously depends on a number of parameters as mentioned elsewhere in this thread. Fact is that many small sites scale better than one really big site; because each site is actually fairly small and reasonably static, Drupal's page caching mechanism should be very effective. (I'm somewhat puzzled by Ber's numbers and would like to know more about why he thinks otherwise.) Creating "forward compatible modules" does not look like a practical option. It's better to stick with Drupal 4.6 and to upgrade to Drupal 4.7 when the time is right. Upgrading to Drupal 4.7 (including custom modules) should be relatively straightforward but might take a bit of time depending on the amount of custom changes and custom modules. One thing you can do is start off with PHPTemplate-based themes rather than XTemplate-based themes; PHPTemplate will be the default theme engine in Drupal 4.7. If you are somewhat adventurous you could start out with Drupal CVS HEAD (the forthcoming Drupal 4.7); it shouldn't be too bad if you stick with core modules. A good formula for success is to participate in the development of Drupal; you'll quickly learn the ins and outs and will be able to weight decisions much more efficiently. Tao, some of us (including myself) are very interested in improving Drupal's multi-site features. It is inevitable that you'll run into some gotchas so I'm hoping you'll participate in the development of said feature. (Eg. Adrian has been working on a patch to lock or restrict certain settings.) So welcome on board, and looking forward to your contributions. ;) -- Dries Buytaert :: http://www.buytaert.net/

These were measured. But now you mention that caching I am somehwat puzzled too. But I think I know the answer. All sites are non-community sites. Meaning that they are not updated as frequent as a community site would.

If you are measuring static sites, your conclusion makes sense. If, however, your site has a dynamic aspect (forums, comments, RSS aggregator, etc) this is no longer valid, I think. -- Dries Buytaert :: http://www.buytaert.net/

Thanks for both the information and the encouragement! I have a cvs head version for my own testing, but I think I am going to stick with 4.6 for what we are rolling out to others. I am happy to contribute as I learn. I am moving my development notes off of our internal wiki and on to a public blog at http://www.citris-uc.org/blog/1, if anyone wants to keep up with my trials and tribulations. cheers, -tao Dries Buytaert wrote:

On 23 Nov 2005, at 22:43, Tao Starbow wrote:

...

Good point. Right now our infrastructure consists of a single dell poweredge (3GHz Xeno, 2GB ram, 70 GB scsi raid), running FreeBSD. We can pretty much dedicate this box to serving Drupal (apache and mysql on the same box). We are using the engineering department's network, so I am not worried about running out of bandwidth.

Looks like that machine should be able to host quite a few Drupal sites; I'm thinking 250 sites should be possible, but it obviously depends on a number of parameters as mentioned elsewhere in this thread. Fact is that many small sites scale better than one really big site; because each site is actually fairly small and reasonably static, Drupal's page caching mechanism should be very effective. (I'm somewhat puzzled by Ber's numbers and would like to know more about why he thinks otherwise.)

Creating "forward compatible modules" does not look like a practical option. It's better to stick with Drupal 4.6 and to upgrade to Drupal 4.7 when the time is right. Upgrading to Drupal 4.7 (including custom modules) should be relatively straightforward but might take a bit of time depending on the amount of custom changes and custom modules. One thing you can do is start off with PHPTemplate-based themes rather than XTemplate-based themes; PHPTemplate will be the default theme engine in Drupal 4.7.

If you are somewhat adventurous you could start out with Drupal CVS HEAD (the forthcoming Drupal 4.7); it shouldn't be too bad if you stick with core modules. A good formula for success is to participate in the development of Drupal; you'll quickly learn the ins and outs and will be able to weight decisions much more efficiently.

Tao, some of us (including myself) are very interested in improving Drupal's multi-site features. It is inevitable that you'll run into some gotchas so I'm hoping you'll participate in the development of said feature. (Eg. Adrian has been working on a patch to lock or restrict certain settings.) So welcome on board, and looking forward to your contributions. ;)

-- Dries Buytaert :: http://www.buytaert.net/