Hi Drupal devs, First, thank you all for an amazingly useful tool. I am a programmer at the CITRIS center at UC Berkeley. We recently finished moving our own web site (http://www.citris-uc.org) over to Drupal and I am now working on the infrastructure to allow us to start providing Drupal-based sites for more of the campus community. So far I have been able to do everything I need with the available modules and a bit of hacking, but I am now starting to create custom modules. I was wondering if it is possible to create modules that will be compatible with both 4.6 and 4.7? Also, does anyone have advice on how to calculate how many Drupal sites a server can handle? Installing new hardware can take months around here, so I need to stay well ahead of the curve. thanks, Tao Starbow CITRIS
Not really...the node type identifying and form creation functions are very different. I'd suggest writing for 4.6, keeping this in mind: http://drupal.org/node/22218 It's all the functionality you need to change between 4.6 and 4.7. On 11/23/05, Tao Starbow <starbow@citris-uc.org> wrote:
Hi Drupal devs,
First, thank you all for an amazingly useful tool.
I am a programmer at the CITRIS center at UC Berkeley. We recently finished moving our own web site (http://www.citris-uc.org) over to Drupal and I am now working on the infrastructure to allow us to start providing Drupal-based sites for more of the campus community. So far I have been able to do everything I need with the available modules and a bit of hacking, but I am now starting to create custom modules. I was wondering if it is possible to create modules that will be compatible with both 4.6 and 4.7?
Also, does anyone have advice on how to calculate how many Drupal sites a server can handle? Installing new hardware can take months around here, so I need to stay well ahead of the curve.
thanks, Tao Starbow CITRIS
On Nov 23, 2005, at 12:11 PM, Tao Starbow wrote:
Hi Drupal devs,
First, thank you all for an amazingly useful tool.
I am a programmer at the CITRIS center at UC Berkeley. We recently finished moving our own web site (http://www.citris-uc.org) over to Drupal and I am now working on the infrastructure to allow us to start providing Drupal-based sites for more of the campus community. So far I have been able to do everything I need with the available modules and a bit of hacking, but I am now starting to create custom modules. I was wondering if it is possible to create modules that will be compatible with both 4.6 and 4.7?
Also, does anyone have advice on how to calculate how many Drupal sites a server can handle?
Bryght runs over a thousand. It's totally traffic dependent.
Installing new hardware can take months around here, so I need to stay well ahead of the curve.
thanks, Tao Starbow CITRIS
Thanks. I had hoped there was some way to emulate the new 4.7 infrastructure in 4.6. I guess I will wait on the 4.7 issues until we upgrade everything.
Also, does anyone have advice on how to calculate how many Drupal sites a server can handle?
Bryght runs over a thousand. It's totally traffic dependent.
Over a thousand sites on a single server? Wow, that is much more scalable that I would have guessed. I am coming from the JSP world, which is more or a resource hog. Is there a more useful metric, like total simultaneous authorized users or total hits per hour? How does Bryght make the call, "time to pop another server in the rack"?
Tao Starbow wrote:
Bryght runs over a thousand. It's totally traffic dependent.
Over a thousand sites on a single server? Wow, that is much more scalable that I would have guessed. I am coming from the JSP world, which is more or a resource hog. Is there a more useful metric, like total simultaneous authorized users or total hits per hour?
Ever the helpful answer: "It depends". I have a few very high-traffic sites (>50K pages per day) and they'd all be dying on their arse without several patches found in the forums, persistent connections and some ruthless timeout limits. Out of the box, Drupal 4.6 doesn't cut it for scaling (*for those clients*). It's a "3-D" sort of thing with server performance. You balance memory usage and indexing and thread spawning on the MySQL side, possibly using one or more external DB servers. At the front end you balance memory, simultaneous connections, threads spawned and process-murder with Apache/$webserver. Plus, of course, it always works just fine until you throw users at it. In the end all this stuff comes down to a process of screwing with everything until it stops being broken, then leaving it the hell alone. That's not a Drupal thing. That applies to everything I ever worked on. jh
In the end all this stuff comes down to a process of screwing with everything until it stops being broken, then leaving it the hell alone. That's not a Drupal thing. That applies to everything I ever worked on.
spoken like someone who's done it before. performance optimization really is an art, and a messy one at that.
Hi Tao, Glad to hear things are working for you. Rather than ask a very open ended question about how Drupal scales - for which the only reasonable answer is "it depends" - you might try outlining your requirements. How many sites, how many users, how are the sites going to be used, administered etc. Also some information on your current IT infrastructure - types of servers, network etc. would be helpful. Dan
Hi Drupal devs,
First, thank you all for an amazingly useful tool.
I am a programmer at the CITRIS center at UC Berkeley. We recently finished moving our own web site (http://www.citris-uc.org) over to Drupal and I am now working on the infrastructure to allow us to start providing Drupal-based sites for more of the campus community. So far I have been able to do everything I need with the available modules and a bit of hacking, but I am now starting to create custom modules. I was wondering if it is possible to create modules that will be compatible with both 4.6 and 4.7?
Also, does anyone have advice on how to calculate how many Drupal sites a server can handle? Installing new hardware can take months around here, so I need to stay well ahead of the curve.
thanks, Tao Starbow CITRIS
Hi Dan, Good point. Right now our infrastructure consists of a single dell poweredge (3GHz Xeno, 2GB ram, 70 GB scsi raid), running FreeBSD. We can pretty much dedicate this box to serving Drupal (apache and mysql on the same box). We are using the engineering department's network, so I am not worried about running out of bandwidth. We hope to be able to serve as many sites as there is interest. The first stage is to set up sites for about half-a-dozen projects and centers and see how they use them. My feeling is that we could end up filling a very important niche on campus, and demand could grow quite large, once the word gets out. And CITRIS is not just UCB. Our charter is to encourage inter-departmental & multi-campus collaboration at UCB, UC Santa Cruz, UC David & UC Merced. My guess is that the typical site will have a few dozen, or less, registered users producing content that is consumed by a few thousand, or less, viewers. This is my initial target as I pick the supported modules and default configurations. But that is just my guess, and the groups might end up using the system very differently. That is why Drupal's combination of features and costomizabilty seems like such a great fit. -tao Dan Robinson wrote:
Hi Tao,
Glad to hear things are working for you. Rather than ask a very open ended question about how Drupal scales - for which the only reasonable answer is "it depends" - you might try outlining your requirements. How many sites, how many users, how are the sites going to be used, administered etc. Also some information on your current IT infrastructure - types of servers, network etc. would be helpful.
Dan
Hi Drupal devs,
First, thank you all for an amazingly useful tool.
I am a programmer at the CITRIS center at UC Berkeley. We recently finished moving our own web site (http://www.citris-uc.org) over to Drupal and I am now working on the infrastructure to allow us to start providing Drupal-based sites for more of the campus community. So far I have been able to do everything I need with the available modules and a bit of hacking, but I am now starting to create custom modules. I was wondering if it is possible to create modules that will be compatible with both 4.6 and 4.7?
Also, does anyone have advice on how to calculate how many Drupal sites a server can handle? Installing new hardware can take months around here, so I need to stay well ahead of the curve.
thanks, Tao Starbow CITRIS
Tao Starbow wrote:
Good point. Right now our infrastructure consists of a single dell poweredge (3GHz Xeno, 2GB ram, 70 GB scsi raid), running FreeBSD. We can pretty much dedicate this box to serving Drupal (apache and mysql on the same box). We are using the engineering department's network, so I am not worried about running out of bandwidth.
The main limiting number with the old drupal.org server was the number of concurrent, active registered users. Other factors were the fact that due to a lot of content being added the caching was less effective than we thought (has been worked on) and that MySQL used quite a lot of memory (we only had 1G). Cheers, Gerhard
The main limiting number with the old drupal.org server was the number of concurrent, active registered users. Other factors were the fact that due to a lot of content being added the caching was less effective than we thought (has been worked on) and that MySQL used quite a lot of memory (we only had 1G).
also it was getting slammed by misbehaving web crawlers. if we had an automatic ban module, it might have extended the life of that server.
On factor I found, and one that we used to estimate our business model is that it hardly matters if you run one site with 1000 visitors or 100 with 100. Hope that helps you finetuning your model. Ber Op woensdag 23 november 2005 22:43, schreef Tao Starbow:
Hi Dan,
Good point. Right now our infrastructure consists of a single dell poweredge (3GHz Xeno, 2GB ram, 70 GB scsi raid), running FreeBSD. We can pretty much dedicate this box to serving Drupal (apache and mysql on the same box). We are using the engineering department's network, so I am not worried about running out of bandwidth.
We hope to be able to serve as many sites as there is interest. The first stage is to set up sites for about half-a-dozen projects and centers and see how they use them. My feeling is that we could end up filling a very important niche on campus, and demand could grow quite large, once the word gets out. And CITRIS is not just UCB. Our charter is to encourage inter-departmental & multi-campus collaboration at UCB, UC Santa Cruz, UC David & UC Merced.
My guess is that the typical site will have a few dozen, or less, registered users producing content that is consumed by a few thousand, or less, viewers. This is my initial target as I pick the supported modules and default configurations. But that is just my guess, and the groups might end up using the system very differently. That is why Drupal's combination of features and costomizabilty seems like such a great fit.
-tao
Dan Robinson wrote:
Hi Tao,
Glad to hear things are working for you. Rather than ask a very open ended question about how Drupal scales - for which the only reasonable answer is "it depends" - you might try outlining your requirements. How many sites, how many users, how are the sites going to be used, administered etc. Also some information on your current IT infrastructure - types of servers, network etc. would be helpful.
Dan
Hi Drupal devs,
First, thank you all for an amazingly useful tool.
I am a programmer at the CITRIS center at UC Berkeley. We recently finished moving our own web site (http://www.citris-uc.org) over to Drupal and I am now working on the infrastructure to allow us to start providing Drupal-based sites for more of the campus community. So far I have been able to do everything I need with the available modules and a bit of hacking, but I am now starting to create custom modules. I was wondering if it is possible to create modules that will be compatible with both 4.6 and 4.7?
Also, does anyone have advice on how to calculate how many Drupal sites a server can handle? Installing new hardware can take months around here, so I need to stay well ahead of the curve.
thanks, Tao Starbow CITRIS
So you are saying that Drupal acts like one big app, independent of the number of sites (I am assuming you mean 10,000 visitors vs 100 sites with 100 visitors)? That's really good to know. Is there a rule of thumb for 1 registered user = X anonymous users? -tao Bèr Kessels wrote:
On factor I found, and one that we used to estimate our business model is that it hardly matters if you run one site with 1000 visitors or 100 with 100.
Hope that helps you finetuning your model.
Ber Op woensdag 23 november 2005 22:43, schreef Tao Starbow:
Hi Dan,
Good point. Right now our infrastructure consists of a single dell poweredge (3GHz Xeno, 2GB ram, 70 GB scsi raid), running FreeBSD. We can pretty much dedicate this box to serving Drupal (apache and mysql on the same box). We are using the engineering department's network, so I am not worried about running out of bandwidth.
We hope to be able to serve as many sites as there is interest. The first stage is to set up sites for about half-a-dozen projects and centers and see how they use them. My feeling is that we could end up filling a very important niche on campus, and demand could grow quite large, once the word gets out. And CITRIS is not just UCB. Our charter is to encourage inter-departmental & multi-campus collaboration at UCB, UC Santa Cruz, UC David & UC Merced.
My guess is that the typical site will have a few dozen, or less, registered users producing content that is consumed by a few thousand, or less, viewers. This is my initial target as I pick the supported modules and default configurations. But that is just my guess, and the groups might end up using the system very differently. That is why Drupal's combination of features and costomizabilty seems like such a great fit.
-tao
Dan Robinson wrote:
Hi Tao,
Glad to hear things are working for you. Rather than ask a very open ended question about how Drupal scales - for which the only reasonable answer is "it depends" - you might try outlining your requirements. How many sites, how many users, how are the sites going to be used, administered etc. Also some information on your current IT infrastructure - types of servers, network etc. would be helpful.
Dan
Hi Drupal devs,
First, thank you all for an amazingly useful tool.
I am a programmer at the CITRIS center at UC Berkeley. We recently finished moving our own web site (http://www.citris-uc.org) over to Drupal and I am now working on the infrastructure to allow us to start providing Drupal-based sites for more of the campus community. So far I have been able to do everything I need with the available modules and a bit of hacking, but I am now starting to create custom modules. I was wondering if it is possible to create modules that will be compatible with both 4.6 and 4.7?
Also, does anyone have advice on how to calculate how many Drupal sites a server can handle? Installing new hardware can take months around here, so I need to stay well ahead of the curve.
thanks, Tao Starbow CITRIS
This is what i calculated on about 40 sites. When one site was very active, I saw the same issues / problems then when several sites had slightly higher load. Op woensdag 23 november 2005 23:38, schreef Tao Starbow:
So you are saying that Drupal acts like one big app, independent of the number of sites (I am assuming you mean 10,000 visitors vs 100 sites with 100 visitors)?
Sorry, it was late :) 10.000 == 100 x 100 I am not sure if this is linear in every amount, but the way drupal multisite works, I would think it does.
That's really good to know. Is there a rule of thumb for 1 registered user = X anonymous users?
That really depends on the used modules. In fact it all does. Especially contributed modules often perform very bad. But I do not have any such figures.
-tao
Bèr Kessels wrote:
On factor I found, and one that we used to estimate our business model is that it hardly matters if you run one site with 1000 visitors or 100 with 100.
Hope that helps you finetuning your model.
Ber
Op woensdag 23 november 2005 22:43, schreef Tao Starbow:
Hi Dan,
Good point. Right now our infrastructure consists of a single dell poweredge (3GHz Xeno, 2GB ram, 70 GB scsi raid), running FreeBSD. We can pretty much dedicate this box to serving Drupal (apache and mysql on the same box). We are using the engineering department's network, so I am not worried about running out of bandwidth.
We hope to be able to serve as many sites as there is interest. The first stage is to set up sites for about half-a-dozen projects and centers and see how they use them. My feeling is that we could end up filling a very important niche on campus, and demand could grow quite large, once the word gets out. And CITRIS is not just UCB. Our charter is to encourage inter-departmental & multi-campus collaboration at UCB, UC Santa Cruz, UC David & UC Merced.
My guess is that the typical site will have a few dozen, or less, registered users producing content that is consumed by a few thousand, or less, viewers. This is my initial target as I pick the supported modules and default configurations. But that is just my guess, and the groups might end up using the system very differently. That is why Drupal's combination of features and costomizabilty seems like such a great fit.
-tao
Dan Robinson wrote:
Hi Tao,
Glad to hear things are working for you. Rather than ask a very open ended question about how Drupal scales - for which the only reasonable answer is "it depends" - you might try outlining your requirements. How many sites, how many users, how are the sites going to be used, administered etc. Also some information on your current IT infrastructure - types of servers, network etc. would be helpful.
Dan
Hi Drupal devs,
First, thank you all for an amazingly useful tool.
I am a programmer at the CITRIS center at UC Berkeley. We recently finished moving our own web site (http://www.citris-uc.org) over to Drupal and I am now working on the infrastructure to allow us to start providing Drupal-based sites for more of the campus community. So far I have been able to do everything I need with the available modules and a bit of hacking, but I am now starting to create custom modules. I was wondering if it is possible to create modules that will be compatible with both 4.6 and 4.7?
Also, does anyone have advice on how to calculate how many Drupal sites a server can handle? Installing new hardware can take months around here, so I need to stay well ahead of the curve.
thanks, Tao Starbow CITRIS
When one site was very active, I saw the same issues / problems then when several sites had slightly higher load.
I don't agree with Ber's conclusion. Ber, what test methodology did you use to draw that conclusion? Because each site has its own cache, multiple sites should scale better than one big site. No? -- Dries Buytaert :: http://www.buytaert.net/
On Thu, 2005-11-24 at 10:19 +0100, Dries Buytaert wrote:
When one site was very active, I saw the same issues / problems then when several sites had slightly higher load.
I don't agree with Ber's conclusion. Ber, what test methodology did you use to draw that conclusion?
Because each site has its own cache, multiple sites should scale better than one big site. No?
-- Dries Buytaert :: http://www.buytaert.net/
I think that may somewhat depend on mysql's query cache as well in a mysql environment... Ber's notation of 1 site w/1000 visitor vs 100 sites with 100 visitors(10K users total) seems to be inline with the idea that many small sites scale better than 1 large site. .darrel.
On 23 Nov 2005, at 22:43, Tao Starbow wrote:
Good point. Right now our infrastructure consists of a single dell poweredge (3GHz Xeno, 2GB ram, 70 GB scsi raid), running FreeBSD. We can pretty much dedicate this box to serving Drupal (apache and mysql on the same box). We are using the engineering department's network, so I am not worried about running out of bandwidth.
Looks like that machine should be able to host quite a few Drupal sites; I'm thinking 250 sites should be possible, but it obviously depends on a number of parameters as mentioned elsewhere in this thread. Fact is that many small sites scale better than one really big site; because each site is actually fairly small and reasonably static, Drupal's page caching mechanism should be very effective. (I'm somewhat puzzled by Ber's numbers and would like to know more about why he thinks otherwise.) Creating "forward compatible modules" does not look like a practical option. It's better to stick with Drupal 4.6 and to upgrade to Drupal 4.7 when the time is right. Upgrading to Drupal 4.7 (including custom modules) should be relatively straightforward but might take a bit of time depending on the amount of custom changes and custom modules. One thing you can do is start off with PHPTemplate-based themes rather than XTemplate-based themes; PHPTemplate will be the default theme engine in Drupal 4.7. If you are somewhat adventurous you could start out with Drupal CVS HEAD (the forthcoming Drupal 4.7); it shouldn't be too bad if you stick with core modules. A good formula for success is to participate in the development of Drupal; you'll quickly learn the ins and outs and will be able to weight decisions much more efficiently. Tao, some of us (including myself) are very interested in improving Drupal's multi-site features. It is inevitable that you'll run into some gotchas so I'm hoping you'll participate in the development of said feature. (Eg. Adrian has been working on a patch to lock or restrict certain settings.) So welcome on board, and looking forward to your contributions. ;) -- Dries Buytaert :: http://www.buytaert.net/
Op donderdag 24 november 2005 09:42, schreef Dries Buytaert:
(I'm somewhat puzzled by Ber's numbers and would like to know more about why he thinks otherwise.)
These were measured. But now you mention that caching I am somehwat puzzled too. But I think I know the answer. All sites are non-community sites. Meaning that they are not updated as frequent as a community site would. Does anybody know a good method to measure the amount of cached pages / uncached pages. Furhtermore, this was measured with 4.6 only. Ber
These were measured. But now you mention that caching I am somehwat puzzled too. But I think I know the answer. All sites are non-community sites. Meaning that they are not updated as frequent as a community site would.
If you are measuring static sites, your conclusion makes sense. If, however, your site has a dynamic aspect (forums, comments, RSS aggregator, etc) this is no longer valid, I think. -- Dries Buytaert :: http://www.buytaert.net/
Op donderdag 24 november 2005 10:23, schreef Dries Buytaert:
If, however, your site has a dynamic aspect (forums, comments, RSS aggregator, etc) this is no longer valid, I think.
* These sites have hardly any registered users. only those adminstrating and moderating and those adding a few stories / images a day. * I do use node aggragator on several sites and aggregator on hardly any. it is ran every two hours. * Comments are neglecatble. Average of all sites brings me a comment rate of 0.2 (two comments on ten posts) * No forums, no buddylists and no dynamic filters. I hope this clarifies it even more: I think the conclusion again is: "it depends", in this case on the sort of site. Ber
Thanks for both the information and the encouragement! I have a cvs head version for my own testing, but I think I am going to stick with 4.6 for what we are rolling out to others. I am happy to contribute as I learn. I am moving my development notes off of our internal wiki and on to a public blog at http://www.citris-uc.org/blog/1, if anyone wants to keep up with my trials and tribulations. cheers, -tao Dries Buytaert wrote:
On 23 Nov 2005, at 22:43, Tao Starbow wrote:
Good point. Right now our infrastructure consists of a single dell poweredge (3GHz Xeno, 2GB ram, 70 GB scsi raid), running FreeBSD. We can pretty much dedicate this box to serving Drupal (apache and mysql on the same box). We are using the engineering department's network, so I am not worried about running out of bandwidth.
Looks like that machine should be able to host quite a few Drupal sites; I'm thinking 250 sites should be possible, but it obviously depends on a number of parameters as mentioned elsewhere in this thread. Fact is that many small sites scale better than one really big site; because each site is actually fairly small and reasonably static, Drupal's page caching mechanism should be very effective. (I'm somewhat puzzled by Ber's numbers and would like to know more about why he thinks otherwise.)
Creating "forward compatible modules" does not look like a practical option. It's better to stick with Drupal 4.6 and to upgrade to Drupal 4.7 when the time is right. Upgrading to Drupal 4.7 (including custom modules) should be relatively straightforward but might take a bit of time depending on the amount of custom changes and custom modules. One thing you can do is start off with PHPTemplate-based themes rather than XTemplate-based themes; PHPTemplate will be the default theme engine in Drupal 4.7.
If you are somewhat adventurous you could start out with Drupal CVS HEAD (the forthcoming Drupal 4.7); it shouldn't be too bad if you stick with core modules. A good formula for success is to participate in the development of Drupal; you'll quickly learn the ins and outs and will be able to weight decisions much more efficiently.
Tao, some of us (including myself) are very interested in improving Drupal's multi-site features. It is inevitable that you'll run into some gotchas so I'm hoping you'll participate in the development of said feature. (Eg. Adrian has been working on a patch to lock or restrict certain settings.) So welcome on board, and looking forward to your contributions. ;)
-- Dries Buytaert :: http://www.buytaert.net/
wondering if it is possible to create modules that will be compatible with both 4.6 and 4.7?
Yes. You write both modules, put each them into a .inc file and all the module is if (function_exists('_node_name')) { include 'mymodule_4.7.inc'; } else { include 'mymodule_4.7.inc'; } But I do not think this approach worths it...
I was wondering if it is possible to create modules that will be compatible with both 4.6 and 4.7?
You could write a backport of selected 4.7 features as a 4.6 module, and then write your 4.6 module using the backported 4.7 methods. This would allow you to write with 4.7-like code, and thus ease your eventual upgrading. I've taken this approach with 4.7 javascript, see: http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/nedjo/modules/dru...
On Wednesday 23 November 2005 15:11, Tao Starbow wrote:
I was wondering if it is possible to create modules that will be compatible with both 4.6 and 4.7?
This particular upgrade happens to have the new forms API, which is a clean break with previous form-handling architecture, so although it's *possible* to to this, with a lot of checking for functions existing or not, it's generally not *practical* to do so. Easier to just create a clean copy of the module then optimize for 4.7+. In the past, though, multi-version modules haven't been too hard, at least within a close range of versions. I have several contributed modules that used the same code base for 4.5 and 4.6, with just a little conditional logic. Of course, no one can say what will be the situation between 4.7 and 4.8. :-) Drupal's philosophy seems to be one of treating database schemas and logic flow patterns as "semi-solid" and the code itself as being largely "fluid". I think that's one of the things I like about Drupal, because although it means some recode work with each version, it also tends to keep out the cruft. If you have application-specific functions that are independent of Drupal itself, you can easily separate them into an "include" file that is shared across multiple versions of your module. Hope this helps. Scott -- ------------------------------------------------------------------------------- Scott Courtney Drupal user name: "syscrusher" http://drupal.org/user/9184 scott at 4th dot com Drupal projects: http://drupal.org/project/user/9184 Sandbox: http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/syscrusher
participants (13)
-
Bèr Kessels -
Dan Robinson -
Darrel O'Pry -
Dries Buytaert -
Earl Dunovant -
Gerhard Killesreiter -
John Handelaar -
Karoly Negyesi -
Kieran Lal -
Moshe Weitzman -
Nedjo Rogers -
Syscrusher -
Tao Starbow