The Drupal Diet - Making bootstrap faster
This is more of an RFC than a DEP, so please forgive the looser format and trademark verbosity. :-) My current push for Drupal 6 is to make it faster for non-opcode non-cached users. Drupal 5 was, according to Dries' benchmarks, a slight step backwards from Drupal 4.7 in that regard, so let's reverse that trend with a vengeance. Far and away the slowest part of the Drupal life-cycle is bootstrapping. According to Rasmus' keynote at Drupalcon, we spend just over 50% of the entire process just pulling code off disk and parsing it. A typical page load, however, uses only a small fraction of that. Thus, the biggest target for optimization is "load less code", but without violating the corollary, "load fewer files". As an example, I recently tried breaking up some core modules and loading page callbacks and form only when needed[1]. Even with that primitive breakup, we were able to get an 8-18% improvement in page load time and a 23% decrease in memory usage. I hate sayings like "the numbers speak for themselves", but in this case they do. On-demand loading of lesser-used module code has the potential to be a huge win, and the extra code required to make it possible is minimal. (The code linked in that issue only adds ~10 lines of code; the rest of the patch is just moving code around.) That of course begs the question, how to split up the code in a module? In general, I see 5 logical divisions of code within a module: 1) Rare hooks. hook_install() and hook_update() are the classic cases here, although I think hook_menu() in Drupal 6 may be moving in that direction. These are hooks called only at very specific, rare times. The other 99% of the time they're dead weight. 2) Common hooks. This is basically every hook that isn't one of the few rare hooks. These may be called at any time, more or less often. (For the time being I am going to lump hook_load(), hook_update(), etc. in here, even though they're technically not hooks as we've discussed previously.) 3) Page handlers. These are functions whose primary purpose in life is to be called from menu_execute_active_handler(). They serve no other serious function. 4) Form builders. These are the form definition, validation, and submission functions, as well as their sub-call helpers. 5) API functions. These are functions specifically exposed to other modules to do stuff, for some definition of stuff. 6 (nobody expects the Spanish inquisition!)) Utility functions. These functions are mostly intended for internal use but can sometimes be useful to other modules. The line between a utility function and an API function is very blurry since PHP functions have no concept of namespace or visibility. There aren't that many Rare Hooks, and most of them are already in .install files so I will ignore those for now. Common Hooks by nature need to be readily-available at any time. It is possible to dynamically load those, too, but that's a more complex issue, and one that merlinofchaos[2] and chx[3] have already started to address. I am therefore not going to deal with those, either. API functions also have to be readily-available, and utility functions probably should, too. So now that we've said we need to load most types of code, what does that leave us? That leaves us the two types of code that are used the least but take up the most lines of code. merlin and webchick recently went through a few core modules and cataloged what functions were of what type[4], and the results are clear: We spend most of our code on page and form handling, and yet only one page is ever handled per page load and, generally, only 1-2 forms! In terms of actual lines of code, all four modules in question (system, user, comment, block) are majority pages and forms, in some cases by over 2/3. That means page handlers and form handlers are the safest to factor out into separate on-demand files but also the biggest win from doing so. It's nice how that works out. So now we need a mechanism for on-demand loading of page handlers and form handlers, subject to the following conditions: 1) It should be an optional optimization. We don't want to force all modules to break up, because many, I'd say the majority, are small and simple enough that it would be a case of over-optimization to require, say, every form to be in a separate file or every module to have a .pages file. We also don't want to make module authoring an overly-difficult process with a dozen magic files. The degenerate case should be exactly how things work now. 2) It should be flexible. Different modules need to be optimized differently. Putting all page handlers into a single .pages file for a module could still mean loading 10x as much code as we really need. Module authors need to be able to factor their own modules in the way that makes the most sense for that module, which could mean one on-demand file or several. 3) It is impossible to determine the module that provides a function from the function name alone. Sure all functions (should) use $modulename_<something> as their format, but many modules have an underscore in their name. Given a function named "foo_bar_baz", is that the "bar_baz" function of the "foo" module, or the "baz" function of the "foo_bar" module? We can't tell. Therefore, unless we are going to simply exclude modules with such names from this system (and I think that's a really bad idea) we will have to explicitly specify the module or path for a given auxiliary file. 4) Modules may call page handlers and form handlers from other modules. Core does this in places (node.module calls a page handler from system.module, for instance) as do various contribs, so we can't assume that the calling module is the providing module. 4) Page handlers are called from the menu system; therefore, the logical place to decide if additional code is needed is the menu system. Since we can't presume or deduce a module from the handler, that means it has to be specified explicitly in hook_menu(). 5) Form handlers are called from drupal_get_form(), or from drupal_execute(). Many are parameters to drupal_get_form() being used as a page handler, but not all. drupal_execute() may be called from anywhere at any time, too, so forms need to be either already loaded or loadable on-demand at any time. I therefore propose (finally I get to this part!) to split off page handlers and form handlers in similar ways. == Page Handlers == Only one page handler is called per page load, so we only need to worry about a page handler becoming available in menu_execute_active_handler(). Modules provide information to the menu system via hook_menu(), so each menu item can optionally specify information on what file to load in order to make the handler available. That could be one of two ways: Pass a full path (eg, drupal_get_path('module', 'foo') . '/foo.pages.inc' ) or specify a file name and module name separately. For simple flexibility I favor the former. It's simple and effective and works for cross-module calls. It's also what's already implemented in the patch I mentioned earlier[1]. == Form Handlers == Forms are nearly always accessed via drupal_get_form() or drupal_execute(). We can therefore do the same sort of centralized improvement for the form system in those functions as we can for page handlers using menu_execute_active_handler(). That is, add a key to hook_forms() to specify a file in which the form lives. Here we can safely presume that the module implementing hook_forms() is also the home of the form functions in question, so we need specify only a file and not a module or path. If a module author wishes to split off one or more forms to another file, hook_forms() becomes a requirement just as it does for specifying an alternate callback function. drupal_get_form() and drupal_execute() then simply check for the existence of that key and include_one() the file if necessary. The total code involved should, like the page handler, be quite limited. Note: I will likely want to wait on implementing the forms part until the FAPI 3 patch lands, because I really don't want to tangle with both eaton and chx on that. :-) The nice thing about this approach, too, is that it doesn't have to be implemented all at once. Because the degenerate case still works, the initial implementation can work on only one or two core modules as a demonstration. The rest of core can be optimized module-at-a-time. That makes the patch easier to review as well as easier to maintain with the rest of core still being actively developed. Given the benchmarks that merlin found with the initial attempt, I'd say whatever the total performance gain is it should be substantial. To the potential problem of module authors "over-factoring" and hiding useful utility functions in a page handler when they shouldn't, I believe that really is solved by best practice guidelines. As a worst-case, a module author can manually include_once() a file out of another module's directory at no worse a cost than the extra parse time. It's still a net-win overall since even if one module gets sloppy-loaded the rest of the system is still well-factored, so there's still a net-reduction in the amount of code involved. Sooo... Now that the three of you who made it all the way through this email have gotten here, thoughts on this approach? Any caveats I'm missing? Any use cases I don't know about? Does this have a snowball's chance in hell of being accepted? <dons flame-retardant suit> [1] http://drupal.org/node/140218 [2] http://drupal.org/node/116165 [3] http://drupal.org/node/140218#comment-236614 [4] http://drupal.org/node/116165#comment-229856 -- Larry Garfield AIM: LOLG42 larry@garfieldtech.com ICQ: 6817012 "If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it." -- Thomas Jefferson
Larry, two specific examples that I can think of are the node module's search code and taxonomy module's admin interface code. Patches to split those into separate files would be great. -Robert
Looking through filter.module, adminstration and tips counts for about 500 of 1500 lines total, which could be loaded conditionally, I assume. Is what we're talking about? There has been some discussion about doing this with ecommerce, which is extremely heavy on the admin form side of things, so I'm interested in whether hook_menu is the right place to put the includes()? Robert Douglass wrote:
Larry,
two specific examples that I can think of are the node module's search code and taxonomy module's admin interface code. Patches to split those into separate files would be great.
-Robert
Larry Garfield wrote:
Sooo... Now that the three of you who made it all the way through this email have gotten here, thoughts on this approach? Any caveats I'm missing? Any use cases I don't know about? Does this have a snowball's chance in hell of being accepted?
<dons flame-retardant suit>
Well, we are just getting a locale module cleanup patch in, which does lead to such an optimization. Before the patch, we had proxy functions in locale.module to include locale.inc and call the appropriate functions (a hundred of useless lines of code). After the patch all locale.module menu items are going through this function: /** * Wrapper function to be able to set callbacks in locale.inc */ function locale_inc_callback() { $args = func_get_args(); $function = array_shift($args); include_once './includes/locale.inc'; return call_user_func_array($function, $args); } Dries noted that we might make this more generic, but that is for a separate menu improvement. Seems like this is coming along now, so we will be able to simplify how locale module works after your patch gets submitted. :) PS. Locale module is such a beast that it only has admin pages and forms, so we will have a 425 line lightweight locale module, and a 2066 line heavyweight locale.inc after the patch gets accepted. Also 120 lines of that 425 are the _help and _menu hooks, which would only be required on a small fraction of the pages (we only provide admin help for example). Gabor
+1 for the idea
Sooo... Now that the three of you who made it all the way through this email have gotten here, thoughts on this approach? Any caveats I'm missing? Any use cases I don't know about? Does this have a snowball's chance in hell of being accepted?
Maybe I missed something.... Why not simply start by adding a .page for pages and .form for forms? If the function doesn't exist when it's needed, look for it in the module directory. Or is this what you've already done? Similar factoring could occur for hooks as well. Is this the over-factoring you want to avoid? I don't like having to pass an additional argument telling the system where things live. I think it should have pretty good defaults that work most of the time, and the path argument only overrides in special cases. Doug Green 904-583-3342 www.civicactions.com www.douggreenconsulting.com
On Thu, 3 May 2007 07:43:19 -0400, "Doug Green" <douggreen@douggreenconsulting.com> wrote:
+1 for the idea
Sooo... Now that the three of you who made it all the way through this email have gotten here, thoughts on this approach? Any caveats I'm missing? Any use cases I don't know about? Does this have a snowball's chance in hell of being accepted?
Maybe I missed something....
Why not simply start by adding a .page for pages and .form for forms? If the function doesn't exist when it's needed, look for it in the module directory. Or is this what you've already done?
There's two reasons why we can't do that. 1) user_menu() defines menu callbacks that specify a function in system.module. I've written contribs that specify node_add(), views_get_view(), and panels_get_panel() (or whatever that function is called...) as a page callback. We have to allow cross-module callbacks. We can determine the module that declares the menu item at menu_rebuild() time, but we can't determine the module that provides a function at page load time unless it's been pre-determined because our module naming scheme is ambiguous. The same holds true for forms; the form id and callback function do not adequately identify the providing module. (If someone has an idea for how we can efficiently and definitively determine the providing module for a page or form handler by its name alone, please tell me!) 2) By mandating .pages and .forms, we disallow module authors from further factoring their code out. system.module, for instance, breaks down fairly neatly into 3 different "sets" of page handler functions. We want to allow that module to get split 3 ways instead of just one. Views is another textbook example. One of the added benefits of this task is that the Views UI and Views Theme Wizard modules can simply get folded into the main Views module in their own (separate) page library files. That way they're always available but don't take up load time.
Similar factoring could occur for hooks as well. Is this the over-factoring you want to avoid?
Factoring out hooks is something I'm not dealing with at the moment, as that offers other challenges I don't want to deal with right now. I was talking more about utility functions that module authors may put into their page files instead of their .module files, for which I propose a solution of "file a bug with the module author for doing it wrong". :-)
I don't like having to pass an additional argument telling the system where things live. I think it should have pretty good defaults that work most of the time, and the path argument only overrides in special cases.
The default that works all of the time is the "leave things in the .module". The additional code to put page callbacks elsewhere is one line per menu item. For forms, it's one line per form item in hook_forms(), although it does add the requirement that one use hook_forms(). I can't think of a simpler way that will work in all cases. --Larry Garfield
Quoting Larry Garfield <larry@garfieldtech.com>:
This is more of an RFC than a DEP, so please forgive the looser format and trademark verbosity. :-)
My current push for Drupal 6 is to make it faster for non-opcode non-cached users. Drupal 5 was, according to Dries' benchmarks, a slight step backwards from Drupal 4.7 in that regard, so let's reverse that trend with a vengeance.
Far and away the slowest part of the Drupal life-cycle is bootstrapping. ...
<dons flame-retardant suit>
[1] http://drupal.org/node/140218 [2] http://drupal.org/node/116165 [3] http://drupal.org/node/140218#comment-236614 [4] http://drupal.org/node/116165#comment-229856
And then there is http://drupal.org/node/116820 Earnie
Doug Green asks an interesting question. I look forward to seeing the answer/response. Suggestion: Since the module author will know best which functions are least called, make this super easy to do and then depend on module authors to optimize their own modules. +1 for most any effort to improve Drupal's non-opcode, non-cached performance. :-)
Larry Garfield wrote:
== Form Handlers ==
Forms are nearly always accessed via drupal_get_form() or drupal_execute(). We can therefore do the same sort of centralized improvement for the form system in those functions as we can for page handlers using
I am totally against form getting its own special handling. 1) logically speaking, pages may or may not be forms; having .page and .form is confusing, because in my mind, pages are forms. 2) pages may also contain forms. Do we then have a .page which also loads a .form because the page contains a form? 3) Sometimes my page is MOSTLY a form, but I have a little extra code, so I don't call drupal_get_form as my callback. I think splitting forms off are clumsy. (As someone who has been driving this work, I don't feel compelled to say that this is all very important, etc, so don't take this as a big negative; just that I think this is, in particular, a poor point to split things at).
On Thursday 03 May 2007, Earl Miles wrote:
Larry Garfield wrote:
== Form Handlers ==
Forms are nearly always accessed via drupal_get_form() or drupal_execute(). We can therefore do the same sort of centralized improvement for the form system in those functions as we can for page handlers using
I am totally against form getting its own special handling.
1) logically speaking, pages may or may not be forms; having .page and .form is confusing, because in my mind, pages are forms.
Sometimes they are, sometimes they aren't. I'd say the two most often used forms are the user login block and the search block. Neither of those is a form.
2) pages may also contain forms. Do we then have a .page which also loads a .form because the page contains a form?
Such is a possibility.
3) Sometimes my page is MOSTLY a form, but I have a little extra code, so I don't call drupal_get_form as my callback.
I think splitting forms off are clumsy.
(As someone who has been driving this work, I don't feel compelled to say that this is all very important, etc, so don't take this as a big negative; just that I think this is, in particular, a poor point to split things at).
The root of the problem is drupal_execute(). drupal_execute() may be called at any time, in any page handler. We also want to ensure that drupal_get_form() may be called at any time. Page handler separation is predicated on the assumption that page handlers are only ever called from menu_execute_active_handler(). That assumption does not hold true for forms. That leaves us 3 options for forms: 1) Forms that are used as a page handler, if factored out to separate files, may never be called from drupal_execute(). I do not consider this acceptable. 2) Forms may not be factored out into separate files at all. Everything still works, but then there's a lot of form code hanging around we never use. 3) Forms get a separate loading mechanism from pages. This way drupal_get_form() and drupal_execute() still function everywhere and we can still factor out forms to separate files where appropriate. I want to emphasize that "where appropriate". Not all handlers or forms should be split out into separate files. For instance, I'd argue the most often used page handler is node_view(), and the most often used forms are the login block and search block. We should *not* split those out into separate files. They're used often enough that their extra weight on other page loads is a good trade off for avoiding the disk hit. Similar decisions can/should be made on a module-by-module basis. So no, we're not ballooning the disk IO this way. -- Larry Garfield AIM: LOLG42 larry@garfieldtech.com ICQ: 6817012 "If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it." -- Thomas Jefferson
On Friday 04 May 2007, Larry Garfield wrote:
Sometimes they are, sometimes they aren't. I'd say the two most often used forms are the user login block and the search block. Neither of those is a form.
And of course I meant to say neither of those is a *page*, since both forms are, clearly, forms. <sigh> -- Larry Garfield AIM: LOLG42 larry@garfieldtech.com ICQ: 6817012 "If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it." -- Thomas Jefferson
Larry Garfield wrote:
1) Forms that are used as a page handler, if factored out to separate files, may never be called from drupal_execute(). I do not consider this acceptable.
This doesn't follow. If I put my form in foo.pages.inc and use drupal_get_form as the callback, and provide my include so that my foo.pages.inc is loaded when that menu is hit, my form will be there. This, of course, isn't true if this form is also available in a non-page-context, which can happen, but in that case the form should be embedded into the .module anyway.
2) Forms may not be factored out into separate files at all. Everything still works, but then there's a lot of form code hanging around we never use.
Again, see above.
3) Forms get a separate loading mechanism from pages. This way drupal_get_form() and drupal_execute() still function everywhere and we can still factor out forms to separate files where appropriate.
Since 1 and 2 don't follow, this conclusion is inaccurate.
I want to emphasize that "where appropriate". Not all handlers or forms should be split out into separate files. For instance, I'd argue the most often used page handler is node_view(), and the most often used forms are the login block and search block. We should *not* split those out into separate files. They're used often enough that their extra weight on other page loads
You actually mean node_page(), not node_view() =) webchick and I had talked about node having two pages files; one that was exclusively for node_page. I still think that's a good idea.
On Fri, 04 May 2007 11:15:12 -0700, Earl Miles <merlin@logrus.com> wrote:
Larry Garfield wrote:
1) Forms that are used as a page handler, if factored out to separate files, may never be called from drupal_execute(). I do not consider this acceptable.
This doesn't follow. If I put my form in foo.pages.inc and use drupal_get_form as the callback, and provide my include so that my foo.pages.inc is loaded when that menu is hit, my form will be there. This, of course, isn't true if this form is also available in a non-page-context, which can happen, but in that case the form should be embedded into the .module anyway.
drupal_execute() is a non-page context. Take for example node_type_form, the form function for which is node_form. It's a non-settings-page form that is used as a parameter to a callback on drupal_get_form, and is rarely used but reasonably large so it's a great candidate for factoring out of the main node.module file. If we factor it out into a pages include file for the node module, then it gets loaded when a user visits admin/content/types/page. Great. What happens, though, if someone calls drupal_execute('node_type_form', ...)? How does the system know where to find that form and how to load it if drupal_execute() doesn't have some knowledge of where forms live? Either drupal_get_form() and drupal_execute() have to be able to dynamically load the form or that form now becomes inaccessible except through the page handler. If you have a suggestion for how to factor page forms out into separate files but not have a form-loading system without breaking drupal_execute() please share it, because I can't think of one.
I want to emphasize that "where appropriate". Not all handlers or forms should be split out into separate files. For instance, I'd argue the most often used page handler is node_view(), and the most often used forms are the login block and search block. We should *not* split those out into separate files. They're used often enough that their extra weight on other page loads
You actually mean node_page(), not node_view() =)
Yes. Yes I do. :-) --Larry Garfield
Larry Garfield wrote:
On Fri, 04 May 2007 11:15:12 -0700, Earl Miles <merlin@logrus.com> wrote:
Larry Garfield wrote:
1) Forms that are used as a page handler, if factored out to separate files, may never be called from drupal_execute(). I do not consider this acceptable. This doesn't follow. If I put my form in foo.pages.inc and use drupal_get_form as the callback, and provide my include so that my foo.pages.inc is loaded when that menu is hit, my form will be there. This, of course, isn't true if this form is also available in a non-page-context, which can happen, but in that case the form should be embedded into the .module anyway.
drupal_execute() is a non-page context.
So is almost every other API call.
Take for example node_type_form, the form function for which is node_form. It's a non-settings-page form that is used as a parameter to a callback on drupal_get_form, and is rarely used but reasonably large so it's a great candidate for factoring out of the main node.module file.
Sure, by itself it is. But node_form is, in core Drupal. only used via node_add and node_edit. Factor those out into their own pages. node forms, however, are a bad example because they are part of a fairly complex specialized hook system which isn't going to be addressed well by just this patch.
If we factor it out into a pages include file for the node module, then it gets loaded when a user visits admin/content/types/page. Great.
What happens, though, if someone calls drupal_execute('node_type_form', ...)? How does the system know where to find that form and how to load it if drupal_execute() doesn't have some knowledge of where forms live? Either drupal_get_form() and drupal_execute() have to be able to dynamically load the form or that form now becomes inaccessible except through the page handler.
Well. To be honest I've been pretty against a lot of the uses of drupal_execute that everyone is so fond of, because it's tying the form to the object which I *hate*, strongly, and I simply don't understand why everyone loves it, beyond getting to cheat a little on validation. So I'm not really *for* this kind of thing; IMO if you're going to use drupal_execute('node_type_form') you should probably have to call an include prior to using it, or node.module includes an API that does that so the user doesn't have to.
For some reason I was lying awake at night thinking about drupal boot strapping (dear lord help me). And it occured to me that although the frequency of use is certainly the right measure to decide what to load, that it probably isn't by technology, so I came up with a couple of different ways to think about this. BY MENU PATH 1) System stuff - api's etc that need loading all the time. 2) node viewing pages (node* paths/routes) (not so sure this would work) 3) User settings (not sure about this one, but maybe anything with a user* menu path 4) admin pages (basically everything with an admin/blah route path You could even consider doing the admin pages based on a permisssions test (access admin pages). The module_name.module files would always load, but maybe there would be room for a modulename.admin.module or a modulename.user.module file that would be conditionally loaded based on either path or user permissions or some other criteria. BY NAMESPACE Or alternatively a namespace based approach where when a module is going to make a call that relies on another module it basically does an "import" of namespace (the way its done in java) This would make a bootstrap call that would load the modules that claim that namespace. (etiher in .info files or in a hook).
On Thu, 10 May 2007 06:43:21 -0700, David Metzler <metzlerd@metzlerd.com> wrote:
For some reason I was lying awake at night thinking about drupal boot strapping (dear lord help me).
I can recommend my therapist. He's used to that by now. :-)
And it occured to me that although the frequency of use is certainly the right measure to decide what to load, that it probably isn't by technology, so I came up with a couple of different ways to think about this.
I'm not sure what you mean by technology?
BY MENU PATH 1) System stuff - api's etc that need loading all the time. 2) node viewing pages (node* paths/routes) (not so sure this would work) 3) User settings (not sure about this one, but maybe anything with a user* menu path 4) admin pages (basically everything with an admin/blah route path
You could even consider doing the admin pages based on a permisssions test (access admin pages). The module_name.module files would always load, but maybe there would be room for a modulename.admin.module or a modulename.user.module file that would be conditionally loaded based on either path or user permissions or some other criteria.
BY NAMESPACE Or alternatively a namespace based approach where when a module is going to make a call that relies on another module it basically does an "import" of namespace (the way its done in java) This would make a bootstrap call that would load the modules that claim that namespace. (etiher in .info files or in a hook).
By permission is interesting. Tweaking menu_execute_active_handler() to do the permission check first before the include should be trivial, and be a small performance boost on 403 pages. That said, right now the patch is breaking down code by module and by function. The "by function" part is up to each module author to decide. That's better than forcing an "admin pages", "other pages", "node pages", etc. split because we'll only ever call one page callback per page load. Why load all 70 admin page callbacks when we're only using one of them? The extreme case is one page callback per file, but I don't think that's a wise way to go in most cases. Different modules, though, will "fit" with different breakdowns. That should be flexibility left to the module author. The namespace approach is what chx suggested with magic function loading. I experimented with it, but eventually decided it was too opaque to deal with at this stage. I want to stick to page callbacks for right now, as those are the biggest dead weight on most page calls. More info in the issue queue: http://drupal.org/node/140218 . Reviews welcome. :-) The sooner we get this in, the sooner we can start optimizing core modules. --Larry Garfield
On 5/10/07, Larry Garfield <larry@garfieldtech.com> wrote:
On Thu, 10 May 2007 06:43:21 -0700, David Metzler <metzlerd@metzlerd.com> wrote:
For some reason I was lying awake at night thinking about drupal boot strapping (dear lord help me).
I can recommend my therapist. He's used to that by now. :-)
One more thing going for Drupal: we don't need the generalists in software, or open source, we have Drupal specialized therapists! Oh ... Druplicon's eyes ...
participants (10)
-
Chris Johnson -
David Metzler -
Doug Green -
Earl Miles -
Earnie Boyd -
Gabor Hojtsy -
Khalid Baheyeldin -
Larry Garfield -
Robert Douglass -
sime