[consulting] Does anyone offer Drupal performance tuning services?

Mon Apr 27 16:22:06 UTC 2009

Maybe I shouldn't wade into this with my highly opinionated take on
things, but I'm feeling curmudgeonly today.

First, some free advice.  As alluded to by another writer, very often
performance problems are caused by badly written modules.  Most of
Drupal's contrib code is, in fact, not written in a very optimal way
for performance.  This is not a criticism, by the way.  Functionality
first, after all and avoid the premature optimization.

If one is running a D6 site with a large number of modules, the sheer
number of modules plus the fact that most of them are probably written
"inefficiently" is probably the biggest cause of performance problems
-- absent any really gross configuration errors (missing database
indexes, tiny memory sizes, etc.).  As someone who is running over a
hundred contrib modules on large number of sites, I see this situation
all the time.

Lots of modules means lots of PHP code.  Simply opening, reading and
parsing all those PHP files takes a significant amount of time per
request.  This is one big reason why PHP opcode caches like APC are
one of the key first steps in making Drupal run faster.  It's also why
well written modules include avoiding loading code that won't be used
for the request when possible.

Second, complexity of the performance problem space .  Despite there
being some large, obvious places to look for performance problems,
it's really a very large, complicated problem.  There are almost
innumerable factors involved.  Let me just list a few of them:

1.  efficiency of the code of the modules being used.
2.  end case inefficiencies in core (I'm being charitable, perhaps).
3.  anomalous edge case SQL queries.
4.  wrong PHP configuration (fairly simple).
5.  wrong Apache configuration (quite more complex).
6.  wrong database configuration (even more complex).
7.  host system configuration errors (large problem space).
8.  host system limitations (obscure problem space).
9.  VPS configuration errors (obscure problem space.
10. network problems (large can of worms by itself).

What this means is, I don't believe any one person knows enough to
actually solve all problems in all problem spaces.  Probably not any 2
people, either.  It probably takes 3 or 4 people, say a Drupal expert,
a DB expert, a host / sysadmin expert and a senior generalist, and
then maybe even a network expert, if it turns out that's where the
problem is, as discovered by the previous 3 or 4 experts.

That's not to say one guy can't make a big improvement, or enough
improvement to satisfy the client.  It IS a tradeoff of money versus
incremental improvement.  But to get every last bit of performance out
of a system is a really a Hard Problem -- it IS rocket science, so to
speak.

The team I work with (about a dozen folks at OpenBand) will be
optimizing the heck out of our large D6 distribution over the next few
months.  It's going to be a very interesting experience.

On Mon, Apr 27, 2009 at 10:12 AM, Sam Cohen <sam at samcohen.com> wrote:
>>
>>
>> Sam, are you indeed asking of a consultant to essentially guarantee "x"
>> amount of utility in advance of taking the job? If you remove that from you
>> spec (or maybe you didn't even intend that), I think folks are close to
>> being on the same page.
>>
>> Shai
>
> Only a guarantee is the sense that they will apply their expertise to
> optimize the server -- as much as they can in the time they have allotted --
> and they will analyze the site and make recommendations for any further work
> that needs to be done.
>
> I don't expect miracles, just a few hours work from someone who knows what
> they're doing.
>
> My expectations are such that the consultant might not do anything other
> than analyze it and may come to the conclusion that everything is as
> optimized as it can be and the only solution would to spend a lot more to do
> x, y, and z.  That would be fine too.
>
> Sam
>
>
>
>
>
>
>
>
>
>
>
>>
>> On Mon, Apr 27, 2009 at 10:35 AM, Sam Cohen <sam at samcohen.com> wrote:
>>>
>>>>
>>>>
>>>> Incidentally, I'm very curious why you assume that going through all of
>>>> your server logs, analyzing your queries, going through "lots of modules"
>>>> and cleaning up any messes found would take a few hours?
>>>
>>> I never said the few hours would include "cleaning up any messes."  I
>>> said they can make a few suggestions.  For example,  coming back with some
>>> recommendations, such as saying  xyz module is causing a lot of the problems
>>> and you may want to take the following steps.
>>>
>>>
>>>>
>>>> It reminds me of all the people who ask us for help on a site or with a
>>>> problem that tell us "it's really an easy site/fix", to which I always want
>>>> to say: "fine, if it's so easy, do it yourself." Really, if it was so easy
>>>> (I promise you it is not), would you be spending your time looking for an
>>>> expert?
>>>
>>> Thanks Alex.  It's always nice to know that I remind you of the type of
>>> person you consider yourself far superior to.
>>>
>>> I never said I thought it was easy, which is why I want to hire an
>>> expert.  Your point of disagreement seems to be that you don't think
>>> anything of value can be accomplished in a few hours.  It seems to be that
>>> quite a bit can be accomplished.  No, it won't be as much as a weeks worth
>>> of improvements, but at the very least the server can be tweaked and
>>> optimized to best handle a Drupal site.
>>>
>>> Sam
>>>
>>>>
>>>>
>>>> --
>>>> Alex Urevick-Ackelsberg
>>>> ZivTech, LLC
>>>> http://zivtech.com
>>>> alex at zivtech.com
>>>> office: (267) 940-7737
>>>> cell: (215) 866-8956
>>>> skype: zivtech
>>>> aim: zivtech
>>>>
>>>>
>>>> On Mon, Apr 27, 2009 at 8:44 AM, Sam Cohen <sam at samcohen.com> wrote:
>>>>>
>>>>>
>>>>> On Sun, Apr 26, 2009 at 3:03 PM, Alex Urevick-Ackelsberg
>>>>> <Alex at zivtech.com> wrote:
>>>>>>
>>>>>> Like Greg said, if you can find someone willing to do this for a flat
>>>>>> fee, then I'd love to hear more about it. As far as I can tell, there are
>>>>>> far too many variables to make it possible to estimate accurately up front.
>>>>>> If I were trying to figure out a fixed price I'd personally try to think
>>>>>> about what the highest possible price could be, and then charge that price.
>>>>>> Otherwise, you are asking the consultant to take a considerable financial
>>>>>> risk.
>>>>>>
>>>>>> As Khalid mentioned, many shops that handle scaling will offer you
>>>>>> more of a fixed price if they control the servers, which seems reasonable.
>>>>>>
>>>>>> I, for one, think that you are seriously undervaluing server and
>>>>>> general LAMP stack support, but maybe I'm not understanding what you are
>>>>>> looking for. What do you mean by, for example, "Load Balancing"? That makes
>>>>>> it seem to me that you are talking about scaling a site that requires more
>>>>>> than one server, but given the budgets you seem to be working with, that
>>>>>> doesn't seem like a good assumption (someone who requires more than one
>>>>>> server should be able to afford to pay market rates to get it done
>>>>>> correctly).
>>>>>>
>>>>>> So, my question is: what exactly are the problems that you are facing
>>>>>> that require a server/scaling specialist?
>>>>>
>>>>>
>>>>>
>>>>> The scenario here is a D6 site on a VPS with a lot of modules that is
>>>>> quite sluggish.  What I'd hope someone can accomplish in a  couple of hours
>>>>> would be to analyze all server and software configs to see if anything can
>>>>> be installed or tweaked to improve performance.  This might include MySQL,
>>>>> PHP, Apache settings -- or even if there's any unecessary software on the
>>>>> VPS than can be disabled or tweaked. And installing anything that might
>>>>> help.  There's a lot of information out there on ways to do this yourself,
>>>>> but I'd rather hire someone who knows what they are doing to take care of
>>>>> it.  I'd also like them, if it's possible, to be able to test the VPS to
>>>>> make sure that it's performing as well as it should be.
>>>>>
>>>>> As for the site, I imagine that this person can use devel module to
>>>>> analyze the query queue and perhaps make some suggestions -- maybe there's a
>>>>> poorly written module that's eating up a lot of resources.  Maybe there's a
>>>>> caching module I overlooked.  Maybe there's alternate approaches to getting
>>>>> the job done that wouldn't use as many resources.
>>>>>
>>>>> It seems to me someone should be able to accomplish a lot in these two
>>>>> areas in a few hours -- and it would be a great service I'd love to be able
>>>>> to offer to my clients.
>>>>>
>>>>> (As for load balancing, my thinking there was with regard to a managed
>>>>> dedicated server I run that has about 30 drupal sites. My thinking was that
>>>>> such a person might know ways to install software that would allow me to
>>>>> balance the load so that one site wouldn't be using up more resources -- or
>>>>> at least help me analyze what resources each site is using.)
>>>>>
>>>>> Maybe I'm wrong, but it does seem like for a fixed rate for a few hours
>>>>> someone who knows what they are doing should be able to accomplish a lot and
>>>>> make recommendations for going forward.  It would also be nice to have such
>>>>> a person available as a site grows and if it needs to scale.
>>>>>
>>>>> Thanks,
>>>>> Sam
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> --
>>>>>> Alex Urevick-Ackelsberg
>>>>>> ZivTech, LLC
>>>>>> http://zivtech.com
>>>>>> alex at zivtech.com
>>>>>> office: (267) 940-7737
>>>>>> cell: (215) 866-8956
>>>>>> skype: zivtech
>>>>>> aim: zivtech
>>>>>>
>>>>>>
>>>>>> On Sat, Apr 25, 2009 at 11:34 AM, Sam Cohen <sam at samcohen.com> wrote:
>>>>>>>
>>>>>>> Thanks Greg,
>>>>>>>
>>>>>>> I understand there are a lot of firms offering this service, but what
>>>>>>> I'm really hoping for is someone who had a deep understanding of server and
>>>>>>> performance issues and can spend a couple of hours analyzing a site and
>>>>>>> making adjustments.
>>>>>>>
>>>>>>> I'm not looking for a large shop that will charge thousands of
>>>>>>> dollars for this service, but a consultant who has expertise in this area
>>>>>>> and would charge fair market value for a few hours of their time.
>>>>>>>
>>>>>>> By flat rate, I mean a willingness to give a price upfront based on
>>>>>>> the site specs and hosting situation.
>>>>>>>
>>>>>>> Perhaps I'm wrong, but it seems that someone with this expertise,
>>>>>>> might be charging somewhere in $50 to $100 an hour range, and in just a
>>>>>>> couple of hours they should be able to accomplish quite a lot for a site.
>>>>>>>
>>>>>>> This would be a really valuable service for those of us in the biz
>>>>>>> who might be very knowledgeable in other areas but don't know much about
>>>>>>> servers, bottlenecks, load balance, etc -- and don't really want to learn
>>>>>>> either.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Sam
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Apr 25, 2009 at 10:43 AM, Greg Knaddison
>>>>>>> <Greg at growingventuresolutions.com> wrote:
>>>>>>>>
>>>>>>>> We hit this topic less than a month ago -
>>>>>>>> http://lists.drupal.org/pipermail/consulting/2009-March/003088.html
>>>>>>>>
>>>>>>>> And read the followups as well.
>>>>>>>>
>>>>>>>> Also, charging for performance tuning as a flat rate is pretty
>>>>>>>> interesting.  If you find anyone who takes that arrangement please
>>>>>>>> write back to the list about the experience.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Greg
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Apr 24, 2009 at 8:17 AM, Sam Cohen <sam at samcohen.com> wrote:
>>>>>>>> > Does anyone out there offer a Drupal performance site and server
>>>>>>>> > tuning
>>>>>>>> > service -- for a flat rate?
>>>>>>>> >
>>>>>>>> > Ideally this would include analyzing a site, making changes to the
>>>>>>>> > site and
>>>>>>>> > server as needed in order to improve performance.
>>>>>>>> >
>>>>>>>> > I think this service would probably be useful to a lot of us, but
>>>>>>>> > I
>>>>>>>> > understand there's that concern about sharing rates over an email
>>>>>>>> > list, so
>>>>>>>> > please email me privately with what you would include and what you
>>>>>>>> > would
>>>>>>>> > charge for such a service.
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Greg Knaddison
>>>>>>>> http://knaddison.com | 303-800-5623 |
>>>>>>>> http://growingventuresolutions.com
>>>>>>>> _______________________________________________
>>>>>>>> consulting mailing list
>>>>>>>> consulting at drupal.org
>>>>>>>> http://lists.drupal.org/mailman/listinfo/consulting
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> consulting mailing list
>>>>>>> consulting at drupal.org
>>>>>>> http://lists.drupal.org/mailman/listinfo/consulting
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> consulting mailing list
>>>>>> consulting at drupal.org
>>>>>> http://lists.drupal.org/mailman/listinfo/consulting
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> consulting mailing list
>>>>> consulting at drupal.org
>>>>> http://lists.drupal.org/mailman/listinfo/consulting
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> consulting mailing list
>>>> consulting at drupal.org
>>>> http://lists.drupal.org/mailman/listinfo/consulting
>>>>
>>>
>>>
>>> _______________________________________________
>>> consulting mailing list
>>> consulting at drupal.org
>>> http://lists.drupal.org/mailman/listinfo/consulting
>>>
>>
>>
>> _______________________________________________
>> consulting mailing list
>> consulting at drupal.org
>> http://lists.drupal.org/mailman/listinfo/consulting
>>
>
>
> _______________________________________________
> consulting mailing list
> consulting at drupal.org
> http://lists.drupal.org/mailman/listinfo/consulting
>
>