[development] memcache

[Steve Rude]

I don't want to hijack the file cache maintainer thread, so I am starting my
own thread about this since I have some questions.

I did talk to Robert about this in January, and I have done extensive
testing with his code.  His code patches core files all over the place in
order to check memcache if there is a cached version available and then load
that first.  He has stated at http://drupal.org/project/memcache that "A set
of patches that can be applied selectively to Drupal to cache various
objects for high performance. The patches are necessary and this cannot be
avoided."

However, if we are doing object level caching for nodes than we can cache a
complete loaded node without the need to hit the database until a node_save,
until a new module is enabled or until the max life is hit.

If the code in node_load hit the static variable $nodes first, then secondly
hit cache_get then we would not need to have a set of patches, and this
could be an option in the performance settings.  You could then allow the
pluggable cache.inc that I just realized has been merged into HEAD (As of
January) to handle storage.  This would only take a small change to the core
modules in a few places.

For example in node.module,

if (is_numeric($param)) {
    if ($cachable && isset($nodes[$param])) {
      return is_object($nodes[$param]) ? drupal_clone($nodes[$param]) :
$nodes[$param];
    }
    $cond = 'n.nid = %d';
    $arguments[] = $param;
  }

could be changed into something like this:

if (is_numeric($param)) {
  if($cachable) {
    if (isset($nodes[$param])) {
      return is_object($nodes[$param]) ? drupal_clone($nodes[$param]) :
$nodes[$param];
    }
    //Attempt to load from cache return false on cache miss
    $node = cache_get($param, 'cache_node');
    if($node) {
      //Load into static $nodes
      $nodes[$param] = is_object($node) ? drupal_clone($node) : $node;
      return $node;
    }
  }
  $cond = 'n.nid = %d';
  $arguments[] = $param;
}

I have time to work on this and offer performance testing.  I am just not
sure how far anyone has gone down these roads or is currently working on
it.  I have been working on custom module development for clients for about
7 months now and I have not really had time to contribute to the Drupal
project personally other than by spreading the good word and having clients
use Drupal.  This is one area that I feel could be substantially improved
upon in order for us to allow Drupal to scale beyond the current ability.

According to Dries testing on his website, he has seen Drupal into the 80+
Requests per second area (ref.
http://buytaert.net/drupal-webserver-configurations-compared).  With some of
the testing I have done with caching, I have seen it go as high as 500-700
Requests per second.

In my testing out of the box I have seen Drupal do about 10 requests per
second, with normal caching about 100 requests per second and with
aggressive caching as high as 250 request per second.

Thanks,

Steve Rude


On 3/6/07, Khalid Baheyeldin <kb at 2bits.com> wrote:
>
> See this http://drupal.org/project/memcache
>
> Talk to Robert about it before you duplicate it.
>
> On 3/6/07, Steve Rude <steve at achieveinternet.com> wrote:
> >
> > I have done much testing with memory based object caching using
> > memcached and I think it would be nice if there was the ability to choose
> > what type of caching you wanted (memory/file/db) and have the cache.incfunctionality basically an abstraction layer that the storage mechanism can
> > be taken care of by other include files ( e.g. cache-memcache.inc
> > cache-file.inc.  The same way that the database abstraction works.
> >
> > I would be willing to begin some work on this if anyone is interested.
> > We are building a Drupal site now that will need to scale to >30m page views
> > per month and I would like to work within an accepted method to help grow
> > and scale Drupal.
> >
> > If anyone has any prior work or projects that can be looked at to help
> > get this going down the correct path, please point me in the right
> > direction.
> >
> > Thanks,
> >
> > Steve Rude
> >
> > On 3/6/07, Karoly Negyesi <karoly at negyesi.net> wrote:
> > >
> > > Hi,
> > >
> > > I might be mistaken but I thought file caching was something much
> > > sought
> > > after. I found posts relating to it as early as 2002.
> > >
> > > And yet, when the possibility has arisen for it to happen cleanly
> > > (pluggable cache.inc), the code for it has been dumped into CVS on
> > > August
> > > 26, 2006 and not a single cvs commit since then.
> > >
> > > It's not used much according to the issues but that probably has
> > > something
> > > to do with not having a single release.
> > >
> > > Is the community truly interested? If yes, then please a maintainer
> > > step
> > > up.
> > >
> > > And yes, I have contacted the current maintainers three months ago to
> > > no
> > > avail.
> > >
> > > Regards,
> > >
> > > NK
> > >
> >
> >
> >
> > --
> >
> > Steve Rude
> > Lead Web Developer
> >
> >     800-618-8777phone / 858-225-0479 fax
> >      www.achieveinternet.com
> >      steve at achieveinternet.com
> >
> >     Achieve Internet is a Division of
> >     Web Page Maintenance, Inc.
>
>
>
>
> --
> 2bits.com
> http://2bits.com
> Drupal development, customization and consulting.
>



-- 

Steve Rude
Lead Web Developer

    800-618-8777phone / 858-225-0479 fax
    www.achieveinternet.com
    steve at achieveinternet.com

    Achieve Internet is a Division of
    Web Page Maintenance, Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.drupal.org/pipermail/development/attachments/20070306/736f2a9f/attachment-0001.htm