[development] Caching, caching, caching...

[Ken Rickard]

All-

In our large Savannah project (http://new.savannahnow.com) we have been
running some tests with both Jeremy's file-based cache and with object-based
caching.  Some notes.

 1)  I like the filecache patch and have privately sent Jeremy some 4.6 code
that remove tthe *wildcard delete barrier to its usage.  The solution is to
create unique folders for each cache type (variable, filter, page, et. al.)
and then purge that folder when sending cache_clear_all('filter', TRUE).
Happy to share the code with others.

 2) I was under some internal pressure to reduce the sheer volume of db
queries on a page.  Object caching allows that.  In production right now, we
cache the following objects:

 $node
 $user
 $block
 $profile

In the cases of $node, $profile $user, we intercept the _load() function and
check to see if a cached version of the object exists.  It it does, we load
it and skip entirely the hook_user hook_nodeapi hook_load steps of the
code.  If it doesn't exist, we load the object normally and then cache the
result.

In order to make this work, we also have to know when to flush the
individual object cache when a user makes a change.  We flush on node_edit,
block_save and user_edit, for example.

(Apologies for not having good benchmarks, but if memory serves, it
increased our page load speed about 15-20% and took 60-70% of the load off
MySQL).  In the case of a user page, it cuts our queries from around 140 to
25 or fewer after the initial page view.

Some problems / issues / goals going forward:

1) Definitely a cache API.  Many contrib modules (and some 4.6 core) use
cache_clear_all() far too loosely.  As a result, we flush the cache far too
often.

1a) In fact, the only way to clear a cache of any sort is with
cache_clear_all($cid) or cache_clear_all($type, $wildcard = TRUE) or
cache_clear_all().  We need more than one function in order to be more clear
about what the code needs to accomplish.  The block.module, for instance,
runs cache_clear_all() on 3 occassions.  But it never really intends to
clear the entire cache, does it?

2) I'm not sure that I have properly identified all the times to clear the
object cache.

3) Some of this is made unnecessary by the fastcache or page cache, but an
object cache applies to logged in users as well.  For that reason, even
caching the $user object would be a benefit.

4) I am more than happy to contribute notes and code to getting this into
4.8.  The work I've done is all in 4.6.x, but the approach can apply to CVS.

--
Ken Rickard
agentrickard

Message: 5
> Date: Wed, 12 Jul 2006 10:21:49 +0200
> From: Dries Buytaert <dries.buytaert at gmail.com>
> Subject: Re: [development] Caching, caching, caching...
> To: development at drupal.org
> Message-ID: <DB3DEF7E-FA41-4AD3-A210-C177CA4B2853 at gmail.com>
> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
>
>
> On 12 Jul 2006, at 00:01, Gerhard Killesreiter wrote:
> > Now the question is if we want to go down this road. To properly
> > invalidate the cache requires to implement all sorts of hooks (forum
> > module needs _comment and _user for example) which you would not need
> > otherwise. This complicates the code quite a bit.
>
> Personally, I'd not optimize with MySQL's query cache in mind.  If
> the folks at MySQL improve their query cache implementation, our
> efforts might have been moot.  I'd focus my energy on high-level
> improvements, rather than low-level improvements.  That is, I'd focus
> architectural changes that reduce the number of SQL queries or that
> simplify complex queries (eg. less joins).  For example, the node
> table and node_comment_statistics table have a one-on-one relation
> and could potentially be merged.  I'm not sure what it would buy us,
> but it might be worth it.
>
> Maybe we should run the modified devel.module on drupal.org to see
> what the most expensive queries are?  You know, accumulate results
> over multiple page views.  Do we have recent data query data from
> drupal.org?  Let's start by identifying some key bottlenecks.
>
> --
> Dries Buytaert  ::  http://www.buytaert.net/
>
>
>
> ------------------------------
>
> --
> [ Drupal development list | http://lists.drupal.org/ ]
>
> End of development Digest, Vol 43, Issue 38
> *******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.drupal.org/pipermail/development/attachments/20060712/608ec88d/attachment-0001.htm