[development] Database / SQL future thoughts

[Larry Garfield]

On Tuesday 05 May 2009 8:39:08 pm David Metzler wrote:
> Hey Chris,
>
> I'm probably not the person you aimed this discussion at but I do
> have something that might be relevant.  (I'm just a lowly contrib
> developer, but I do have a few years of drupal dev under my belt).
>
> If I understand correctly where Drupal is going with storage
> abstraction, I can't see Bigtable as a viable replacement for SQL, so
> I think I'm in the same camp as you, but I do see folks starting to
> get interested in using the right storage mechanism for the right
> task.  The PDO SQLLite discussion we had a while ago was I think
> somewhat analogous. (although admittedly not the same)
>
> I could see using some kind of BigTable style storage for storing the
> cached data for  drupal and therefore scaling cached pages across
> servers.  Then the cache wouldn't rely strongly on the important
> relational data in a backend mysql or postgres engine.  This seems
> like a productive use of those kind of storage techniques.  I could
> even see caching common search results using the cache api in the
> search module (if it doesn't already).
>
> There are many in this community who will, rightly so, express
> concern about the added complexity of relying on multiple storage
> engines, but this is one case where I think the results might be
> worth it.
>
> Dave

In many many cases, the storage engine depends on your scale.  For the run of 
the mill Drupal site, for instance, MySQL-based caching is perfectly fine and 
is also the lowest barrier to entry (since the main system runs on SQL).  For 
higher end sites, memcache, CouchDB, Flatfile, or whatever else could make more 
sense.  Fortunately those high end sites are also the ones run by people who, 
if we provide them with proper mechanisms to swap out backends cleanly, would 
know how to do so and why.  

So we just need to make more key parts of Drupal pluggable and those who 
actually care about such differences can leverage them as appropriate, and 
those who don't need to care won't have to.

-- 
Larry Garfield
larry at garfieldtech.com