[development] Database / SQL future thoughts

[David Metzler]

Hey Chris,

I'm probably not the person you aimed this discussion at but I do  
have something that might be relevant.  (I'm just a lowly contrib  
developer, but I do have a few years of drupal dev under my belt).

If I understand correctly where Drupal is going with storage  
abstraction, I can't see Bigtable as a viable replacement for SQL, so  
I think I'm in the same camp as you, but I do see folks starting to  
get interested in using the right storage mechanism for the right  
task.  The PDO SQLLite discussion we had a while ago was I think  
somewhat analogous. (although admittedly not the same)

I could see using some kind of BigTable style storage for storing the  
cached data for  drupal and therefore scaling cached pages across  
servers.  Then the cache wouldn't rely strongly on the important  
relational data in a backend mysql or postgres engine.  This seems  
like a productive use of those kind of storage techniques.  I could  
even see caching common search results using the cache api in the  
search module (if it doesn't already).

There are many in this community who will, rightly so, express  
concern about the added complexity of relying on multiple storage  
engines, but this is one case where I think the results might be  
worth it.

Dave

On May 5, 2009, at 9:23 AM, Chris Johnson wrote:

> Perhaps before this discussion gets too far astray -- or maybe such
> digression is impossible to avoid!  :-)
>
> My point in posting this was to elicit folks who were interested in
> the database usage and performance for the core parts of Drupal, and
> in particular the folks who I through the past years have observed as
> being especially astute and involved (and if I left your name off that
> short list, it's only because I forgot your name or other such failure
> on my part -- no slight or insult intended).
>
> I was particularly hopeful that folks would read the comments, and
> take to heart those written in favor of SQL, such as those by Robert
> Young.  I didn't want to bias the conversation by injecting my opinion
> too early, but I'm in the camp that says things like BigTable are
> really just reinventions of older technology, AND that the death of
> RDBMS/SQL is not nearly as soon as some might think.
>
> For Drupal, I think it may make sense to attempt to make the SQL as
> simple as possible, avoiding the really complex capabilities and doing
> more of that work in PHP, where most of the developers actually have
> some expertise.  This is especially true since MySQL's MyISAM table
> engine is really poor at doing anything beyond that.  If we were to
> move to using MVCC[1] engines like Postgres, Oracle and MySQL with
> InnoDB or Falcon, then more complex SQL would be useful.
> Unfortunately, most Drupal developers don't have the SQL skills
> needed, and even worse, most people are not going to have the ability
> to physically optimize the database for performance (i.e. skilled DBA
> person, access to MySQL settings at most hosting services, etc.).
>
> Hence, the former course of using only the simplest, most primitive
> SQL statements appeals to me for Drupal.
>
> It's that last point I'd really like to have criticized.  Is it
> possible to do that?  Especially in light of DBTNG, where is the line
> between simple statements and complex statements?
>
> Lastly, although many Drupal-based sites are database limited, many
> are PHP-code interpretation (# of files/lines parsed) or bandwidth
> (fast JS/image pages) limited, too.  My goal is simple:  make my
> hundred module websites respond in less than 2 seconds!  :-D
>
> Thanks for participating in this discussion, no matter which  
> tangent you're on.
>
> ..chris
>
> [1] http://en.wikipedia.org/wiki/Multiversion_concurrency_control