Hey Chris, I'm probably not the person you aimed this discussion at but I do have something that might be relevant. (I'm just a lowly contrib developer, but I do have a few years of drupal dev under my belt). If I understand correctly where Drupal is going with storage abstraction, I can't see Bigtable as a viable replacement for SQL, so I think I'm in the same camp as you, but I do see folks starting to get interested in using the right storage mechanism for the right task. The PDO SQLLite discussion we had a while ago was I think somewhat analogous. (although admittedly not the same) I could see using some kind of BigTable style storage for storing the cached data for drupal and therefore scaling cached pages across servers. Then the cache wouldn't rely strongly on the important relational data in a backend mysql or postgres engine. This seems like a productive use of those kind of storage techniques. I could even see caching common search results using the cache api in the search module (if it doesn't already). There are many in this community who will, rightly so, express concern about the added complexity of relying on multiple storage engines, but this is one case where I think the results might be worth it. Dave On May 5, 2009, at 9:23 AM, Chris Johnson wrote:
Perhaps before this discussion gets too far astray -- or maybe such digression is impossible to avoid! :-)
My point in posting this was to elicit folks who were interested in the database usage and performance for the core parts of Drupal, and in particular the folks who I through the past years have observed as being especially astute and involved (and if I left your name off that short list, it's only because I forgot your name or other such failure on my part -- no slight or insult intended).
I was particularly hopeful that folks would read the comments, and take to heart those written in favor of SQL, such as those by Robert Young. I didn't want to bias the conversation by injecting my opinion too early, but I'm in the camp that says things like BigTable are really just reinventions of older technology, AND that the death of RDBMS/SQL is not nearly as soon as some might think.
For Drupal, I think it may make sense to attempt to make the SQL as simple as possible, avoiding the really complex capabilities and doing more of that work in PHP, where most of the developers actually have some expertise. This is especially true since MySQL's MyISAM table engine is really poor at doing anything beyond that. If we were to move to using MVCC[1] engines like Postgres, Oracle and MySQL with InnoDB or Falcon, then more complex SQL would be useful. Unfortunately, most Drupal developers don't have the SQL skills needed, and even worse, most people are not going to have the ability to physically optimize the database for performance (i.e. skilled DBA person, access to MySQL settings at most hosting services, etc.).
Hence, the former course of using only the simplest, most primitive SQL statements appeals to me for Drupal.
It's that last point I'd really like to have criticized. Is it possible to do that? Especially in light of DBTNG, where is the line between simple statements and complex statements?
Lastly, although many Drupal-based sites are database limited, many are PHP-code interpretation (# of files/lines parsed) or bandwidth (fast JS/image pages) limited, too. My goal is simple: make my hundred module websites respond in less than 2 seconds! :-D
Thanks for participating in this discussion, no matter which tangent you're on.
..chris
[1] http://en.wikipedia.org/wiki/Multiversion_concurrency_control