From larry at garfieldtech.com Tue Jul 1 16:17:58 2008 From: larry at garfieldtech.com (Larry Garfield) Date: Tue, 1 Jul 2008 11:17:58 -0500 Subject: [Database] Test message Message-ID: <279013fe0d975be509bb440727483cc0@localhost> Let's make sure this goes through. --Larry Garfield From larry at garfieldtech.com Sun Jul 6 19:32:06 2008 From: larry at garfieldtech.com (Larry Garfield) Date: Sun, 6 Jul 2008 14:32:06 -0500 Subject: [Database] Toward better databases Message-ID: <200807061432.07002.larry@garfieldtech.com> Hi all. Curiously Dries hasn't joined the list yet, but everyone else has so I'll make a welcome post. :-) I'd like to introduce: Joshua Drake - PostgreSQL guru extraordinaire that Dries put me in touch with when we hit a wall on the Postgres driver. Barry Jaspan - Barry stepped forward to write the postgres driver for the new database API, which is a good thing, too, because it forced me to rewrite most of it. :-) David Strauss - Drupal's resident MySQL-fu master. Narayan Newton - The guy who actually runs our server cluster, and also knows MySQL better than most. Larry Garfield - The poor sod who thought rewriting the database layer would only take a month or two. :-) If there's any other MySQL or Postgres or SQLite or PDO gurus you know, feel free to invite them. Long term there is plenty that we can and should do to improve our database handling. Short-term, though, I just want to get the damned patch[1] committed. :-) Right now, as far as I can determine the core system and MySQL are done and working and solid. The problem right now is the PostgreSQL driver. I finally have a working postgres install, but every time I try to install Drupal with the Postgres driver PHP itself dies on me. I've been trying to get a debugger working at home so that I can investigate, but so far nothing has actually worked. :-) Barry, is that something you'll be able to look into in the near future? You've spent far more time with the postgres driver than I have. If not, I'll keep trying to get a debugger working or Josh, if you're willing I can bring you up to speed on the architecture and you can have a look-see. The other problem we realized recently is that we are relying on Schema API to tell us if we need to BLOB-escape a bytea field in PostgreSQL (which does seem to be the case), and then similar information for other, sillier databases like Oracle. Schema API only tells us about the main Drupal database. That means only MySQL and SQLite can be used for foreign databases. This is a problem. The solution we came up with is to integrate a table-schema-derivation method into the new API so that we can pick such things up on the fly as needed. At that point I just sort of ran out of steam and Barry got busy, so that's not happened yet. Since you can't mix and match database drivers in D6 anyway, I am tempted to ask Dries if we can add that in a follow-up patch because right now this patch is blocking a lot of other work (including cleaning up Schema API, which may not even happen in D7 but needs to now that it's been OOPified). So, that's where we stand right now. Postgres support is blocking pretty much everything. So, Barry, Josh, can either of you lend a hand while I keep trying to get a debugger to play nice? [1] http://drupal.org/node/225450 -- Larry Garfield larry at garfieldtech.com From barry.jaspan at acquia.com Mon Jul 7 13:44:40 2008 From: barry.jaspan at acquia.com (Barry Jaspan) Date: Mon, 7 Jul 2008 09:44:40 -0400 Subject: [Database] Toward better databases In-Reply-To: <200807061432.07002.larry@garfieldtech.com> References: <200807061432.07002.larry@garfieldtech.com> Message-ID: Howdy, everyone. Here's my current brain-dump: 1. Larry is correct that the pgsql driver is not currently working. I haven't had a chance to bring it up to date with the latest API/ internal changes Larry made in the PDO patch. However, there is not actually any difficulty related to pgsql specifically; interacting with pgsql through the PDO_PGSQL interface works fine and I've had everything (including the blob support) working fine before. It's just a question of updating it for the latest version of the patch, which I haven't done yet. You know, the day job keeps me pretty busy. :-) 2. The database inspection code for pgsql (and mysql) is already written and working in Drupal's schema.module. This code will allow the database layer to determine the type of a table column that does not exist in Drupal's schema. I do not think schema.module in its current state is ready to be committed to core. However, we don't really need all of it, or even all of the database inspection code. Right now all we need is something like db_inspect_column($table, $column) which returns a Schema API field structure for the column, including its type. The subset of schema.module should not be hard to pull out and add to the PDO patch. One snafu, however: db_inspect_column() may require database privileges that other Drupal operations do not. We'll have to look into that and whether it is an acceptable requirement (the answer will probably be "yes, because there is no alternative"). 3. Larry isn't going to like this, but part of the reason I haven't mustered the time to finish the pgsql driver is that I am not 100% convinced the design for the PDO-based database layer is right yet, and figuring out a proposal to fix it requires more time than I've had. 3a. One issue is that the way that SQL and placeholders are generated seems fragile and inefficient to me. It ends up being fairly awkward for the pgsql driver to keep everything straight in the way it needs to in order to do its most basic function of calling bindParam() correctly (though it is much better than in early drafts of the patch). 3b. I also have some concerns that the public-facing API is more complex and incongruent than it needs to be (I've expressed these to Larry before, and he disagrees). 3c. Also, the new database layer is substantially more complex and difficult to understand than our current layer (just trying to determine exactly where queries get executed is a challenge) and I'm concerned it will both meet substantial community resistance and (more importantly) substantially reduce the number of people who are able to contribute to it. Of course, this is equally true of the Fields in Core project I'm working on, so maybe it is just unavoidable at this point in Drupal's development. Perhaps the existence of this mailing list will motivate me to re- visit the pgsql driver, try to get it working again, and in the process figure out what I really think regarding at least 3a and 3b. Oh, and look into 2 as well. :-) Thanks, Barry ------- Barry Jaspan Code Jester, Acquia Inc. barry.jaspan at acquia.com On Jul 6, 2008, at 3:32 PM, Larry Garfield wrote: > Hi all. Curiously Dries hasn't joined the list yet, but everyone > else has so > I'll make a welcome post. :-) > > I'd like to introduce: > > Joshua Drake - PostgreSQL guru extraordinaire that Dries put me in > touch with > when we hit a wall on the Postgres driver. > > Barry Jaspan - Barry stepped forward to write the postgres driver > for the new > database API, which is a good thing, too, because it forced me to > rewrite > most of it. :-) > > David Strauss - Drupal's resident MySQL-fu master. > > Narayan Newton - The guy who actually runs our server cluster, and > also knows > MySQL better than most. > > Larry Garfield - The poor sod who thought rewriting the database > layer would > only take a month or two. :-) > > If there's any other MySQL or Postgres or SQLite or PDO gurus you > know, feel > free to invite them. Long term there is plenty that we can and > should do to > improve our database handling. Short-term, though, I just want to > get the > damned patch[1] committed. :-) > > Right now, as far as I can determine the core system and MySQL are > done and > working and solid. The problem right now is the PostgreSQL driver. I > finally have a working postgres install, but every time I try to > install > Drupal with the Postgres driver PHP itself dies on me. I've been > trying to > get a debugger working at home so that I can investigate, but so far > nothing > has actually worked. :-) Barry, is that something you'll be able to > look > into in the near future? You've spent far more time with the > postgres driver > than I have. If not, I'll keep trying to get a debugger working or > Josh, if > you're willing I can bring you up to speed on the architecture and > you can > have a look-see. > > The other problem we realized recently is that we are relying on > Schema API to > tell us if we need to BLOB-escape a bytea field in PostgreSQL (which > does > seem to be the case), and then similar information for other, sillier > databases like Oracle. Schema API only tells us about the main Drupal > database. That means only MySQL and SQLite can be used for foreign > databases. This is a problem. > > The solution we came up with is to integrate a table-schema- > derivation method > into the new API so that we can pick such things up on the fly as > needed. At > that point I just sort of ran out of steam and Barry got busy, so > that's not > happened yet. Since you can't mix and match database drivers in D6 > anyway, I > am tempted to ask Dries if we can add that in a follow-up patch > because right > now this patch is blocking a lot of other work (including cleaning > up Schema > API, which may not even happen in D7 but needs to now that it's been > OOPified). > > So, that's where we stand right now. Postgres support is blocking > pretty much > everything. So, Barry, Josh, can either of you lend a hand while I > keep > trying to get a debugger to play nice? > > [1] http://drupal.org/node/225450 > > -- > Larry Garfield > larry at garfieldtech.com > _______________________________________________ > Database mailing list > Database at drupal.org > http://mail.drupal.org/mailman/listinfo/database From larry at garfieldtech.com Tue Jul 8 01:42:04 2008 From: larry at garfieldtech.com (Larry Garfield) Date: Mon, 7 Jul 2008 20:42:04 -0500 Subject: [database] [Database] Toward better databases In-Reply-To: References: <200807061432.07002.larry@garfieldtech.com> Message-ID: <200807072042.04709.larry@garfieldtech.com> On Monday 07 July 2008 8:44:40 am you wrote: > Howdy, everyone. Here's my current brain-dump: > > 1. Larry is correct that the pgsql driver is not currently working. > I haven't had a chance to bring it up to date with the latest API/ > internal changes Larry made in the PDO patch. However, there is not > actually any difficulty related to pgsql specifically; interacting > with pgsql through the PDO_PGSQL interface works fine and I've had > everything (including the blob support) working fine before. It's > just a question of updating it for the latest version of the patch, > which I haven't done yet. You know, the day job keeps me pretty > busy. :-) I know the feeling. :-) However, the problem I'm running into I have managed to track down (10 minutes with a working debugger is worth weeks without one, dear god!), and it is highly confusing. The problem comes when creating the variable table. All of the test queries run fine, but when it gets to this query: CREATE TABLE variable ( name varchar(128) NOT NULL default '', value text NOT NULL, PRIMARY KEY (name) ) PHP dies completely. I have not figured out why. That query runs fine within the psql command line, and it runs fine in a sample script I've created that, I think, is mimicing the way the new Drupal driver works. I am highly confused by this, and it is making me sad. :-( Josh, do you have any idea why the above query could be failing? > 2. The database inspection code for pgsql (and mysql) is already > written and working in Drupal's schema.module. This code will allow > the database layer to determine the type of a table column that does > not exist in Drupal's schema. I do not think schema.module in its > current state is ready to be committed to core. However, we don't > really need all of it, or even all of the database inspection code. > Right now all we need is something like > > db_inspect_column($table, $column) > > which returns a Schema API field structure for the column, including > its type. The subset of schema.module should not be hard to pull out > and add to the PDO patch. > > One snafu, however: db_inspect_column() may require database > privileges that other Drupal operations do not. We'll have to look > into that and whether it is an acceptable requirement (the answer will > probably be "yes, because there is no alternative"). > > 3. Larry isn't going to like this, but part of the reason I haven't > mustered the time to finish the pgsql driver is that I am not 100% > convinced the design for the PDO-based database layer is right yet, > and figuring out a proposal to fix it requires more time than I've had. > > 3a. One issue is that the way that SQL and placeholders are generated > seems fragile and inefficient to me. It ends up being fairly awkward > for the pgsql driver to keep everything straight in the way it needs > to in order to do its most basic function of calling bindParam() > correctly (though it is much better than in early drafts of the patch). Once I actually get the damned thing installed, I will take a look and see if it can be simplified. > 3b. I also have some concerns that the public-facing API is more > complex and incongruent than it needs to be (I've expressed these to > Larry before, and he disagrees). It is as simple as we can make it and meet the desired functionality. :-) (Actually it has changed considerably based on your input so far, which has been very good for it.) > 3c. Also, the new database layer is substantially more complex and > difficult to understand than our current layer (just trying to > determine exactly where queries get executed is a challenge) and I'm > concerned it will both meet substantial community resistance and (more > importantly) substantially reduce the number of people who are able to > contribute to it. Of course, this is equally true of the Fields in > Core project I'm working on, so maybe it is just unavoidable at this > point in Drupal's development. I think it is. The old layer was basically just a dummy wrapper around mysql_query() and pgsql_query(). Anything more featureful than that is going to be an increase in complexity. Given what we're asking it to do, I think it's actually reasonably straightforward, at least to someone who is OOP-knowledgeable. > Perhaps the existence of this mailing list will motivate me to re- > visit the pgsql driver, try to get it working again, and in the > process figure out what I really think regarding at least 3a and 3b. > Oh, and look into 2 as well. :-) Yes, please, and soon. :-) Even with the delayed code freeze there's a crapload of work that has to be done after the initial patch lands, and right now Postgres support is the reason it hasn't landed yet. -- Larry Garfield larry at garfieldtech.com From jd at commandprompt.com Tue Jul 8 02:57:24 2008 From: jd at commandprompt.com (Joshua D. Drake) Date: Mon, 07 Jul 2008 19:57:24 -0700 Subject: [database] [Database] Toward better databases In-Reply-To: <200807072042.04709.larry@garfieldtech.com> References: <200807061432.07002.larry@garfieldtech.com> <200807072042.04709.larry@garfieldtech.com> Message-ID: <1215485844.19052.75.camel@jd-laptop> On Mon, 2008-07-07 at 20:42 -0500, Larry Garfield wrote: > On Monday 07 July 2008 8:44:40 am you wrote: > > Howdy, everyone. Here's my current brain-dump: > CREATE TABLE variable ( > name varchar(128) NOT NULL default '', > value text NOT NULL, > PRIMARY KEY (name) > ) > I would also note that the above is horribly bad. You just said that your primary key is name but that its default is '', that protects against NOT NULL but allows a single '' as well as a single ' ' or ' ' or ' ' etc... Sincerely, Joshua D. Drake From david at fourkitchens.com Tue Jul 8 03:02:52 2008 From: david at fourkitchens.com (David Timothy Strauss) Date: Mon, 7 Jul 2008 22:02:52 -0500 (CDT) Subject: [database] [Database] Toward better databases In-Reply-To: <1215485844.19052.75.camel@jd-laptop> Message-ID: <1783319887.434501215486172332.JavaMail.root@mail-2.01.com> MySQL requires non-null defaults for non-null columns. What's the problem? ----- "Joshua D. Drake" wrote: > On Mon, 2008-07-07 at 20:42 -0500, Larry Garfield wrote: > > On Monday 07 July 2008 8:44:40 am you wrote: > > > Howdy, everyone. Here's my current brain-dump: > > > CREATE TABLE variable ( > > name varchar(128) NOT NULL default '', > > value text NOT NULL, > > PRIMARY KEY (name) > > ) > > > > I would also note that the above is horribly bad. > > You just said that your primary key is name but that its default is > '', > that protects against NOT NULL but allows a single '' as well as a > single ' ' or ' ' or ' ' etc... > > Sincerely, > > Joshua D. Drake > > > _______________________________________________ > Database mailing list > Database at drupal.org > http://mail.drupal.org/mailman/listinfo/database From jd at commandprompt.com Tue Jul 8 02:52:23 2008 From: jd at commandprompt.com (Joshua D. Drake) Date: Mon, 07 Jul 2008 19:52:23 -0700 Subject: [database] [Database] Toward better databases In-Reply-To: <200807061432.07002.larry@garfieldtech.com> References: <200807061432.07002.larry@garfieldtech.com> Message-ID: <1215485543.19052.67.camel@jd-laptop> On Sun, 2008-07-06 at 14:32 -0500, Larry Garfield wrote: > Hi all. Curiously Dries hasn't joined the list yet, but everyone else has so > I'll make a welcome post. :-) > Larry Garfield - The poor sod who thought rewriting the database layer would > only take a month or two. :-) > Doesn't that disqualify him? :P > So, that's where we stand right now. Postgres support is blocking pretty much > everything. So, Barry, Josh, can either of you lend a hand while I keep > trying to get a debugger to play nice? > > [1] http://drupal.org/node/225450 > Possibly, let's take a look. Joshua D. Drake From jd at commandprompt.com Tue Jul 8 02:53:32 2008 From: jd at commandprompt.com (Joshua D. Drake) Date: Mon, 07 Jul 2008 19:53:32 -0700 Subject: [database] [Database] Toward better databases In-Reply-To: <200807072042.04709.larry@garfieldtech.com> References: <200807061432.07002.larry@garfieldtech.com> <200807072042.04709.larry@garfieldtech.com> Message-ID: <1215485612.19052.70.camel@jd-laptop> On Mon, 2008-07-07 at 20:42 -0500, Larry Garfield wrote: > On Monday 07 July 2008 8:44:40 am you wrote: > > Howdy, everyone. Here's my current brain-dump: > > > > 1. Larry is correct that the pgsql driver is not currently working. > > I haven't had a chance to bring it up to date with the latest API/ > > internal changes Larry made in the PDO patch. However, there is not > > actually any difficulty related to pgsql specifically; interacting > > with pgsql through the PDO_PGSQL interface works fine and I've had > > everything (including the blob support) working fine before. It's > > just a question of updating it for the latest version of the patch, > > which I haven't done yet. You know, the day job keeps me pretty > > busy. :-) > > I know the feeling. :-) > > However, the problem I'm running into I have managed to track down (10 minutes > with a working debugger is worth weeks without one, dear god!), and it is > highly confusing. The problem comes when creating the variable table. All > of the test queries run fine, but when it gets to this query: > > CREATE TABLE variable ( > name varchar(128) NOT NULL default '', > value text NOT NULL, > PRIMARY KEY (name) > ) > > PHP dies completely. I have not figured out why. That query runs fine within > the psql command line, and it runs fine in a sample script I've created that, > I think, is mimicing the way the new Drupal driver works. I am highly > confused by this, and it is making me sad. :-( > > Josh, do you have any idea why the above query could be failing? Well what does the database say? log_min_messages = debug2 log_min_error_statement = error reload, produce error, produce log, post :) Joshua D. Drake From larry at garfieldtech.com Tue Jul 8 03:28:20 2008 From: larry at garfieldtech.com (Larry Garfield) Date: Mon, 7 Jul 2008 22:28:20 -0500 Subject: [database] [Database] Toward better databases In-Reply-To: <1215485612.19052.70.camel@jd-laptop> References: <200807061432.07002.larry@garfieldtech.com> <200807072042.04709.larry@garfieldtech.com> <1215485612.19052.70.camel@jd-laptop> Message-ID: <200807072228.20389.larry@garfieldtech.com> On Monday 07 July 2008 9:53:32 pm Joshua D. Drake wrote: > > Josh, do you have any idea why the above query could be failing? > > Well what does the database say? > > log_min_messages = debug2 > log_min_error_statement = error > > reload, produce error, produce log, post :) > > Joshua D. Drake The Postgres log says: 2008-07-07 22:19:36 CDT LOG: could not receive data from client: Connection reset by peer 2008-07-07 22:19:36 CDT LOG: unexpected EOF on client connection 2008-07-07 22:19:36 CDT DEBUG: server process (PID 6543) exited with exit code 0 Apache/PHP's error log says: [Mon Jul 07 22:19:37 2008] [notice] child pid 6187 exit signal Segmentation fault (11) So something is highly screwed up somewhere. Can someone try this on another computer? (I don't have another available to me at the moment. My desktop is dead. ) I am wondering if it's an issue with the specific PHP I have installed. -- Larry Garfield larry at garfieldtech.com From larry at garfieldtech.com Tue Jul 8 04:19:12 2008 From: larry at garfieldtech.com (Larry Garfield) Date: Mon, 7 Jul 2008 23:19:12 -0500 Subject: [database] [Database] Toward better databases In-Reply-To: <200807072228.20389.larry@garfieldtech.com> References: <200807061432.07002.larry@garfieldtech.com> <1215485612.19052.70.camel@jd-laptop> <200807072228.20389.larry@garfieldtech.com> Message-ID: <200807072319.12062.larry@garfieldtech.com> On Monday 07 July 2008 10:28:20 pm Larry Garfield wrote: > On Monday 07 July 2008 9:53:32 pm Joshua D. Drake wrote: > > > Josh, do you have any idea why the above query could be failing? > > > > Well what does the database say? > > > > log_min_messages = debug2 > > log_min_error_statement = error > > > > reload, produce error, produce log, post :) > > > > Joshua D. Drake > > The Postgres log says: > > 2008-07-07 22:19:36 CDT LOG: could not receive data from client: > Connection reset by peer > 2008-07-07 22:19:36 CDT LOG: unexpected EOF on client connection > 2008-07-07 22:19:36 CDT DEBUG: server process (PID 6543) exited with exit > code 0 > > Apache/PHP's error log says: > > [Mon Jul 07 22:19:37 2008] [notice] child pid 6187 exit signal Segmentation > fault (11) > > So something is highly screwed up somewhere. Can someone try this on > another computer? (I don't have another available to me at the moment. My > desktop is dead. ) I am wondering if it's an issue with the specific > PHP I have installed. So chx (Josh: Karoly Negyesi, another of Drupal's top developers) just gave the code a whirl with Postgres 8.1 and PHP 5.2.x (I think 5.2.3) and got a white screen of death in the same place, so it sounds like it's not just me. Bah! -- Larry Garfield larry at garfieldtech.com From jd at commandprompt.com Tue Jul 8 04:24:18 2008 From: jd at commandprompt.com (Joshua D. Drake) Date: Mon, 07 Jul 2008 21:24:18 -0700 Subject: [database] [Database] Toward better databases In-Reply-To: <200807072319.12062.larry@garfieldtech.com> References: <200807061432.07002.larry@garfieldtech.com> <1215485612.19052.70.camel@jd-laptop> <200807072228.20389.larry@garfieldtech.com> <200807072319.12062.larry@garfieldtech.com> Message-ID: <1215491058.19052.77.camel@jd-laptop> On Mon, 2008-07-07 at 23:19 -0500, Larry Garfield wrote: > On Monday 07 July 2008 10:28:20 pm Larry Garfield wrote: > > On Monday 07 July 2008 9:53:32 pm Joshua D. Drake wrote: > > > > Josh, do you have any idea why the above query could be failing? > So chx (Josh: Karoly Negyesi, another of Drupal's top developers) just gave > the code a whirl with Postgres 8.1 and PHP 5.2.x (I think 5.2.3) and got a > white screen of death in the same place, so it sounds like it's not just me. > Bah! If apache is segfaulting its not postgresql :(. You can actually see it via the 2008-07-07 22:19:36 CDT LOG: unexpected EOF on client connection, that says that something disconnected from PostgreSQL unexpectedly. Looks like we have a PHP bug. Joshua D. Drake > From jd at commandprompt.com Tue Jul 8 04:25:04 2008 From: jd at commandprompt.com (Joshua D. Drake) Date: Mon, 07 Jul 2008 21:25:04 -0700 Subject: [database] [Database] Toward better databases In-Reply-To: <1783319887.434501215486172332.JavaMail.root@mail-2.01.com> References: <1783319887.434501215486172332.JavaMail.root@mail-2.01.com> Message-ID: <1215491104.19052.79.camel@jd-laptop> On Mon, 2008-07-07 at 22:02 -0500, David Timothy Strauss wrote: > MySQL requires non-null defaults for non-null columns. What's the problem? The problem is below: '' ' ' ' ' ' ' Now, which one is right and how do you select it? Joshua D. Drake From david at fourkitchens.com Tue Jul 8 04:36:54 2008 From: david at fourkitchens.com (David Timothy Strauss) Date: Mon, 7 Jul 2008 23:36:54 -0500 (CDT) Subject: [database] [Database] Toward better databases In-Reply-To: <1215491104.19052.79.camel@jd-laptop> Message-ID: <545905817.449411215491814785.JavaMail.root@mail-2.01.com> Unless we're trimming the values, '' != ' ' != ' ' != ' '. I don't see the problem. ----- "Joshua D. Drake" wrote: > On Mon, 2008-07-07 at 22:02 -0500, David Timothy Strauss wrote: > > MySQL requires non-null defaults for non-null columns. What's the > problem? > > The problem is below: > > '' > ' ' > ' ' > ' ' > > Now, which one is right and how do you select it? > > Joshua D. Drake From jd at commandprompt.com Tue Jul 8 15:10:49 2008 From: jd at commandprompt.com (Joshua D. Drake) Date: Tue, 08 Jul 2008 08:10:49 -0700 Subject: [database] [Database] Toward better databases In-Reply-To: <545905817.449411215491814785.JavaMail.root@mail-2.01.com> References: <545905817.449411215491814785.JavaMail.root@mail-2.01.com> Message-ID: <1215529849.29901.4.camel@jd-laptop> On Mon, 2008-07-07 at 23:36 -0500, David Timothy Strauss wrote: > Unless we're trimming the values, '' != ' ' != ' ' != ' '. I don't see the problem. Obviously. It is generally not a good idea to have a blank, padded blank etc.. as a potential for a primary key. It is probably a good idea to add a check constraint to ensure that there is actually meaningful and contextual data in the column. Please do not top post. Joshua D. Drake From barry.jaspan at acquia.com Tue Jul 8 15:12:19 2008 From: barry.jaspan at acquia.com (Barry Jaspan) Date: Tue, 8 Jul 2008 11:12:19 -0400 Subject: [database] [Database] Toward better databases In-Reply-To: <200807072228.20389.larry@garfieldtech.com> References: <200807061432.07002.larry@garfieldtech.com> <200807072042.04709.larry@garfieldtech.com> <1215485612.19052.70.camel@jd-laptop> <200807072228.20389.larry@garfieldtech.com> Message-ID: <7914F0EE-EDCA-4D9D-BD62-8A548519BBF7@acquia.com> This sounds like a problem with the way bindParam() is being used on blob params. I noticed during initial development that if you use bindParam() incorrectly, it causes Apache to crash (presumably because bindParam() is dealing directly with internal pointers and is somehow trashing memory). The specific issue I recall is that bindParam() accepts a reference to its argument (i.e.: a pointer) and if the referenced PHP variable goes out of scope before the query is executed, bang! The problems were all fixed at one point but the API has changed and it is entirely possible they have crept back. I never really even finished the code so I would not be at all surprised at this. The fact that a seemingly unrelated query is now causing a crash is unsurprising. Memory corruption bugs are like that. So, I'd say you should ignore this until I or someone else makes a full pass at updating the pgsql driver. Thanks, Barry ------- Barry Jaspan Code Jester, Acquia Inc. barry.jaspan at acquia.com On Jul 7, 2008, at 11:28 PM, Larry Garfield wrote: > On Monday 07 July 2008 9:53:32 pm Joshua D. Drake wrote: > >>> Josh, do you have any idea why the above query could be failing? >> >> Well what does the database say? >> >> log_min_messages = debug2 >> log_min_error_statement = error >> >> reload, produce error, produce log, post :) >> >> Joshua D. Drake > > The Postgres log says: > > 2008-07-07 22:19:36 CDT LOG: could not receive data from client: > Connection > reset by peer > 2008-07-07 22:19:36 CDT LOG: unexpected EOF on client connection > 2008-07-07 22:19:36 CDT DEBUG: server process (PID 6543) exited > with exit > code 0 > > Apache/PHP's error log says: > > [Mon Jul 07 22:19:37 2008] [notice] child pid 6187 exit signal > Segmentation > fault (11) > > So something is highly screwed up somewhere. Can someone try this > on another > computer? (I don't have another available to me at the moment. My > desktop > is dead. ) I am wondering if it's an issue with the specific PHP > I have > installed. > > -- > Larry Garfield > larry at garfieldtech.com > _______________________________________________ > Database mailing list > Database at drupal.org > http://mail.drupal.org/mailman/listinfo/database From larry at garfieldtech.com Tue Jul 8 15:20:08 2008 From: larry at garfieldtech.com (Larry Garfield) Date: Tue, 8 Jul 2008 10:20:08 -0500 Subject: [database] [Database] Toward better databases In-Reply-To: <7914F0EE-EDCA-4D9D-BD62-8A548519BBF7@acquia.com> References: <7914F0EE-EDCA-4D9D-BD62-8A548519BBF7@acquia.com> Message-ID: On Tue, 8 Jul 2008 11:12:19 -0400, Barry Jaspan wrote: > This sounds like a problem with the way bindParam() is being used on > blob params. I noticed during initial development that if you use > bindParam() incorrectly, it causes Apache to crash (presumably because > bindParam() is dealing directly with internal pointers and is somehow > trashing memory). > > The specific issue I recall is that bindParam() accepts a reference to > its argument (i.e.: a pointer) and if the referenced PHP variable goes > out of scope before the query is executed, bang! The problems were > all fixed at one point but the API has changed and it is entirely > possible they have crept back. I never really even finished the code > so I would not be at all surprised at this. > > The fact that a seemingly unrelated query is now causing a crash is > unsurprising. Memory corruption bugs are like that. > > So, I'd say you should ignore this until I or someone else makes a > full pass at updating the pgsql driver. > > Thanks, > > Barry Well, it is hard to take a full pass at updating the driver until we're able to actually install it. :-) And as I said before, time is short so I'm trying to see what I can do myself in the meantime. Also, a follow-up from last night. chx and I are running the exact same version of PHP, 5.2.4 with the suoshin add-on and XDebug, as installed by Kubuntu 8.04 Hardy Herron. I'm on Postgres 8.3 and he's on 8.1, and we get the exact same symptoms. Moonshine in IRC tried on his PHP 5.2.6 system and got an entirely different fatal error, as somewhere in the process it was trying to load the block module and calling the system table to do so, before any tables were created. We were confused by this. He said he'd keep looking into it, but I'm still trying to work out why we're segfaulting. A bindParam() bug sounds possible, but all of the queries up to that point (which DO run) have no arguments. Perhaps that's the actual problem? Calling execute() with an empty array? --Larry Garfield From barry.jaspan at acquia.com Tue Jul 8 15:26:58 2008 From: barry.jaspan at acquia.com (Barry Jaspan) Date: Tue, 8 Jul 2008 11:26:58 -0400 Subject: [database] [Database] Toward better databases In-Reply-To: <1783319887.434501215486172332.JavaMail.root@mail-2.01.com> References: <1783319887.434501215486172332.JavaMail.root@mail-2.01.com> Message-ID: <92F42BCD-DCA8-4416-BEEA-36F12D054261@acquia.com> David, On Jul 7, 2008, at 11:02 PM, David Timothy Strauss wrote: > MySQL requires non-null defaults for non-null columns. What's the > problem? Ummmm.. really? My understanding is that MySQL provides implicit default values for NOT NULL columns that are not given a default value when not in strict mode, but that isn't the same as requiring that a non-null default be provided. During the D6 switch to Schema API, I tried to remove all the unnecessary/incorrect default values on columns. For the variable table, a default name of '' is semantically ridiculous; we should never have a variable whose name is the empty string and there should never be an INSERT statement into that table that does not provide a value for name. So why have the default value at all? Barry > > > ----- "Joshua D. Drake" wrote: > >> On Mon, 2008-07-07 at 20:42 -0500, Larry Garfield wrote: >>> On Monday 07 July 2008 8:44:40 am you wrote: >>>> Howdy, everyone. Here's my current brain-dump: >> >>> CREATE TABLE variable ( >>> name varchar(128) NOT NULL default '', >>> value text NOT NULL, >>> PRIMARY KEY (name) >>> ) >>> >> >> I would also note that the above is horribly bad. >> >> You just said that your primary key is name but that its default is >> '', >> that protects against NOT NULL but allows a single '' as well as a >> single ' ' or ' ' or ' ' etc... >> >> Sincerely, >> >> Joshua D. Drake >> >> >> _______________________________________________ >> Database mailing list >> Database at drupal.org >> http://mail.drupal.org/mailman/listinfo/database > _______________________________________________ > Database mailing list > Database at drupal.org > http://mail.drupal.org/mailman/listinfo/database From barry.jaspan at acquia.com Tue Jul 8 15:29:38 2008 From: barry.jaspan at acquia.com (Barry Jaspan) Date: Tue, 8 Jul 2008 11:29:38 -0400 Subject: [database] [Database] Toward better databases In-Reply-To: References: <7914F0EE-EDCA-4D9D-BD62-8A548519BBF7@acquia.com> Message-ID: On Jul 8, 2008, at 11:20 AM, Larry Garfield wrote: >> Well, it is hard to take a full pass at updating the driver until >> we're able to actually install it. :-) And as I said before, time >> is short so I'm trying to see what I can do myself in the meantime. Try taking out the code that uses fopen etc to pass blobs properly to bindParam(); just treat blobs like everything else. This will make blob queries fail but should avoid the segfault. Note that without blob queries working, Drupal will not install properly (the cache won't work, for example), but at least we will have identified the problem. Barry From barry.jaspan at acquia.com Tue Jul 8 15:31:42 2008 From: barry.jaspan at acquia.com (Barry Jaspan) Date: Tue, 8 Jul 2008 11:31:42 -0400 Subject: [database] [Database] Toward better databases In-Reply-To: <200807072042.04709.larry@garfieldtech.com> References: <200807061432.07002.larry@garfieldtech.com> <200807072042.04709.larry@garfieldtech.com> Message-ID: > Even with the delayed code freeze there's a > crapload of work that has to be done after the initial patch lands, > and right > now Postgres support is the reason it hasn't landed yet. My point is that I'm not sure I'd support this patch being applied as- is even if the pgsql was working. So it is not fully accurate to say the pgsql support is the hold-up, though it certainly doesn't help. Barry From david at fourkitchens.com Tue Jul 8 15:57:54 2008 From: david at fourkitchens.com (David Strauss) Date: Tue, 8 Jul 2008 15:57:54 +0000 Subject: [database] [Database] Toward better databases In-Reply-To: <92F42BCD-DCA8-4416-BEEA-36F12D054261@acquia.com> References: <1783319887.434501215486172332.JavaMail.root@mail-2.01.com><92F42BCD-DCA8-4416-BEEA-36F12D054261@acquia.com> Message-ID: <675664007-1215532681-cardhu_decombobulator_blackberry.rim.net-2126785096-@bxe027.bisx.prod.on.blackberry> Oh, I see. I was including implicit defaults. However, if you have a non-null varchar, the implicit default is "". There's little to be gained from not specifying defaults and letting MySQL set the defaults unless we use the defaults data elsewhere in the abstraction system. I do agree that removing absurd defaults is a semantic improvement. -----Original Message----- From: Barry Jaspan Date: Tue, 8 Jul 2008 11:26:58 To: David Timothy Strauss Cc: Joshua D. Drake; Subject: Re: [database] [Database] Toward better databases David, On Jul 7, 2008, at 11:02 PM, David Timothy Strauss wrote: > MySQL requires non-null defaults for non-null columns. What's the > problem? Ummmm.. really? My understanding is that MySQL provides implicit default values for NOT NULL columns that are not given a default value when not in strict mode, but that isn't the same as requiring that a non-null default be provided. During the D6 switch to Schema API, I tried to remove all the unnecessary/incorrect default values on columns. For the variable table, a default name of '' is semantically ridiculous; we should never have a variable whose name is the empty string and there should never be an INSERT statement into that table that does not provide a value for name. So why have the default value at all? Barry > > > ----- "Joshua D. Drake" wrote: > >> On Mon, 2008-07-07 at 20:42 -0500, Larry Garfield wrote: >>> On Monday 07 July 2008 8:44:40 am you wrote: >>>> Howdy, everyone. Here's my current brain-dump: >> >>> CREATE TABLE variable ( >>> name varchar(128) NOT NULL default '', >>> value text NOT NULL, >>> PRIMARY KEY (name) >>> ) >>> >> >> I would also note that the above is horribly bad. >> >> You just said that your primary key is name but that its default is >> '', >> that protects against NOT NULL but allows a single '' as well as a >> single ' ' or ' ' or ' ' etc... >> >> Sincerely, >> >> Joshua D. Drake >> >> >> _______________________________________________ >> Database mailing list >> Database at drupal.org >> http://mail.drupal.org/mailman/listinfo/database > _______________________________________________ > Database mailing list > Database at drupal.org > http://mail.drupal.org/mailman/listinfo/database From jd at commandprompt.com Tue Jul 8 16:44:57 2008 From: jd at commandprompt.com (Joshua D. Drake) Date: Tue, 08 Jul 2008 09:44:57 -0700 Subject: [database] [Database] Toward better databases In-Reply-To: <675664007-1215532681-cardhu_decombobulator_blackberry.rim.net-2126785096-@bxe027.bisx.prod.on.blackberry> References: <1783319887.434501215486172332.JavaMail.root@mail-2.01.com> <92F42BCD-DCA8-4416-BEEA-36F12D054261@acquia.com> <675664007-1215532681-cardhu_decombobulator_blackberry.rim.net-2126785096-@bxe027.bisx.prod.on.blackberry> Message-ID: <1215535497.29901.18.camel@jd-laptop> On Tue, 2008-07-08 at 15:57 +0000, David Strauss wrote: > Oh, I see. I was including implicit defaults. > > However, if you have a non-null varchar, the implicit default is "". There's little to be gained from not specifying defaults and letting MySQL set the defaults unless we use the defaults data elsewhere in the abstraction system. > > I do agree that removing absurd defaults is a semantic improvement. There is nothing semantic about good design. Please do not top post. Sincerely, Joshua D. Drake From david at fourkitchens.com Tue Jul 8 16:50:45 2008 From: david at fourkitchens.com (David Strauss) Date: Tue, 8 Jul 2008 16:50:45 +0000 Subject: [database] [Database] Toward better databases Message-ID: <1188331524-1215535852-cardhu_decombobulator_blackberry.rim.net-220960661-@bxe027.bisx.prod.on.blackberry> BlackBerries can only top-post. ------Original Message------ From: Joshua D. Drake To: David Strauss Cc: Barry Jaspan Cc: database at drupal.org Subject: Re: [database] [Database] Toward better databases Sent: Jul 8, 2008 09:44 On Tue, 2008-07-08 at 15:57 +0000, David Strauss wrote: > Oh, I see. I was including implicit defaults. > > However, if you have a non-null varchar, the implicit default is "". There's little to be gained from not specifying defaults and letting MySQL set the defaults unless we use the defaults data elsewhere in the abstraction system. > > I do agree that removing absurd defaults is a semantic improvement. There is nothing semantic about good design. Please do not top post. Sincerely, Joshua D. Drake From jd at commandprompt.com Tue Jul 8 18:24:14 2008 From: jd at commandprompt.com (Joshua D. Drake) Date: Tue, 08 Jul 2008 11:24:14 -0700 Subject: [database] [Database] Toward better databases In-Reply-To: References: <200807061432.07002.larry@garfieldtech.com> Message-ID: <1215541454.29901.41.camel@jd-laptop> On Mon, 2008-07-07 at 09:44 -0400, Barry Jaspan wrote: > Howdy, everyone. Here's my current brain-dump: > One snafu, however: db_inspect_column() may require database > privileges that other Drupal operations do not. We'll have to look > into that and whether it is an acceptable requirement (the answer will > probably be "yes, because there is no alternative"). > With PostgreSQL you will not need extra perms. Any user that is allowed to connect to the database can execute SELECT from information_schema.* . > 3. Larry isn't going to like this, but part of the reason I haven't > mustered the time to finish the pgsql driver is that I am not 100% > convinced the design for the PDO-based database layer is right yet, > and figuring out a proposal to fix it requires more time than I've had. > I am not against PDO but is there a reason we didn't just include a light and generic wrapper on top of the old fashion functions? Isn't that what PDO is? (note: not much of a php guy here). > 3a. One issue is that the way that SQL and placeholders are generated > seems fragile and inefficient to me. It ends up being fairly awkward > for the pgsql driver to keep everything straight in the way it needs > to in order to do its most basic function of calling bindParam() > correctly (though it is much better than in early drafts of the patch). > Remind me, are we doing client side or server side prepare? > 3c. Also, the new database layer is substantially more complex and > difficult to understand than our current layer (just trying to > determine exactly where queries get executed is a challenge) and I'm > concerned it will both meet substantial community resistance and (more > importantly) substantially reduce the number of people who are able to > contribute to it. Some caveats before I say what I am about to say: Lots of community is good Difficult APIs are bad Having an API that is sufficiently well designed that limits the number of community that could actually develop for it? Good. Weeding out those who would then subject other community members to potential bad code and design is not a bad thing. > Of course, this is equally true of the Fields in > Core project I'm working on, so maybe it is just unavoidable at this > point in Drupal's development. > Likely. Drupal is growing up, that means things get a lot more complicated. Its just the nature of the beast. When I look at PostgreSQL 7.3 versus 8.3 it is amazing how much more complicated the product has become. It is the nature of adding features, the more features you add, the more concepts you have to grok. That makes things complicated. Sincerely, Joshua D. Drake From larry at garfieldtech.com Tue Jul 8 21:00:31 2008 From: larry at garfieldtech.com (Larry Garfield) Date: Tue, 8 Jul 2008 16:00:31 -0500 Subject: [database] [Database] Toward better databases In-Reply-To: <1215541454.29901.41.camel@jd-laptop> References: <1215541454.29901.41.camel@jd-laptop> Message-ID: <89f35e2e2d784c96f886b35a106de3cc@localhost> On Tue, 08 Jul 2008 11:24:14 -0700, "Joshua D. Drake" wrote: >> 3. Larry isn't going to like this, but part of the reason I haven't >> mustered the time to finish the pgsql driver is that I am not 100% >> convinced the design for the PDO-based database layer is right yet, >> and figuring out a proposal to fix it requires more time than I've had. >> > > I am not against PDO but is there a reason we didn't just include a > light and generic wrapper on top of the old fashion functions? Isn't > that what PDO is? (note: not much of a php guy here). The generic wrapper around the old fashioned functions is what we have in Drupal 6 and earlier, and it is very very limiting. PDO is not just a wrapper around those. It is a single userspace interface with swappable C-level drivers for each supported backend, so that the userspace code is the same either way. (At least that's the goal; Sadly it doesn't abstract out things like funky blob handling, which is where the problem comes in.) >> 3a. One issue is that the way that SQL and placeholders are generated >> seems fragile and inefficient to me. It ends up being fairly awkward >> for the pgsql driver to keep everything straight in the way it needs >> to in order to do its most basic function of calling bindParam() >> correctly (though it is much better than in early drafts of the patch). >> > > Remind me, are we doing client side or server side prepare? We're relying on PDO for the prepare. In MySQL, emulating them in PDO rather than letting MySQL do them is faster, as MySQL's prepared statements bypass the query cache for some bizarre reason. In Postgres, I think we're letting Postgres do it. --Larry Garfield From larry at garfieldtech.com Fri Jul 11 15:07:14 2008 From: larry at garfieldtech.com (Larry Garfield) Date: Fri, 11 Jul 2008 10:07:14 -0500 Subject: [database] Postgres version Message-ID: <1604fde9d88afb69f3751f0ab2132dc4@localhost> Josh, perhaps you can weigh in here, since you likely know how much older versions of postgres are used better than I do: http://drupal.org/node/24820 --Larry Garfield From jd at commandprompt.com Fri Jul 11 15:19:52 2008 From: jd at commandprompt.com (Joshua D. Drake) Date: Fri, 11 Jul 2008 08:19:52 -0700 Subject: [database] Postgres version In-Reply-To: <1604fde9d88afb69f3751f0ab2132dc4@localhost> References: <1604fde9d88afb69f3751f0ab2132dc4@localhost> Message-ID: <1215789592.15349.5.camel@jd-laptop> On Fri, 2008-07-11 at 10:07 -0500, Larry Garfield wrote: > Josh, perhaps you can weigh in here, since you likely know how much older versions of postgres are used better than I do: http://drupal.org/node/24820 That link brings up a unicode problem with mysql? Joshua D. Drake > > --Larry Garfield > > _______________________________________________ > Database mailing list > Database at drupal.org > http://mail.drupal.org/mailman/listinfo/database > From larry at garfieldtech.com Fri Jul 11 21:57:31 2008 From: larry at garfieldtech.com (Larry Garfield) Date: Fri, 11 Jul 2008 16:57:31 -0500 Subject: [database] Postgres version In-Reply-To: <1215789592.15349.5.camel@jd-laptop> References: <1215789592.15349.5.camel@jd-laptop> Message-ID: <458113a202652a3997dff19c54f428e7@localhost> On Fri, 11 Jul 2008 08:19:52 -0700, "Joshua D. Drake" wrote: > > > On Fri, 2008-07-11 at 10:07 -0500, Larry Garfield wrote: >> Josh, perhaps you can weigh in here, since you likely know how much > older versions of postgres are used better than I do: > http://drupal.org/node/24820 > > That link brings up a unicode problem with mysql? > > Joshua D. Drake Stupid copy and paste... http://drupal.org/node/248205 --Larry Garfield From jd at commandprompt.com Fri Jul 11 23:04:38 2008 From: jd at commandprompt.com (Joshua D. Drake) Date: Fri, 11 Jul 2008 16:04:38 -0700 Subject: [database] Postgres version In-Reply-To: <458113a202652a3997dff19c54f428e7@localhost> References: <1215789592.15349.5.camel@jd-laptop> <458113a202652a3997dff19c54f428e7@localhost> Message-ID: <1215817478.15349.52.camel@jd-laptop> On Fri, 2008-07-11 at 16:57 -0500, Larry Garfield wrote: > On Fri, 11 Jul 2008 08:19:52 -0700, "Joshua D. Drake" > wrote: > > Joshua D. Drake > > Stupid copy and paste... > > http://drupal.org/node/248205 > > --Larry Garfield Done... I personally think that 8.1 is the way to go. Joshua D. Drake -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ United States PostgreSQL Association: http://www.postgresql.us/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate From larry at garfieldtech.com Sat Jul 19 07:07:43 2008 From: larry at garfieldtech.com (Larry Garfield) Date: Sat, 19 Jul 2008 02:07:43 -0500 Subject: [database] Merge queries Message-ID: <200807190207.43706.larry@garfieldtech.com> So much progress has been made, now that I have a working Postgres install thanks to James Walker and Sammy Spets. :-) The next item on the list is Merge queries. http://en.wikibooks.org/wiki/SQL_dialects_reference/Write_queries/Replace_query Basically nothing supports the SQL:2003 merge query AFAIK. In MySQL, we're using ON DUPLICATE KEY UPDATE, which is working like a champ. For Postgres, the original concept was to do: // start transaction. try { db_insert(); } catch (Exception $e) { db_update(); } // end transaction Since the return value from db_update() is the number of rows *affected*, not *changed*, which means our usual update-or-insert routine doesn't work. However, Sammy pointed out that logic will not work, period. If the INSERT statement throws an exception, it's because the database itself is now in an invalid state and will not accept anything other than a rollback. Even if we didn't have the transaction wrapping there, we couldn't have any transactions active at all for a merge query with this logic. So this is not an option. So we need a way to handle merge queries in Postgres. Josh, this is where you come in. :-) How can we do an atomic, non-exception-using merge query or reasonable emulation? We've just kicked Postgres 7.4 to the curb, so whatever Postgres 8.1 can do is available to us. -- Larry Garfield larry at garfieldtech.com From david at fourkitchens.com Sat Jul 19 07:21:30 2008 From: david at fourkitchens.com (David Timothy Strauss) Date: Sat, 19 Jul 2008 02:21:30 -0500 (CDT) Subject: [database] Merge queries In-Reply-To: <200807190207.43706.larry@garfieldtech.com> Message-ID: <2010684346.2806711216452090900.JavaMail.root@mail-2.01.com> "Since the return value from db_update() is the number of rows *affected*, not *changed*, which means our usual update-or-insert routine doesn't work." I assume you mean it returns the number of rows *changed* rather than *matching* rows. The way I solve the issue is by doing the operations: (1) UPDATE (2) If updated changed zero rows, do a SELECT on the criteria (3) If the SELECT returned zero rows, do an INSERT This would all be in a transaction. ----- "Larry Garfield" wrote: > So much progress has been made, now that I have a working Postgres > install > thanks to James Walker and Sammy Spets. :-) > > The next item on the list is Merge queries. > > http://en.wikibooks.org/wiki/SQL_dialects_reference/Write_queries/Replace_query > > Basically nothing supports the SQL:2003 merge query AFAIK. > > In MySQL, we're using ON DUPLICATE KEY UPDATE, which is working like a > champ. > > For Postgres, the original concept was to do: > > // start transaction. > try { > db_insert(); > } > catch (Exception $e) { > db_update(); > } > // end transaction > > Since the return value from db_update() is the number of rows > *affected*, not > *changed*, which means our usual update-or-insert routine doesn't > work. > > However, Sammy pointed out that logic will not work, period. If the > INSERT > statement throws an exception, it's because the database itself is now > in an > invalid state and will not accept anything other than a rollback. > Even if we > didn't have the transaction wrapping there, we couldn't have any > transactions > active at all for a merge query with this logic. So this is not an > option. > > So we need a way to handle merge queries in Postgres. Josh, this is > where you > come in. :-) How can we do an atomic, non-exception-using merge query > or > reasonable emulation? We've just kicked Postgres 7.4 to the curb, so > > whatever Postgres 8.1 can do is available to us. > > -- > Larry Garfield > larry at garfieldtech.com > _______________________________________________ > Database mailing list > Database at drupal.org > http://mail.drupal.org/mailman/listinfo/database From larry at garfieldtech.com Sat Jul 19 20:42:41 2008 From: larry at garfieldtech.com (Larry Garfield) Date: Sat, 19 Jul 2008 15:42:41 -0500 Subject: [database] Merge queries In-Reply-To: <2010684346.2806711216452090900.JavaMail.root@mail-2.01.com> References: <2010684346.2806711216452090900.JavaMail.root@mail-2.01.com> Message-ID: <200807191542.41745.larry@garfieldtech.com> On Saturday 19 July 2008 2:21:30 am David Timothy Strauss wrote: > "Since the return value from db_update() is the number of rows *affected*, > not *changed*, which means our usual update-or-insert routine doesn't > work." > > I assume you mean it returns the number of rows *changed* rather than > *matching* rows. Er. Yes. Yes I do. > The way I solve the issue is by doing the operations: > (1) UPDATE > (2) If updated changed zero rows, do a SELECT on the criteria > (3) If the SELECT returned zero rows, do an INSERT > > This would all be in a transaction. Three queries? Yipes. I guess that could work as a generic version, although I'd prefer to see something more Postgres-optimized in the Postgres driver if at all possible. Remember we're talking about the code that is the bulk of variable_set() and potentially cache_set(), so it will frequently be called multiple times in rapid succession. -- Larry Garfield larry at garfieldtech.com From david at fourkitchens.com Sat Jul 19 20:53:04 2008 From: david at fourkitchens.com (David Timothy Strauss) Date: Sat, 19 Jul 2008 15:53:04 -0500 (CDT) Subject: [database] Merge queries In-Reply-To: <200807191542.41745.larry@garfieldtech.com> Message-ID: <47285207.2838791216500784102.JavaMail.root@mail-2.01.com> ----- "Larry Garfield" wrote: > > The way I solve the issue is by doing the operations: > > (1) UPDATE > > (2) If updated changed zero rows, do a SELECT on the criteria > > (3) If the SELECT returned zero rows, do an INSERT > > > > This would all be in a transaction. > > Three queries? Yipes. I guess that could work as a generic version, > although > I'd prefer to see something more Postgres-optimized in the Postgres > driver if > at all possible. Remember we're talking about the code that is the > bulk of > variable_set() and potentially cache_set(), so it will frequently be > called > multiple times in rapid succession. I'd suggest creating a function in MySQL and PostgreSQL that handles the queries. The three queries in and of themselves don't take much time; the enemy is the latency of three queries. If an in-database function handles the logic, I think we'll be in good shape. BTW, I've implemented the algorithm above as pressflow_transaction_update() in my Pressflow Transaction module. From jd at commandprompt.com Sat Jul 19 21:32:39 2008 From: jd at commandprompt.com (Joshua D. Drake) Date: Sat, 19 Jul 2008 14:32:39 -0700 Subject: [database] Merge queries In-Reply-To: <47285207.2838791216500784102.JavaMail.root@mail-2.01.com> References: <47285207.2838791216500784102.JavaMail.root@mail-2.01.com> Message-ID: <1216503159.3235.5.camel@jd-laptop> On Sat, 2008-07-19 at 15:53 -0500, David Timothy Strauss wrote: > ----- "Larry Garfield" wrote: > I'd suggest creating a function in MySQL and PostgreSQL that handles the queries. The three queries in and of themselves don't take much time; the enemy is the latency of three queries. If an in-database function handles the logic, I think we'll be in good shape. > That would be my suggestion as well. Joshua D. Drake -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ United States PostgreSQL Association: http://www.postgresql.us/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate From larry at garfieldtech.com Sat Jul 19 21:50:57 2008 From: larry at garfieldtech.com (Larry Garfield) Date: Sat, 19 Jul 2008 16:50:57 -0500 Subject: [database] Merge queries In-Reply-To: <1216503159.3235.5.camel@jd-laptop> References: <47285207.2838791216500784102.JavaMail.root@mail-2.01.com> <1216503159.3235.5.camel@jd-laptop> Message-ID: <200807191650.57994.larry@garfieldtech.com> On Saturday 19 July 2008 4:32:39 pm you wrote: > On Sat, 2008-07-19 at 15:53 -0500, David Timothy Strauss wrote: > > ----- "Larry Garfield" wrote: > > > > I'd suggest creating a function in MySQL and PostgreSQL that handles the > > queries. The three queries in and of themselves don't take much time; the > > enemy is the latency of three queries. If an in-database function handles > > the logic, I think we'll be in good shape. > > That would be my suggestion as well. > > Joshua D. Drake Well in MySQL it's already a single query with INSERT ON DUPLICATE KEY UPDATE, so that's fine. An SQL function for Postgres sounds fine to me, but I have no idea how to write one. :-) Josh, is that something you could take point on? Catch me on IRC if you can and I can walk you through the API for db_merge(), or the unit tests should be fairly self-explanatory. -- Larry Garfield larry at garfieldtech.com