What would be more useful for install/etc
As much as everyone loves speculating about potential directory structures and file formats, theres other stuff to do. -Error avoidance. Module install and updates will happily go ahead and try to do things which are access denied and then think they are done. We need to check for access before doing these. One for each combinataion of alter/create and mysql/postgres to get started. -Error handling. Output to AJAX (as used in update.php, among other places) should never return HTML errors to the client; or the JS could be more forgiving. Either we need to work with PHP's error handling (I've just about given up on this due to PHP's lack of sanity) or make JS able to handle the PHP error spew. -- Neil Drumm http://delocalizedham.com/
As much as everyone loves speculating about potential directory structures and file formats, theres other stuff to do.
The intent of the thread, as solidified in the subject line, was a discussion about 4.8 or 5.0's approach to the install system. That doesn't invalidate the two things you mention, and neither does yours invalidate mine. They are not, however, in any way, related: yours is talking about improving the capabilities of existing features in the install code, and my thread was how best to abuse the install system and methodology for future versions, which will probably open up for commits in a month or so. The two threads are entirely unrelated. It's equivalent to me saying "we really shouldn't think, or heck, even talk, about comments as nodes because we still have three bugs over here that need fixing!" While I appreciate your probably unstated goal to focus all efforts toward 4.7, I heartily admit to ignoring it entirely so that I can focus on an area *I* care about, as opposed to one I know nothing about, nor desire to. -- Morbus Iff ( you are nothing without your robot car, NOTHING! ) Culture: http://www.disobey.com/ and http://www.gamegrene.com/ O'Reilly Author, Weblog, Cook: http://www.oreillynet.com/pub/au/779 icq: 2927491 / aim: akaMorbus / yahoo: morbus_iff / jabber.org: morbus
On 22 Feb 2006, at 7:49 PM, Neil Drumm wrote:
-Error avoidance. Module install and updates will happily go ahead and try to do things which are access denied and then think they are done. We need to check for access before doing these. One for each combinataion of alter/create and mysql/postgres to get started.
This is not a 4.7 thing, but I do believe we need to extend the db api to include db_create_table, db_add_index and db_add_column (including the db_drop_* of all these). Doing this will get rid of those insipid if ($GLOBALS['db_type']) conditionals, and provide a good place for checking these permissions, including possible temporary database credential escalation and very very very hopefully merge our two database files (very very hopefully), meaning less code to maintain (yay), and better support for postgres (yay) and possibly a simpler path to supporting more database types. -- Adrian Rossouw Drupal developer and Bryght Guy http://drupal.org | http://bryght.com
Op woensdag 22 februari 2006 18:49, schreef Neil Drumm:
-Error avoidance. Module install and updates will happily go ahead and try to do things which are access denied and then think they are done. We need to check for access before doing these. One for each combinataion of alter/create and mysql/postgres to get started.
- Duplication avoidance: we have no way to avoid folks from rerunning the upgrade if the previous one failed, other thens some words to "not run the upgrade twice" - split of JoeSchmoe stuff and real stuff (sorry :). Upgrading joeschmoe.com is easy. It was done rather well, even in PHPnuke. Upgrading Bryght or web-log.nl[1] is a biatch, if not plain impossble w/o a lot of custom code. I beleive, that unless we provide the proper API stuff (like upgrade) to the People That Matter, Drupal will continue being "a nice alternative" but nothing more. Bèr -back trying to find ways to upgrade all his multisite sites- Kessels [1] http://willy.boerland.com/myblog/ilse_media_didnt_choose_drupal -- [ Bèr Kessels | Drupal services www.webschuur.com ]
Bèr Kessels wrote:
Op woensdag 22 februari 2006 18:49, schreef Neil Drumm:
-Error avoidance. Module install and updates will happily go ahead and try to do things which are access denied and then think they are done. We need to check for access before doing these. One for each combinataion of alter/create and mysql/postgres to get started.
- Duplication avoidance: we have no way to avoid folks from rerunning the upgrade if the previous one failed, other thens some words to "not run the upgrade twice"
After each update the schema version in the system table is updated as well. So if the update selection page is regenerated after a half-update it will be okay. Maybe not perfect, since the browser may cache the previous form values if you use the back button, but certainly improved in 4.7.
- split of JoeSchmoe stuff and real stuff (sorry :). Upgrading joeschmoe.com is easy. It was done rather well, even in PHPnuke. Upgrading Bryght or web-log.nl[1] is a biatch, if not plain impossble w/o a lot of custom code.
It didn't happen in 4.6 so we can go without it for 4.7. Its not really a huge issue until you get into shared tables, where the same table would be updated twice by two sites. -- Neil Drumm http://delocalizedham.com/
- split of JoeSchmoe stuff and real stuff (sorry :). Upgrading joeschmoe.com is easy. It was done rather well, even in PHPnuke. Upgrading Bryght or web-log.nl[1] is a biatch, if not plain impossble w/o a lot of custom code.
It didn't happen in 4.6 so we can go without it for 4.7. Its not really a huge issue until you get into shared tables, where the same table would be updated twice by two sites.
The main reason I do not use this nifty feature, and also recommend against it for others: No way to have a clean upgrade for a table shared across Drupal sites.
On Wed, Feb 22, 2006 at 08:01:19PM -0500, Khalid B wrote:
On Wed, Feb 22, 2006 at 10:40:37AM -0800, Neil Drumm wrote:
It didn't happen in 4.6 so we can go without it for 4.7. Its not really a huge issue until you get into shared tables, where the same table would be updated twice by two sites.
The main reason I do not use this nifty feature, and also recommend against it for others: No way to have a clean upgrade for a table shared across Drupal sites.
So are y'all saying that the upgrade process doesn't work for multisite Drupal installs that share tables? --Dan -- T H E A N A L Y S I S A N D S O L U T I O N S C O M P A N Y data intensive web and database programming http://www.AnalysisAndSolutions.com/ 4015 7th Ave #4, Brooklyn NY 11232 v: 718-854-0335 f: 718-854-0409
On Wed, Feb 22, 2006 at 08:01:19PM -0500, Khalid B wrote:
On Wed, Feb 22, 2006 at 10:40:37AM -0800, Neil Drumm wrote:
It didn't happen in 4.6 so we can go without it for 4.7. Its not really a huge issue until you get into shared tables, where the same table would be updated twice by two sites.
The main reason I do not use this nifty feature, and also recommend against it for others: No way to have a clean upgrade for a table shared across Drupal sites.
So are y'all saying that the upgrade process doesn't work for multisite Drupal installs that share tables?
I did not try it, but I can't think of how it would work. The first site would update the tables, then what is the second site supposed to do? Ideally it should skip the update of already updated, but ... Many people are running it that way. Don't know if anyone tried upgrading so they can give feedback on that.
See http://drupal.org/node/50840 This is just a start - many more to go.
On Wed, Feb 22, 2006 at 09:07:14PM -0500, Khalid B wrote:
So are y'all saying that the upgrade process doesn't work for multisite Drupal installs that share tables?
I did not try it, but I can't think of how it would work. The first site would update the tables, then what is the second site supposed to do? Ideally it should skip the update of already updated, but ...
Ouch. Let's see if I can do something about that... My thought is to run the upgrade once. Have the process obtain all of the prefix names by parsing all the sites/settings.php files up front. I do something along these lines in the Database Table Creation Script: http://drupal.org/files/issues/create.php.v2.txt http://drupal.org/node/20530 --Dan -- T H E A N A L Y S I S A N D S O L U T I O N S C O M P A N Y data intensive web and database programming http://www.AnalysisAndSolutions.com/ 4015 7th Ave #4, Brooklyn NY 11232 v: 718-854-0335 f: 718-854-0409
On Wed, Feb 22, 2006 at 09:21:04PM -0500, Daniel Convissor wrote:
Ouch. Let's see if I can do something about that...
My thought is to run the upgrade once. Have the process obtain all of the prefix names by parsing all the sites/settings.php files up front. I do something along these lines in the Database Table Creation Script:
http://drupal.org/files/issues/create.php.v2.txt http://drupal.org/node/20530
Was hoping for a little feedback from folks here on that idea. Uh, okay, no takers yet, so let me be more direct. :) * Are the core developers interested in having multi-site support in the update process? (I mean, why work on something just to have it go nowhere, right?) * What do you think is a better/more workable/more Drupal-like approach: a) put an option in the process that the user can select (in update_selection_page(), I guess) that tells the system to update all tables in the database, not just those for the present site. b) an option for the user to say whether to update the shared tables or not. This way, the user will update each site in each site's interface, but the user will only select the shared table run one time. c) other ideas? Thanks, --Dan -- T H E A N A L Y S I S A N D S O L U T I O N S C O M P A N Y data intensive web and database programming http://www.AnalysisAndSolutions.com/ 4015 7th Ave #4, Brooklyn NY 11232 v: 718-854-0335 f: 718-854-0409
* Are the core developers interested in having multi-site support in the update process? (I mean, why work on something just to have it go nowhere, right?)
yes . multi-site is very much a supported feature. so is the update system. their combination needs to be supproted as well.
* What do you think is a better/more workable/more Drupal-like approach:
a) put an option in the process that the user can select (in update_selection_page(), I guess) that tells the system to update all tables in the database, not just those for the present site.
b) an option for the user to say whether to update the shared tables or not. This way, the user will update each site in each site's interface, but the user will only select the shared table run one time.
c) other ideas?
the only solution that is even somewhat workable in my small brain is to warn the user and then upgrade all the sites at once. i think thats the deal with multi-site. you need to do separate sites if you want them to be on different versions. if you can think of more flexible solutions, thats great.
On 2/24/06, Moshe Weitzman <weitzman@tejasa.com> wrote:
* Are the core developers interested in having multi-site support in the update process? (I mean, why work on something just to have it go nowhere, right?)
yes . multi-site is very much a supported feature. so is the update system. their combination needs to be supproted as well.
Agreed. If we say it is supported, then the upgrade should be seamless.
* What do you think is a better/more workable/more Drupal-like approach:
a) put an option in the process that the user can select (in update_selection_page(), I guess) that tells the system to update all tables in the database, not just those for the present site.
b) an option for the user to say whether to update the shared tables or not. This way, the user will update each site in each site's interface, but the user will only select the shared table run one time.
c) other ideas?
the only solution that is even somewhat workable in my small brain is to warn the user and then upgrade all the sites at once. i think thats the deal with multi-site. you need to do separate sites if you want them to be on different versions. if you can think of more flexible solutions, thats great.
We have to tell the user that this will happen and let them press a button to proceed. By the way, what does Bryght think? They are one of the few multisite hosters our there ... Boris?
On 24 Feb 2006, at 9:42 PM, Khalid B wrote:
the only solution that is even somewhat workable in my small brain is to warn the user and then upgrade all the sites at once. i think thats the deal with multi-site. you need to do separate sites if you want them to be on different versions. if you can think of more flexible solutions, thats great.
We have to tell the user that this will happen and let them press a button to proceed.
By the way, what does Bryght think? They are one of the few multisite hosters our there ... We have automated the entire process and don't have any user interaction for upgrades.
We have multiple versions running at the same time, and updating a site means changing it's virtual host config to point to the right version, doing apachectl graceful and running install.php, which does all the updates it can find. a lot of this is scripted, and has no place in drupal itself. -- Adrian Rossouw Drupal developer and Bryght Guy http://drupal.org | http://bryght.com
the only solution that is even somewhat workable in my small brain is to warn the user and then upgrade all the sites at once. i think thats the deal with multi-site. you need to do separate sites if you want them to be on different versions. if you can think of more flexible solutions, thats great.
How about having: a) a separate table for tracking table versions? or b) Using the System table to hold table versions under "schema_versions" and type "table"? I've not really played with multisites myself: Is the system table shared in certain scenarios? or c) using the comments field of each table to hold version information? But I don't think this would be pgSQL friendly. So each update block would check the table version, apply update and update the table version upon completion. This would allow all sites to be able to apply updates irrespective of whether any of the shared tables have already been updated. How do Bryght et al. handle this? -K
On Fri, Feb 24, 2006 at 02:18:46PM -0500, Moshe Weitzman wrote:
the only solution that is even somewhat workable in my small brain is to warn the user and then upgrade all the sites at once
Here are my thoughts at this moment: * Gather the $db_prefix settings from sites/*/settings.php (as previously mentioned). * Add a function to update.php called something like update_db_query() that replaces db_query() for the update process. This function does the table name prefixing then stores an MD5 hash of the prefixed SQL. Save that hash in an array via variable_set(). Before executing a query, check if the hash exists yet or not. If it exists, move on, otherwise, run the query. Sound reasonable? Other ideas? --Dan -- T H E A N A L Y S I S A N D S O L U T I O N S C O M P A N Y data intensive web and database programming http://www.AnalysisAndSolutions.com/ 4015 7th Ave #4, Brooklyn NY 11232 v: 718-854-0335 f: 718-854-0409
Daniel Convissor wrote:
On Fri, Feb 24, 2006 at 02:18:46PM -0500, Moshe Weitzman wrote:
the only solution that is even somewhat workable in my small brain is to warn the user and then upgrade all the sites at once
Here are my thoughts at this moment:
* Gather the $db_prefix settings from sites/*/settings.php (as previously mentioned).
* Add a function to update.php called something like update_db_query() that replaces db_query() for the update process. This function does the table name prefixing then stores an MD5 hash of the prefixed SQL. Save that hash in an array via variable_set(). Before executing a query, check if the hash exists yet or not. If it exists, move on, otherwise, run the query.
Sound reasonable? Other ideas?
--Dan
sounds reasonable. not sure if it would totally fix this problem but it would help. i suggest trying it and then folks will update their multi-site shared DBs and let us know ... i was wondering how onerous it would be to require update writers to craft their updates such that they did the right thing if run a few times. that would cover us for multi_site shared i think.
Hi: I'm looking at /database/updates.inc and see two glaring issues: 1) Some db_type checks only look for 'mysql' not 'mysqli.' I'll submit a patch. 2) Droping columns isn't being done for pgsql because it didn't support doing so prior to 7.4. I'd like to add a database version check inside update.inc. I'm planning on adding the version discovery solely to /update.php. But I'm wondering if the core folks think would find it useful (and worth the cost) to add it for all connections via /includes/database.*.inc? Thanks, --Dan -- T H E A N A L Y S I S A N D S O L U T I O N S C O M P A N Y data intensive web and database programming http://www.AnalysisAndSolutions.com/ 4015 7th Ave #4, Brooklyn NY 11232 v: 718-854-0335 f: 718-854-0409
Hey Again: Oh, and the create table statements don't specify the character set. I'll add that in the patch. --Dan -- T H E A N A L Y S I S A N D S O L U T I O N S C O M P A N Y data intensive web and database programming http://www.AnalysisAndSolutions.com/ 4015 7th Ave #4, Brooklyn NY 11232 v: 718-854-0335 f: 718-854-0409
On Sun, Feb 26, 2006 at 08:06:10PM -0500, Daniel Convissor wrote:
Oh, and the create table statements don't specify the character set. I'll add that in the patch.
Hmm... I now see that update 169 changes the character set on the tables. But the process checks for the database's default character set before running. If the default character set is already UTF-8, it doesn't run. This check was put in the initial commit, 1.186, by unconed. I feel that may be a mistaken attempt at being efficient. Just because a database's default character set is UTF-8 doesn't mean the necessary tables have that default and that the columns in said tables are that character set. It seems the process should run no matter what. Thoughts? --Dan -- T H E A N A L Y S I S A N D S O L U T I O N S C O M P A N Y data intensive web and database programming http://www.AnalysisAndSolutions.com/ 4015 7th Ave #4, Brooklyn NY 11232 v: 718-854-0335 f: 718-854-0409
Daniel Convissor wrote:
Just because a database's default character set is UTF-8 doesn't mean the necessary tables have that default and that the columns in said tables are that character set. It seems the process should run no matter what.
Thoughts?
This is something that's been driving me nuts for a week, so I'll pitch in. Altering the table and columns does not reliably prevent foreign characters from being irretrievably mangled. If we change tables like this, we also need to change their contents. jh
Altering the table and columns does not reliably prevent foreign characters from being irretrievably mangled.
If we change tables like this, we also need to change their contents.
No we don't. I follow the exact recommendations from the MySQL documentation. In the original issue http://drupal.org/node/51110 this is explained. Drupal has been sending UTF-8 data, but MySQL did not know this. So it interpreted the data as another character set (e.g. Latin-1). This meant that character counts and boundaries were off and that case conversion did not work. The fix is to force mysql to only re-interpret the data, and not change it. The underlying bytes remain the same, but MySQL now knows where to place character boundaries and how to do case conversion the right way. http://dev.mysql.com/doc/refman/4.1/en/charset-conversion.html If your data is corrupted by the update, then it was not in UTF-8 to begin with. Steven Wittens
Daniel Convissor wrote:
Hi:
2) Droping columns isn't being done for pgsql because it didn't support doing so prior to 7.4. I'd like to add a database version check inside update.inc. I'm planning on adding the version discovery solely to /update.php. But I'm wondering if the core folks think would find it useful (and worth the cost) to add it for all connections via /includes/database.*.inc?
Maybe we should just drop support for Postgresql 7.3 and earlier. 7.4 was released 2003-11-17, so it's over 2 years old now. The current production release is 8.1, although 8.0, 7.4 and 7.3 all appear to be supported by the Postgres folks as well. Does anyone know of a good reason to maintain support for 7.3 and earlier? Dropping 7.3 seems like it would solve this problem with dropping columns, and probably a few other problems as well. ..chris
On 27 Feb 2006, at 7:18 AM, Chris Johnson wrote:
Does anyone know of a good reason to maintain support for 7.3 and earlier? Dropping 7.3 seems like it would solve this problem with dropping columns, and probably a few other problems as well.
+1 -- Adrian Rossouw Drupal developer and Bryght Guy http://drupal.org | http://bryght.com
On Mon, 2006-02-27 at 07:30 +0200, Adrian Rossouw wrote:
On 27 Feb 2006, at 7:18 AM, Chris Johnson wrote:
Does anyone know of a good reason to maintain support for 7.3 and earlier? Dropping 7.3 seems like it would solve this problem with dropping columns, and probably a few other problems as well.
+1
I think as long as 7.3 is supported ISP's will be running it... I know established shops don't like upgrading. They wait until they suspect an imminent EOL.
On 27 Feb 2006, at 7:12 PM, Darrel O'Pry wrote:
I think as long as 7.3 is supported ISP's will be running it... I know established shops don't like upgrading. They wait until they suspect an imminent EOL.
well. then they can't run Drupal. Our postgres support is spread thin enough without having to support 3-4 versions back. -- Adrian Rossouw Drupal developer and Bryght Guy http://drupal.org | http://bryght.com
On Mon, 2006-02-27 at 19:29 +0200, Adrian Rossouw wrote:
On 27 Feb 2006, at 7:12 PM, Darrel O'Pry wrote:
I think as long as 7.3 is supported ISP's will be running it... I know established shops don't like upgrading. They wait until they suspect an imminent EOL.
well. then they can't run Drupal. oh well... most don't run it anyway.
Our postgres support is spread thin enough without having to support 3-4 versions back.
Since I know jack about postgresql, you are probably right in the support department.
On Mon, Feb 27, 2006 at 07:30:35AM +0200, Adrian Rossouw wrote:
On 27 Feb 2006, at 7:18 AM, Chris Johnson wrote:
Does anyone know of a good reason to maintain support for 7.3 and earlier? Dropping 7.3 seems like it would solve this problem with dropping columns, and probably a few other problems as well. +1
Fine with me. But before I put forth an upgrade system patch doing that, I'd like to hear from a few more core devs, please. --Dan -- T H E A N A L Y S I S A N D S O L U T I O N S C O M P A N Y data intensive web and database programming http://www.AnalysisAndSolutions.com/ 4015 7th Ave #4, Brooklyn NY 11232 v: 718-854-0335 f: 718-854-0409
On Feb 27, 2006, at 6:53 PM, Daniel Convissor wrote:
On Mon, Feb 27, 2006 at 07:30:35AM +0200, Adrian Rossouw wrote:
On 27 Feb 2006, at 7:18 AM, Chris Johnson wrote:
Does anyone know of a good reason to maintain support for 7.3 and earlier? Dropping 7.3 seems like it would solve this problem with dropping columns, and probably a few other problems as well. +1
Fine with me. But before I put forth an upgrade system patch doing that, I'd like to hear from a few more core devs, please.
I'm hardly a core developer, but I know something of the demand for PostgreSQL hosting. In my experience at the low-to-mid level, very very VERY few people use PostgreSQL. Like, on the order of .05% of hosted sites (including my own!). Since PostgresSQL is a fairly rare offering of commodity host, the providers that support it are more likely to be on their tows. Beyond that, sites that use PostgreSQL are going have advanced technical requirements and better access to administrative support. They're all going to be running a version more current than 7.3, especially since we're talking about a minor revision update (7.4 or higher), which is an easy upgrade. If we leave someone behind, maybe they will volunteer to backport :) Allie Micka pajunas interactive, inc. http://www.pajunas.com scalable web hosting and open source strategies
On Mon, Feb 27, 2006 at 07:53:00PM -0500, Daniel Convissor wrote:
On Mon, Feb 27, 2006 at 07:30:35AM +0200, Adrian Rossouw wrote:
On 27 Feb 2006, at 7:18 AM, Chris Johnson wrote:
Does anyone know of a good reason to maintain support for 7.3 and earlier? Dropping 7.3 seems like it would solve this problem with dropping columns, and probably a few other problems as well. +1
Fine with me. But before I put forth an upgrade system patch doing that, I'd like to hear from a few more core devs, please.
I personally don't see good reasons to drop 7.3. Could someone list pros for dropping it? -- Piotrek irc: #debian.pl Mors Drosophilis melanogastribus!
Hello, On Sun, Feb 26, 2006 at 11:18:39PM -0600, Chris Johnson wrote:
Daniel Convissor wrote:
Hi:
2) Droping columns isn't being done for pgsql because it didn't support
4.6 -> 4.7 update drops column for postgres (if there's an update that doesn't do that, that's a bug).
doing so prior to 7.4. I'd like to add a database version check inside
Postgres supports dropping columns since 7.3, not 7.4. For drupal 4.7 we require postgresql 7.3 or newer.
Maybe we should just drop support for Postgresql 7.3 and earlier. 7.4 was released 2003-11-17, so it's over 2 years old now. The current production
Mysql 4.0.1 was released 2001-12-23. Last mysql 3.x was released 2003-11-11. Let's drop 3.x! ;)
Does anyone know of a good reason to maintain support for 7.3 and earlier? Dropping 7.3 seems like it would solve this problem with dropping columns, and probably a few other problems as well.
There is only 1 problem with 7.3 that I know of, so far: it doesn't support "anyelement" in CREATE FUNCTION, which is used to implement IF() function. But that's easy if someone needs too (the best fix would be to use CASE instead of IF...) I don't have anything against dropping 7.3 support, but OTOH I don't see any gains and "dropping" it would change nothing in fact. -- Piotrek irc: #debian.pl Mors Drosophilis melanogastribus!
On Mon, Feb 27, 2006 at 09:24:29AM +0100, Piotr Krukowiecki wrote:
doing so prior to 7.4. I'd like to add a database version check inside
Postgres supports dropping columns since 7.3, not 7.4. For drupal 4.7 we require postgresql 7.3 or newer.
You're right. http://www.postgresql.org/docs/7.3/interactive/features.html --Dan -- T H E A N A L Y S I S A N D S O L U T I O N S C O M P A N Y data intensive web and database programming http://www.AnalysisAndSolutions.com/ 4015 7th Ave #4, Brooklyn NY 11232 v: 718-854-0335 f: 718-854-0409
participants (15)
-
Adrian Rossouw -
Allie Micka -
Bèr Kessels -
Chris Johnson -
Daniel Convissor -
Darrel O'Pry -
John Handelaar -
Karthik -
Khalid B -
Morbus Iff -
Moshe Weitzman -
Neil Drumm -
piotr＠mallorn.ii.uj.edu.pl -
Steven Wittens -
Walt Daniels