ALL Contrib maintainers: UTF-8 update
Heads up, I committed my UTF-8 patch (http://drupal.org/node/40515). Looks big, but it really isn't /that/ scary. (*) But because of this, we need to apply an upgrade to all tables. So, if you have a contrib module with tables, please go to: http://drupal.org/node/22218#utf8_sql ( http://drupal1.osuosl.org/node/22218#utf8_sql ) Don't worry, it's just a quick copy/paste operation, really. Takes only 5 minutes. Aren't you glad drumm made that awesome update system? ;) Thanks, Steven Wittens (*) And it fixes the last of the UTF-8 issues: database field lengths are now guaranteed to be expressed in characters, not bytes. SQL operations like substring, lower, etc are guaranteed to work. Alphabetical ordering can sort accented letters properly. And much more.
First off, thanks for the head up. I'd just been looking at that update. I want to make sure I'm clear on this. Update 169 got the core tables but we still need up update our tables. If you're using the .install files, I'm assuming that can be something as simple as: function review_update_2() { return _system_update_utf8(array('review')); } Does that sound right? andrew On 1/20/06, Steven Wittens <steven@acko.net> wrote:
Heads up,
I committed my UTF-8 patch (http://drupal.org/node/40515). Looks big, but it really isn't /that/ scary. (*)
But because of this, we need to apply an upgrade to all tables. So, if you have a contrib module with tables, please go to:
http://drupal.org/node/22218#utf8_sql ( http://drupal1.osuosl.org/node/22218#utf8_sql )
Don't worry, it's just a quick copy/paste operation, really. Takes only 5 minutes. Aren't you glad drumm made that awesome update system? ;)
Thanks, Steven Wittens
(*) And it fixes the last of the UTF-8 issues: database field lengths are now guaranteed to be expressed in characters, not bytes. SQL operations like substring, lower, etc are guaranteed to work. Alphabetical ordering can sort accented letters properly. And much more.
Ignore me, I just finally scrolled down to the bottom of the handbook page you linked to :p On 1/20/06, andrew morton <drewish@katherinehouse.com> wrote:
First off, thanks for the head up. I'd just been looking at that update.
I want to make sure I'm clear on this. Update 169 got the core tables but we still need up update our tables. If you're using the .install files, I'm assuming that can be something as simple as: function review_update_2() { return _system_update_utf8(array('review')); }
Does that sound right?
andrew
On 1/20/06, Steven Wittens <steven@acko.net> wrote:
Heads up,
I committed my UTF-8 patch (http://drupal.org/node/40515). Looks big, but it really isn't /that/ scary. (*)
But because of this, we need to apply an upgrade to all tables. So, if you have a contrib module with tables, please go to:
http://drupal.org/node/22218#utf8_sql ( http://drupal1.osuosl.org/node/22218#utf8_sql )
Don't worry, it's just a quick copy/paste operation, really. Takes only 5 minutes. Aren't you glad drumm made that awesome update system? ;)
Thanks, Steven Wittens
(*) And it fixes the last of the UTF-8 issues: database field lengths are now guaranteed to be expressed in characters, not bytes. SQL operations like substring, lower, etc are guaranteed to work. Alphabetical ordering can sort accented letters properly. And much more.
Steven Wittens wrote:
Heads up,
I committed my UTF-8 patch (http://drupal.org/node/40515). Looks big, but it really isn't /that/ scary. (*)
Thanks Steven. I was able to update my site but only after ignoring php warnings at the top of the page. I think some people won't realize that the update.php page is still usable. The first warning: Warning: Illegal mix of collations (utf8_general_ci,COERCIBLE) and (latin1_swedish_ci,IMPLICIT) for operation 'like' query: SELECT * FROM access WHERE status = 1 AND type = 'host' AND LOWER('127.0.0.1') LIKE LOWER(mask)
Hi, Just thought that you would like to know that this doesn't work under php 5. there is this error Fatal error: Only variables can be passed by reference in /var/www/surf/database/updates.inc on line 1496 Gordon. On Sat, 2006-01-21 at 03:01 +0100, Steven Wittens wrote:
Heads up,
I committed my UTF-8 patch (http://drupal.org/node/40515). Looks big, but it really isn't /that/ scary. (*)
But because of this, we need to apply an upgrade to all tables. So, if you have a contrib module with tables, please go to:
http://drupal.org/node/22218#utf8_sql ( http://drupal1.osuosl.org/node/22218#utf8_sql )
Don't worry, it's just a quick copy/paste operation, really. Takes only 5 minutes. Aren't you glad drumm made that awesome update system? ;)
Thanks, Steven Wittens
(*) And it fixes the last of the UTF-8 issues: database field lengths are now guaranteed to be expressed in characters, not bytes. SQL operations like substring, lower, etc are guaranteed to work. Alphabetical ordering can sort accented letters properly. And much more.
!DSPAM:1000,43d19ad9162982006516185!
On Friday 20 January 2006 21:01, Steven Wittens wrote:
Heads up,
I committed my UTF-8 patch (http://drupal.org/node/40515). Looks big, but it really isn't /that/ scary. (*)
But because of this, we need to apply an upgrade to all tables. So, if you have a contrib module with tables, please go to:
http://drupal.org/node/22218#utf8_sql ( http://drupal1.osuosl.org/node/22218#utf8_sql )
Don't worry, it's just a quick copy/paste operation, really. Takes only 5 minutes. Aren't you glad drumm made that awesome update system? ;)
Thanks, Steven Wittens
Eek! I happened to be building a new sandbox site for myself and checked out CVS just before or just after you committed that update. When I try to run the update.php script against my previous 4.7 HEAD database, it loops endlessly doing.....something. I'm not sure what it's up to, but the behavior is repeatable. Maybe I'm the only one to whom this will happen, because I may have gotten a weird half-updated Drupal snapshot, but I thought I ought to report it. In this case, no big deal -- just a sandbox, and I can nuke it and start over. But I wouldn't want this to happen on a production site. Yes, I know to back up my database first. In this case, I didn't bother because of it being a sandbox. :-) It seems as if maybe the hang is in a redirection, because I was seeing repeated HTTP requests in my "netstat" output. I'm going to rebuild the site. Again, I'm not asking for help solving this, because I think I just got a bogus half-committed version, but I wanted to report it just on the off chance that someone else also sees it and it starts to smell like a pattern. Scott -- ------------------------------------------------------------------------------- Scott Courtney Drupal user name: "syscrusher" http://drupal.org/user/9184 scott at 4th dot com Drupal projects: http://drupal.org/project/user/9184 Sandbox: http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/syscrusher
Hi, If you are running php5, then there is a fatal bug that I found. Try running the update script with js turned off, and you will see the error. I actually found the Firefox NoScript extension really handy for this. Gordon. On Fri, 2006-01-20 at 22:02 -0500, Syscrusher wrote:
On Friday 20 January 2006 21:01, Steven Wittens wrote:
Heads up,
I committed my UTF-8 patch (http://drupal.org/node/40515). Looks big, but it really isn't /that/ scary. (*)
But because of this, we need to apply an upgrade to all tables. So, if you have a contrib module with tables, please go to:
http://drupal.org/node/22218#utf8_sql ( http://drupal1.osuosl.org/node/22218#utf8_sql )
Don't worry, it's just a quick copy/paste operation, really. Takes only 5 minutes. Aren't you glad drumm made that awesome update system? ;)
Thanks, Steven Wittens
Eek! I happened to be building a new sandbox site for myself and checked out CVS just before or just after you committed that update. When I try to run the update.php script against my previous 4.7 HEAD database, it loops endlessly doing.....something. I'm not sure what it's up to, but the behavior is repeatable. Maybe I'm the only one to whom this will happen, because I may have gotten a weird half-updated Drupal snapshot, but I thought I ought to report it. In this case, no big deal -- just a sandbox, and I can nuke it and start over. But I wouldn't want this to happen on a production site.
Yes, I know to back up my database first. In this case, I didn't bother because of it being a sandbox. :-)
It seems as if maybe the hang is in a redirection, because I was seeing repeated HTTP requests in my "netstat" output.
I'm going to rebuild the site. Again, I'm not asking for help solving this, because I think I just got a bogus half-committed version, but I wanted to report it just on the off chance that someone else also sees it and it starts to smell like a pattern.
Scott
On Friday 20 January 2006 22:47, Gordon Heydon wrote:
If you are running php5, then there is a fatal bug that I found. Try running the update script with js turned off, and you will see the error.
Thanks for the tip. I can't try it because I've already done what I said, that is, rebuilt from scratch. I needed to get some work done on one of my modules quickly. But I'll keep this handy for next time. Thanks! Scott -- ------------------------------------------------------------------------------- Scott Courtney Drupal user name: "syscrusher" http://drupal.org/user/9184 scott at 4th dot com Drupal projects: http://drupal.org/project/user/9184 Sandbox: http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/syscrusher
I am also getting these errors now: Warning: Illegal mix of collations (utf8_general_ci,COERCIBLE) and (latin1_swedish_ci,IMPLICIT) for operation 'like' query: SELECT * FROM access WHERE status = 1 AND type = 'host' AND LOWER('192.168.0.2') LIKE LOWER(mask) in .../includes/database.mysql.inc on line 118
In order to solve this, you either have to run the update for the specific module, or you can do it manually for every table like so: ALTER TABLE vocabulary_node_types CONVERT TO CHARACTER SET utf8; That takes care of the illegal mix of collations errors. On 1/21/06, Khalid B <kb@2bits.com> wrote:
I am also getting these errors now:
Warning: Illegal mix of collations (utf8_general_ci,COERCIBLE) and (latin1_swedish_ci,IMPLICIT) for operation 'like' query: SELECT * FROM access WHERE status = 1 AND type = 'host' AND LOWER('192.168.0.2') LIKE LOWER(mask) in .../includes/database.mysql.inc on line 118
Khalid B wrote:
In order to solve this, you either have to run the update for the specific module, or you can do it manually for every table like so:
ALTER TABLE vocabulary_node_types CONVERT TO CHARACTER SET utf8;
That takes care of the illegal mix of collations errors.
On 1/21/06, Khalid B <kb@2bits.com> wrote:
I am also getting these errors now:
Warning: Illegal mix of collations (utf8_general_ci,COERCIBLE) and (latin1_swedish_ci,IMPLICIT) for operation 'like' query: SELECT * FROM access WHERE status = 1 AND type = 'host' AND LOWER('192.168.0.2') LIKE LOWER(mask) in .../includes/database.mysql.inc on line 118
You just have to scroll down the page and continue with the update. This is a warning, thats all. No need to do anything manually. I already posted about this in this thread.
Something is still out of sync. In _system_update_utf8(), these tables are mentioned: 'moderation_filters', 'moderation_roles', 'moderation_votes' database.mysql does not have these tables, nor any of the core modules. The result is that there errors in the update.
In order to solve this, you either have to run the update for the specific module, or you can do it manually for every table like so:
ALTER TABLE vocabulary_node_types CONVERT TO CHARACTER SET utf8;
As I explained in the original issue, do /not/ run a query like this. This would do an actual character set conversion, e.g. from Latin1 to UTF-8. Drupal has already been using UTF-8 data, we simply didn't tell MySQL about it (it thought it was e.g. Latin 1 data). Example: You had the character 'é' (Unicode U+E9) in a node. Its UTF-8 representation is (in hexadecimal bytes) "C3 A9". If the database character set was Latin1, then MySQL thought this was 2 characters, because in Latin1 each byte is one character. So, if you do a character set conversion, you would end up with the UTF-8 encoding for character C3 followed by the UTF-8 encoding for character A9, which is "C3 83 C2 A9". What we want MySQL instead to do is to realize the bytestream is already UTF-8 and see that the byte sequence "C3 A9" is a single character. The only way to do is to convert all columns to a binary type and then to UTF-8. This means no actual conversion is done, only re-interpretation: See: http://dev.mysql.com/doc/refman/4.1/en/charset-conversion.html Steven
Op zaterdag 21 januari 2006 04:47, schreef Gordon Heydon:
If you are running php5, then there is a fatal bug that I found. Try running the update script with js turned off, and you will see the error.
I actually found the Firefox NoScript extension really handy for this.
I reported a bug here. When there is *any* fatal error in PHP (being due to contribs etc or anything else) then the nice progressbar just hangs. It is not PHP endless looping. Just that there is no longer any proper feedback. http://drupal.org/node/40386 FWIW: just deleting update.js (moving) is a very easy trick too. -- [ Bèr Kessels | Drupal services www.webschuur.com ]
I now get these errors when attempting visiting update.php. Is it just me? Any ideas? Thanks, -Dave Warning: Illegal mix of collations (utf8_general_ci,COERCIBLE) and (latin1_swedish_ci,IMPLICIT) for operation 'like' query: SELECT * FROM access WHERE status = 1 AND type = 'host' AND LOWER('192.168.1.1') LIKE LOWER(mask) in /home/dave/www/dave-cohen.com/includes/database.mysql.inc on line 118 Warning: Illegal mix of collations (utf8_general_ci,COERCIBLE) and (latin1_swedish_ci,IMPLICIT) for operation 'like' query: SELECT * FROM access WHERE status = 0 AND type = 'host' AND LOWER('192.168.1.1') LIKE LOWER(mask) in /home/dave/www/dave-cohen.com/includes/database.mysql.inc on line 118 Parse error: parse error, unexpected T_SL in /home/dave/www/dave-cohen.com/includes/form.inc on line 473
D'oh! I'd be smart to resolve my conflicts (form.inc) before posting to this group. I still saw the "Illegal mix of collations" errors, when I first visited update.php, but the update ran and hopefully solved the problem. So never mind. -Dave On Sunday 22 January 2006 03:33 pm, Dave Cohen wrote:
I now get these errors when attempting visiting update.php. Is it just me? Any ideas?
Thanks, -Dave
Warning: Illegal mix of collations (utf8_general_ci,COERCIBLE) and (latin1_swedish_ci,IMPLICIT) for operation 'like' query: SELECT * FROM access WHERE status = 1 AND type = 'host' AND LOWER('192.168.1.1') LIKE LOWER(mask) in /home/dave/www/dave-cohen.com/includes/database.mysql.inc on line 118
Warning: Illegal mix of collations (utf8_general_ci,COERCIBLE) and (latin1_swedish_ci,IMPLICIT) for operation 'like' query: SELECT * FROM access WHERE status = 0 AND type = 'host' AND LOWER('192.168.1.1') LIKE LOWER(mask) in /home/dave/www/dave-cohen.com/includes/database.mysql.inc on line 118
Parse error: parse error, unexpected T_SL in /home/dave/www/dave-cohen.com/includes/form.inc on line 473
participants (8)
-
andrew morton -
Bèr Kessels -
Dave Cohen -
Gordon Heydon -
Khalid B -
Moshe Weitzman -
Steven Wittens -
Syscrusher