problems with shutdown function in search module
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi there, our search module uses a shutdown function in its cron hook. While re-indexing scratch.d.o to test Doug's patch, I encountered the following situation: 1) There are runnign several update routines from search_update_totals in parallel. 2) The final select query in that function run for hours. I believe that 1) is caused by the fact that the shutdown function isn't governed by our cron semaphore. That is, it runs until it is done and it won't stop other cron jobs from running. So if there are enough dirty workds to update, it is possible that there are several of them running in parallel. I've seen as many as 5. Does anybody consider this a problem? I have no idea what causes 2 but would be grateful for suggestions on how to fix the problem. Cheers, Gerhard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFGtwc4fg6TFvELooQRAnfvAKCsAuBwTKd1cXBwO4QZ0LzEnLrbkgCgzEMm 7ZYafVkZRStDqb3tILCycqk= =dyMQ -----END PGP SIGNATURE-----
On Monday 06 August 2007 19:34, Gerhard Killesreiter wrote:
Hi there,
our search module uses a shutdown function in its cron hook. While re-indexing scratch.d.o to test Doug's patch, I encountered the following situation:
1) There are runnign several update routines from search_update_totals in parallel.
2) The final select query in that function run for hours.
I believe that 1) is caused by the fact that the shutdown function isn't governed by our cron semaphore. That is, it runs until it is done and it won't stop other cron jobs from running. So if there are enough dirty workds to update, it is possible that there are several of them running in parallel. I've seen as many as 5. Does anybody consider this a problem?
I am not sure I understand the problem. If you want to make sure only one process runs at a time, we can use a locking mechanism like I do in one module: Making sure we get the lock prevents a race condition, a problem that make poormanscron.module currently unuseable (see link in code comment). /** * Implementation of hook_cron(). */ function hook_cron() { // Check if we have another instance of cron running. if (variable_get('hook_cron_running', 0)) { return; } // hook_cron is not already running. // See if we can get the lock before another process does. // Algorithm found here: http://drupal.org/node/43511. $result = db_query("UPDATE {variable} SET value = 'i:1;' WHERE name = 'hook_cron_running' AND value = 'i:0;'"); if (db_affected_rows() != 1) { // We didn't get the lock. Another process outran us. watchdog('hook_cron', 'This cron process was outraced.'); return; } // Do stuff here... // then, at the end: // Unlock. variable_set('hook_cron_running', 0); return; } -- http://www.wechange.org/ Because we and the world need to change. http://www.reuniting.info/ Intimate Relationships, peace and harmony in the couple.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Augustin (Beginner) schrieb:
On Monday 06 August 2007 19:34, Gerhard Killesreiter wrote:
Hi there,
our search module uses a shutdown function in its cron hook. While re-indexing scratch.d.o to test Doug's patch, I encountered the following situation:
1) There are runnign several update routines from search_update_totals in parallel.
2) The final select query in that function run for hours.
I believe that 1) is caused by the fact that the shutdown function isn't governed by our cron semaphore. That is, it runs until it is done and it won't stop other cron jobs from running. So if there are enough dirty workds to update, it is possible that there are several of them running in parallel. I've seen as many as 5. Does anybody consider this a problem?
I am not sure I understand the problem. If you want to make sure only one process runs at a time, we can use a locking mechanism like I do in one module:
Yes, of course I could. There is however none currently and it might be that it is actually not needed. I am more concerned with the second part of my question. Cheers, Gerhard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFGtyXDfg6TFvELooQRAkEWAKCiNFxUTW0eMQGKetj/S3Mqvc25OwCcDzGb MvC3NbJQG+NBEbJCorL+JJU= =T3df -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Gerhard Killesreiter schrieb:
Augustin (Beginner) schrieb:
On Monday 06 August 2007 19:34, Gerhard Killesreiter wrote:
Hi there,
our search module uses a shutdown function in its cron hook. While re-indexing scratch.d.o to test Doug's patch, I encountered the following situation:
1) There are runnign several update routines from search_update_totals in parallel.
2) The final select query in that function run for hours.
I believe that 1) is caused by the fact that the shutdown function isn't governed by our cron semaphore. That is, it runs until it is done and it won't stop other cron jobs from running. So if there are enough dirty workds to update, it is possible that there are several of them running in parallel. I've seen as many as 5. Does anybody consider this a problem? I am not sure I understand the problem. If you want to make sure only one process runs at a time, we can use a locking mechanism like I do in one module:
Yes, of course I could. There is however none currently and it might be that it is actually not needed. I am more concerned with the second part of my question.
Ok, the second part (and probably the first too) were due to some setup problem on the server. The update script created tables as MyISAM and the server is configured for InnoDB only. Also, the update script omitted an index. Thanks to Narayan for this info. Cheers, Gerhard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFGt7Aefg6TFvELooQRAtGpAKCby1rsUH+UloG3guTAb+s6c/FCxQCghwEF xhOqE+Dkt6eVjcZvpqh/1Ik= =eOeQ -----END PGP SIGNATURE-----
I am not sure I understand the problem. If you want to make sure only one process runs at a time, we can use a locking mechanism like I do in one module:
Making sure we get the lock prevents a race condition, a problem that make poormanscron.module currently unuseable (see link in code comment).
/** * Implementation of hook_cron(). */ function hook_cron() { // Check if we have another instance of cron running. if (variable_get('hook_cron_running', 0)) { return; } // hook_cron is not already running. // See if we can get the lock before another process does. // Algorithm found here: http://drupal.org/node/43511. $result = db_query("UPDATE {variable} SET value = 'i:1;' WHERE name = 'hook_cron_running' AND value = 'i:0;'"); if (db_affected_rows() != 1) { // We didn't get the lock. Another process outran us. watchdog('hook_cron', 'This cron process was outraced.'); return; } // Do stuff here...
// then, at the end: // Unlock. variable_set('hook_cron_running', 0); return; }
Wouldn't the second process be blocked until the first one completes, instead of returning immediately? There is no equivalent to Oracle's NOWAIT in MySQL as far as I recall. -- 2bits.com http://2bits.com Drupal development, customization and consulting.
participants (3)
-
Augustin (Beginner) -
Gerhard Killesreiter -
Khalid Baheyeldin