Hi, I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript... So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that? I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it. Thanks for your feedback. Sven
Batch API works around the PHP timeout limitation by relying on a client browser to iterate separate requests, each of which stays below the time limitation. So yes, Batch API can only be used in a UI context, which excludes cron. For the same reason, it is not recommended to fire a batch processing inside an API function, since you cannot ensure it will be executed in a safe-for-batch context. Yched Le 15/07/2010 11:01, Sven Decabooter a écrit :
Hi,
I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript...
So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that?
I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it.
Thanks for your feedback.
Sven
That's clear, and it makes sense. Thanks Yves! Any pointers as to how I could have large chunks of data processed on cron in another way? Sven On Thu, Jul 15, 2010 at 1:25 PM, Yves Chedemois <yched.drupal@free.fr>wrote:
Batch API works around the PHP timeout limitation by relying on a client browser to iterate separate requests, each of which stays below the time limitation. So yes, Batch API can only be used in a UI context, which excludes cron.
For the same reason, it is not recommended to fire a batch processing inside an API function, since you cannot ensure it will be executed in a safe-for-batch context.
Yched
Le 15/07/2010 11:01, Sven Decabooter a écrit :
Hi,
I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript...
So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that?
I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it.
Thanks for your feedback.
Sven
Queue D6 backport sounds a good idea - http://drupal.org/project/drupal_queue Never tested it, though. Yves Le 15/07/2010 13:28, Sven Decabooter a écrit :
That's clear, and it makes sense. Thanks Yves!
Any pointers as to how I could have large chunks of data processed on cron in another way?
Sven
On Thu, Jul 15, 2010 at 1:25 PM, Yves Chedemois <yched.drupal@free.fr <mailto:yched.drupal@free.fr>> wrote:
Batch API works around the PHP timeout limitation by relying on a client browser to iterate separate requests, each of which stays below the time limitation. So yes, Batch API can only be used in a UI context, which excludes cron.
For the same reason, it is not recommended to fire a batch processing inside an API function, since you cannot ensure it will be executed in a safe-for-batch context.
Yched
Le 15/07/2010 11:01, Sven Decabooter a écrit :
Hi,
I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript...
So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that?
I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it.
Thanks for your feedback.
Sven
Another option to check is the job_queue module. It takes some changes to your module, but the payoff is huge. Check the queue_mail as an example of a module that uses job_queue. On Thu, Jul 15, 2010 at 7:36 AM, Yves Chedemois <yched.drupal@free.fr>wrote:
Queue D6 backport sounds a good idea - http://drupal.org/project/drupal_queue Never tested it, though.
Yves
Le 15/07/2010 13:28, Sven Decabooter a écrit :
That's clear, and it makes sense. Thanks Yves!
Any pointers as to how I could have large chunks of data processed on cron in another way?
Sven
On Thu, Jul 15, 2010 at 1:25 PM, Yves Chedemois <yched.drupal@free.fr>wrote:
Batch API works around the PHP timeout limitation by relying on a client browser to iterate separate requests, each of which stays below the time limitation. So yes, Batch API can only be used in a UI context, which excludes cron.
For the same reason, it is not recommended to fire a batch processing inside an API function, since you cannot ensure it will be executed in a safe-for-batch context.
Yched
Le 15/07/2010 11:01, Sven Decabooter a écrit :
Hi,
I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript...
So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that?
I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it.
Thanks for your feedback.
Sven
-- Khalid M. Baheyeldin 2bits.com, Inc. http://2bits.com Drupal optimization, development, customization and consulting. Simplicity is prerequisite for reliability. -- Edsger W.Dijkstra Simplicity is the ultimate sophistication. -- Leonardo da Vinci
I've used drupal_queue on a project recently and I very much like it. It's a backport of the Drupal 7 queue system, too, which gives you an easier upgrade path in the future. It was designed to scale to millions of queue items, although not withe the default backend plugin. The default one is good for a couple hundred records, probably. It's a really nice architecture overall. --Larry Garfield On 7/15/10 6:36 AM, Yves Chedemois wrote:
Queue D6 backport sounds a good idea - http://drupal.org/project/drupal_queue Never tested it, though.
Yves
Le 15/07/2010 13:28, Sven Decabooter a écrit :
That's clear, and it makes sense. Thanks Yves!
Any pointers as to how I could have large chunks of data processed on cron in another way?
Sven
On Thu, Jul 15, 2010 at 1:25 PM, Yves Chedemois <yched.drupal@free.fr <mailto:yched.drupal@free.fr>> wrote:
Batch API works around the PHP timeout limitation by relying on a client browser to iterate separate requests, each of which stays below the time limitation. So yes, Batch API can only be used in a UI context, which excludes cron.
For the same reason, it is not recommended to fire a batch processing inside an API function, since you cannot ensure it will be executed in a safe-for-batch context.
Yched
Le 15/07/2010 11:01, Sven Decabooter a écrit :
Hi,
I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript...
So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that?
I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it.
Thanks for your feedback.
Sven
On Jul 15, 2010, at 7:36 AM, Yves Chedemois wrote:
Queue D6 backport sounds a good idea - http://drupal.org/project/drupal_queue Never tested it, though.
(I've overlooked this message.) I use Drupal Queue in Feeds and Push Hub and it is very reliable. Check out the README for how to set it up.
Yves
Le 15/07/2010 13:28, Sven Decabooter a écrit :
That's clear, and it makes sense. Thanks Yves!
Any pointers as to how I could have large chunks of data processed on cron in another way?
Sven
On Thu, Jul 15, 2010 at 1:25 PM, Yves Chedemois <yched.drupal@free.fr> wrote: Batch API works around the PHP timeout limitation by relying on a client browser to iterate separate requests, each of which stays below the time limitation. So yes, Batch API can only be used in a UI context, which excludes cron.
For the same reason, it is not recommended to fire a batch processing inside an API function, since you cannot ensure it will be executed in a safe-for-batch context.
Yched
Le 15/07/2010 11:01, Sven Decabooter a écrit :
Hi,
I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript...
So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that?
I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it.
Thanks for your feedback.
Sven
Alex Barth http://www.developmentseed.org/blog tel (202) 250-3633
Sven Decabooter wrote:
Any pointers as to how I could have large chunks of data processed on cron in another way?
Not by using hook_cron but by creating a separate script that you execute within the server cron. The cron process sets a hard 240 seconds to execute hook_cron implementations. -- Earnie -- http://progw.com -- http://www.for-my-kids.com
On Thu, Jul 15, 2010 at 7:46 AM, Earnie Boyd <earnie@users.sourceforge.net> wrote:
Sven Decabooter wrote:
Any pointers as to how I could have large chunks of data processed on cron in another way?
Not by using hook_cron but by creating a separate script that you execute within the server cron. The cron process sets a hard 240 seconds to execute hook_cron implementations.
But remember that you can always call set_time_limit() in your script to adjust the limit, though it just affects PHP's limit, not the webserver's. andrew
I think drush scripts are your best bet. CLI PHP is not not subject to timeout. On Thu, Jul 15, 2010 at 5:01 AM, Sven Decabooter <sdecabooter@gmail.com> wrote:
Hi, I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript... So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that? I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it. Thanks for your feedback. Sven
Drush scripts (especially bulk node processing) are subject to hitting PHP memory limits when processing large amounts of data. Anyone have ways around that issue? On Thu, Jul 15, 2010 at 8:49 AM, Moshe Weitzman <weitzman@tejasa.com> wrote:
I think drush scripts are your best bet. CLI PHP is not not subject to timeout.
On Thu, Jul 15, 2010 at 5:01 AM, Sven Decabooter <sdecabooter@gmail.com> wrote:
Hi, I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript... So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that? I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it. Thanks for your feedback. Sven
-- Ken Rickard agentrickard@gmail.com http://ken.therickards.com
On Thu, Jul 15, 2010 at 9:04 AM, Ken Rickard <agentrickard@gmail.com> wrote:
Drush scripts (especially bulk node processing) are subject to hitting PHP memory limits when processing large amounts of data.
Anyone have ways around that issue?
Besides <?php ini_set(“memory_limit”,”1G”); ?> ? andrew
unset() and paging your batch chunks with LIMIT are the standard strategies if you wrote the cron yourself. You might also be able to use ini_set('memory_limit','200M'); but actually using less memory is generally preferable. - Ken Winters On Jul 15, 2010, at 9:04 AM, Ken Rickard wrote:
Drush scripts (especially bulk node processing) are subject to hitting PHP memory limits when processing large amounts of data.
Anyone have ways around that issue?
On Thu, Jul 15, 2010 at 8:49 AM, Moshe Weitzman <weitzman@tejasa.com> wrote:
I think drush scripts are your best bet. CLI PHP is not not subject to timeout.
On Thu, Jul 15, 2010 at 5:01 AM, Sven Decabooter <sdecabooter@gmail.com
wrote: Hi, I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript... So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that? I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it. Thanks for your feedback. Sven
-- Ken Rickard agentrickard@gmail.com http://ken.therickards.com
http://drupal.org/project/job_queue I use it all of the time and it works great. Sometimes I will make a custom module with a system weight lower then job_queue and ini_set... in hook_cron to increase a timeout if needed. Cheers, Neil On Thu, Jul 15, 2010 at 9:11 AM, Ken Winters <kwinters@coalmarch.com> wrote:
unset() and paging your batch chunks with LIMIT are the standard strategies if you wrote the cron yourself.
You might also be able to use ini_set('memory_limit','200M'); but actually using less memory is generally preferable.
- Ken Winters
On Jul 15, 2010, at 9:04 AM, Ken Rickard wrote:
Drush scripts (especially bulk node processing) are subject to hitting
PHP memory limits when processing large amounts of data.
Anyone have ways around that issue?
On Thu, Jul 15, 2010 at 8:49 AM, Moshe Weitzman <weitzman@tejasa.com> wrote:
I think drush scripts are your best bet. CLI PHP is not not subject to timeout.
On Thu, Jul 15, 2010 at 5:01 AM, Sven Decabooter <sdecabooter@gmail.com> wrote:
Hi, I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript... So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that? I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it. Thanks for your feedback. Sven
-- Ken Rickard agentrickard@gmail.com http://ken.therickards.com
I ran into this before, and implemented a workaround. No amount of unset() would free memory. The workaround was to ration the number of loop iterations to do via the script, and then end it and start a new one from where you left off. while true do #php CLI or drush script goes here NUM=$? TOTAL=`expr $TOTAL + $NUM` if [ $NUM = 0 ]; then break fi sleep 30 done The PHP CLI or drush script returns the number of nodes processed. If it is 0 then the script exits. If there are more, then another iteration is done. This way, PHP's memory does not grow indefinitely and then bombs out eventually. On Thu, Jul 15, 2010 at 9:04 AM, Ken Rickard <agentrickard@gmail.com> wrote:
Drush scripts (especially bulk node processing) are subject to hitting PHP memory limits when processing large amounts of data.
Anyone have ways around that issue?
On Thu, Jul 15, 2010 at 8:49 AM, Moshe Weitzman <weitzman@tejasa.com> wrote:
I think drush scripts are your best bet. CLI PHP is not not subject to timeout.
On Thu, Jul 15, 2010 at 5:01 AM, Sven Decabooter <sdecabooter@gmail.com> wrote:
Hi, I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript... So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that? I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it. Thanks for your feedback. Sven
-- Ken Rickard agentrickard@gmail.com http://ken.therickards.com
-- Khalid M. Baheyeldin 2bits.com, Inc. http://2bits.com Drupal optimization, development, customization and consulting. Simplicity is prerequisite for reliability. -- Edsger W.Dijkstra Simplicity is the ultimate sophistication. -- Leonardo da Vinci
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Khalid Baheyeldin schrieb:
I ran into this before, and implemented a workaround.
No amount of unset() would free memory.
You should try again using php 5.3, the memory management is said to be much better. This is especially awesome for huge complex structures such as views. Cheers, Gerhard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iEYEARECAAYFAkw/HPsACgkQfg6TFvELooSSvACgygL0KyQWPRSsGzXnn96UB8dH tMkAn2L0ogxomRwikyjTHRVS9EKKcgIS =30be -----END PGP SIGNATURE-----
On Jul 15, 2010, at 9:04 AM, Ken Rickard wrote:
Drush scripts (especially bulk node processing) are subject to hitting PHP memory limits when processing large amounts of data.
Anyone have ways around that issue?
Again: batching. Even tasks that are run with drush need to be worked off in chunks. PHP's just not made for managing memory.
On Thu, Jul 15, 2010 at 8:49 AM, Moshe Weitzman <weitzman@tejasa.com> wrote:
I think drush scripts are your best bet. CLI PHP is not not subject to timeout.
On Thu, Jul 15, 2010 at 5:01 AM, Sven Decabooter <sdecabooter@gmail.com
wrote: Hi, I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript... So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that? I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it. Thanks for your feedback. Sven
-- Ken Rickard agentrickard@gmail.com http://ken.therickards.com
Alex Barth http://www.developmentseed.org/blog tel (202) 250-3633
I just ran into this exact problem. I elected to solve it via the D6 backport of drupal_queue/cron (after trying BatchAPI) and am pretty happy with it. Joe On Jul 15, 2010, at 10:24 AM, Alex Barth wrote:
On Jul 15, 2010, at 9:04 AM, Ken Rickard wrote:
Drush scripts (especially bulk node processing) are subject to hitting PHP memory limits when processing large amounts of data.
Anyone have ways around that issue?
Again: batching. Even tasks that are run with drush need to be worked off in chunks.
PHP's just not made for managing memory.
On Thu, Jul 15, 2010 at 8:49 AM, Moshe Weitzman <weitzman@tejasa.com> wrote:
I think drush scripts are your best bet. CLI PHP is not not subject to timeout.
On Thu, Jul 15, 2010 at 5:01 AM, Sven Decabooter <sdecabooter@gmail.com> wrote:
Hi, I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript... So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that? I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it. Thanks for your feedback. Sven
-- Ken Rickard agentrickard@gmail.com http://ken.therickards.com
Alex Barth http://www.developmentseed.org/blog tel (202) 250-3633
drush has some built-in support for redispatching (typically used when when memory gets too low). this is used my migrate module and drush's own updb command. though others have noted that it can be just as easy to bail out when you get too low on memory and call same drush command on your own. On Thu, Jul 15, 2010 at 9:04 AM, Ken Rickard <agentrickard@gmail.com> wrote:
Drush scripts (especially bulk node processing) are subject to hitting PHP memory limits when processing large amounts of data.
Anyone have ways around that issue?
On Thu, Jul 15, 2010 at 8:49 AM, Moshe Weitzman <weitzman@tejasa.com> wrote:
I think drush scripts are your best bet. CLI PHP is not not subject to timeout.
On Thu, Jul 15, 2010 at 5:01 AM, Sven Decabooter <sdecabooter@gmail.com> wrote:
Hi, I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript... So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that? I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it. Thanks for your feedback. Sven
-- Ken Rickard agentrickard@gmail.com http://ken.therickards.com
On Jul 15, 2010, at 10:34 AM, Moshe Weitzman wrote:
drush has some built-in support for redispatching (typically used when when memory gets too low). this is used my migrate module and drush's own updb command. though others have noted that it can be just as easy to bail out when you get too low on memory and call same drush command on your own.
That's useful to know. Let me digress a little: I have recently seen how Fusion Tables http://tables.googlelabs.com does uploads in the background without repeated page loads through the client. I like that. After the file upload, you can leave the upload page and come back in an hour to check whether it's done. Drupal's Batch API tempts us all too much to use client based batching which isn't good for very heavy tasks. It's on my long list for improvements to Feeds to figure out how to build clean dispatching for batched tasks in the background. Drupal 7's queue and redispatching in drush is going to be of tremendous help. I'd appreciate any further pointers.
On Thu, Jul 15, 2010 at 9:04 AM, Ken Rickard <agentrickard@gmail.com> wrote:
Drush scripts (especially bulk node processing) are subject to hitting PHP memory limits when processing large amounts of data.
Anyone have ways around that issue?
On Thu, Jul 15, 2010 at 8:49 AM, Moshe Weitzman <weitzman@tejasa.com> wrote:
I think drush scripts are your best bet. CLI PHP is not not subject to timeout.
On Thu, Jul 15, 2010 at 5:01 AM, Sven Decabooter <sdecabooter@gmail.com
wrote: Hi, I'm reading contradicting posts about running Batch API processes on cron. This is for Drupal 6 BTW. I have tried implementing a batch functionality that should be run on cron, but it doesn't seem to process the work that needs to be done. I assume this is because running the cron through a commandline command doesn't allow for javascript... So my questions: - Have I implemented Batch API incorrectly, and should it normally work also on cron? - What is the best way to run a process that would normally trigger a php script timeout? Can I use the Queue module for that? I'm sure plenty of people have already tried doing this, so I'm not sure why I can find little consistent information about it. Thanks for your feedback. Sven
-- Ken Rickard agentrickard@gmail.com http://ken.therickards.com
Alex Barth http://www.developmentseed.org/blog tel (202) 250-3633
participants (13)
-
Alex Barth -
andrew morton -
Earnie Boyd -
Gerhard Killesreiter -
Joe Pletcher -
Ken Rickard -
Ken Winters -
Khalid Baheyeldin -
larry＠garfieldtech.com -
Moshe Weitzman -
Neil Hastings -
Sven Decabooter -
Yves Chedemois