I sometimes find myself building a queue in a hook_cron() implementation. I decided to make a separate module for this, job_queue, http://drupal.org/project/job_queue. It provides an API which takes a function name and array of arguments for later execution on cron. The only UI is a list of queued function calls. Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue? -- Neil Drumm http://delocalizedham.com
On 3/12/07, Neil Drumm <drumm@delocalizedham.com> wrote:
I sometimes find myself building a queue in a hook_cron() implementation. I decided to make a separate module for this, job_queue, http://drupal.org/project/job_queue. It provides an API which takes a function name and array of arguments for later execution on cron. The only UI is a list of queued function calls.
Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue?
aggregator (and other, contrib-based aggregation) would benefit from this as well, I believe. -- Boris Mann Vancouver 778-896-2747 San Francisco 415-367-3595 Skype borismann http://www.bryght.com
On 3/12/07, Boris Mann <boris@bryght.com> wrote:
On 3/12/07, Neil Drumm <drumm@delocalizedham.com> wrote:
I sometimes find myself building a queue in a hook_cron() implementation. I decided to make a separate module for this, job_queue, http://drupal.org/project/job_queue. It provides an API which takes a function name and array of arguments for later execution on cron. The only UI is a list of queued function calls.
Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue?
aggregator (and other, contrib-based aggregation) would benefit from this as well, I believe.
Those happen regularly, so cron is best. Ping module could use this since you want to queue those HTTP requests for later. -- Neil Drumm http://delocalizedham.com
On 3/12/07, Neil Drumm <drumm@delocalizedham.com> wrote:
On 3/12/07, Boris Mann <boris@bryght.com> wrote:
Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue?
aggregator (and other, contrib-based aggregation) would benefit from this as well, I believe.
Those happen regularly, so cron is best.
Errr....except for mass numbers of aggregation where one cron run can't crank through everything. Queue might not be the right solution, but I wanted to point out that current cron architecture is not up to the task.
Ping module could use this since you want to queue those HTTP requests for later.
-- Boris Mann Vancouver 778-896-2747 San Francisco 415-367-3595 Skype borismann http://www.bryght.com
On 3/12/07, Boris Mann <boris@bryght.com> wrote:
On 3/12/07, Neil Drumm <drumm@delocalizedham.com> wrote:
On 3/12/07, Boris Mann <boris@bryght.com> wrote:
Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue?
aggregator (and other, contrib-based aggregation) would benefit from this as well, I believe.
Those happen regularly, so cron is best.
Errr....except for mass numbers of aggregation where one cron run can't crank through everything. Queue might not be the right solution, but I wanted to point out that current cron architecture is not up to the task.
Adding individual items to the queue on cron might be a good pattern. That is what I'll be doing to import the pile o' XML that changes every day, which I've got to deal with for a project. I coded this to run queue items on cron until more than half of the PHP allowed execution time has been used. -- Neil Drumm http://delocalizedham.com
On Mon, 2007-03-12 at 18:28 -0700, Neil Drumm wrote:
On 3/12/07, Boris Mann <boris@bryght.com> wrote:
On 3/12/07, Neil Drumm <drumm@delocalizedham.com> wrote:
On 3/12/07, Boris Mann <boris@bryght.com> wrote:
Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue?
aggregator (and other, contrib-based aggregation) would benefit from this as well, I believe.
Those happen regularly, so cron is best.
Errr....except for mass numbers of aggregation where one cron run can't crank through everything. Queue might not be the right solution, but I wanted to point out that current cron architecture is not up to the task.
Adding individual items to the queue on cron might be a good pattern. That is what I'll be doing to import the pile o' XML that changes every day, which I've got to deal with for a project.
I coded this to run queue items on cron until more than half of the PHP allowed execution time has been used.
You could even process the queue with a daemon on another server, so you don't bog your web server. I like it.. I'm already doing it with another module I've written. .darrel.
On 3/13/07 5:24 PM, Darrel O'Pry wrote:
On Mon, 2007-03-12 at 18:28 -0700, Neil Drumm wrote:
On 3/12/07, Boris Mann <boris@bryght.com> wrote:
On 3/12/07, Neil Drumm <drumm@delocalizedham.com> wrote:
On 3/12/07, Boris Mann <boris@bryght.com> wrote:
Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue? aggregator (and other, contrib-based aggregation) would benefit from this as well, I believe. Those happen regularly, so cron is best. Errr....except for mass numbers of aggregation where one cron run can't crank through everything. Queue might not be the right solution, but I wanted to point out that current cron architecture is not up to the task. Adding individual items to the queue on cron might be a good pattern. That is what I'll be doing to import the pile o' XML that changes every day, which I've got to deal with for a project.
I coded this to run queue items on cron until more than half of the PHP allowed execution time has been used.
You could even process the queue with a daemon on another server, so you don't bog your web server. I like it.. I'm already doing it with another module I've written.
.darrel.
+1 yeah this is a long mentioned - ne'er implemented thing I'd love to see ... and Neil, I love the idea of generalizing (i.e. not just a mail queue). -- James Walker :: http://walkah.net/ :: xmpp:walkah@walkah.net
On 3/12/07, Boris Mann <boris@bryght.com> wrote:
On 3/12/07, Neil Drumm <drumm@delocalizedham.com> wrote:
On 3/12/07, Boris Mann <boris@bryght.com> wrote:
Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue?
aggregator (and other, contrib-based aggregation) would benefit from this as well, I believe.
Those happen regularly, so cron is best.
Errr....except for mass numbers of aggregation where one cron run can't crank through everything. Queue might not be the right solution, but I wanted to point out that current cron architecture is not up to the task.
Pathauto's bulk update has a similar need to "alias {configurable number of nodes} in a queued fasion." I'm not sure that it would be best in cron, necessarily, but it is similar to the problem Boris described. If the job_queue api stored the function and the arguments to pass to the function that would be nice. A quick look at http://cvs.drupal.org/viewcvs/drupal/contributions/modules/job_queue/job_que... makes it seem like it does provide that. Yay! Regards, Greg
On Monday 12 March 2007 20:45, Neil Drumm wrote:
I sometimes find myself building a queue in a hook_cron() implementation. I decided to make a separate module for this, job_queue, http://drupal.org/project/job_queue. It provides an API which takes a function name and array of arguments for later execution on cron. The only UI is a list of queued function calls.
Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue?
If you decide to do this as a contrib module, you may want to consider making it part of my "cronplus" module. Cronplus doesn't deal with the queueing, but rather creates a set of API hooks that are called at specific time intervals (daily, weekly, etc.) rather than every time cron.php executes. Of course the timing isn't exact because cronplus itself only triggers on cron runs, but it gives a reasonable approximation for batch jobs. It would seem to me that your queue manager might be an interesting extension to cronplus. I won't be offended if you want to go your own way, but if you want to consider merging into cronplus I'd certainly be willing to let that happen. Kind regards, Scott (Syscrusher) -- ------------------------------------------------------------------------------- Syscrusher (Scott Courtney) Drupal page: http://drupal.org/user/9184 syscrusher at 4th dot com Home page: http://4th.com/
Neil Drumm wrote:
I sometimes find myself building a queue in a hook_cron() implementation. I decided to make a separate module for this, job_queue, http://drupal.org/project/job_queue. It provides an API which takes a function name and array of arguments for later execution on cron. The only UI is a list of queued function calls.
Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue?
actions module could benefit from this GREATLY, since dropping actions into job queue == easy scheduling.
Neil Drumm wrote:
I sometimes find myself building a queue in a hook_cron() implementation. I decided to make a separate module for this, job_queue, http://drupal.org/project/job_queue. It provides an API which takes a function name and array of arguments for later execution on cron. The only UI is a list of queued function calls.
Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue?
Not only cron. A HTTP queue would be better in general :) As we were talking about recently. Update.php has a frontend facing job queue, which is (apart from storing the queue in the session not the database) a very much similar situation. Other frontend facing queues would greatly benefit from this too. Like a long lasting data import in an install profile (we are talking about PO file imports for example). Gabor
On 13 Mar 2007, at 01:45, Neil Drumm wrote:
Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue?
I think there is enough interest in this functionality that it warrants including in core -- especially if we can clean up a thing or two in core as well. Submit a patch! :) -- Dries Buytaert :: http://www.buytaert.net/
I manually implement an amortized queue in my caching module, too. Such a feature in core would be useful. Dries Buytaert wrote:
On 13 Mar 2007, at 01:45, Neil Drumm wrote:
Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue?
I think there is enough interest in this functionality that it warrants including in core -- especially if we can clean up a thing or two in core as well. Submit a patch! :)
-- Dries Buytaert :: http://www.buytaert.net/
On 3/14/07, Dries Buytaert <dries.buytaert@gmail.com> wrote:
On 13 Mar 2007, at 01:45, Neil Drumm wrote:
Is this functionality something that people would like to see in core? Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue?
I think there is enough interest in this functionality that it warrants including in core -- especially if we can clean up a thing or two in core as well. Submit a patch! :)
Writing this into system module is in my todo list of core patches to write. -- Neil Drumm http://delocalizedham.com
What would happen if you re-index the site? Every node gets added to the queue? Steven Wittens
On 3/12/07, Neil Drumm <drumm@delocalizedham.com> wrote:
I sometimes find myself building a queue in a hook_cron() implementation. I decided to make a separate module for this, job_queue, http://drupal.org/project/job_queue. It provides an API which takes a function name and array of arguments for later execution on cron. The only UI is a list of queued function calls.
We implemented something similar to this for the CivicSpace ASP. In general, when people are looking for queues they tend to use other programming languages like Python that provide daemons as part of the core language. PHP's daemon capabilities are lacking so a queue supplements this missing piece with a cron job working through the queue. This is particular important when you want to put a lot of load but it can be spread out. For example, changing the schema of a thousand sites or re-indexing search tables for all sites in a multi-site configuration. Is this functionality something that people would like to see in core?
Search module has a 'queue' of unindexed content, would it benefit from having an abstracted queue?
I'd recommend the following features. 1) Error states for the queue. If a job fails it will set an error level. This is particularly useful for queue logging. 2) A way to tell if queue jobs are still running. This would need to be set for each server instance empirically. If your queued job takes longer than the cron cycle, then you can get escalation issues. You'll want to know wether your server can process 1, 2, 3, or 10 jobs in the queue simultaneously. 3) A global counter for multi-site configurations. If there is a queue, you probably want it to be aware of the resources being expended on other Drupal sites on the same server. Cheers, Kieran --
Neil Drumm http://delocalizedham.com
-- To strive, to seek, to find, and not to yield.
participants (12)
-
Boris Mann -
Darrel O'Pry -
David Strauss -
Dries Buytaert -
Earl Miles -
Gabor Hojtsy -
Greg Knaddison - GVS -
James Walker -
Kieran Lal -
Neil Drumm -
Steven Wittens -
Syscrusher