[development] Queue as a supplement to cron

[Kieran Lal]

On 3/12/07, Neil Drumm <drumm at delocalizedham.com> wrote:
>
> I sometimes find myself building a queue in a hook_cron()
> implementation. I decided to make a separate module for this,
> job_queue, http://drupal.org/project/job_queue. It provides an API
> which takes a function name and array of arguments for later execution
> on cron. The only UI is a list of queued function calls.


We implemented something similar to this for the CivicSpace ASP.  In
general, when people are looking for queues they tend to use other
programming languages like Python that provide daemons as part of the core
language.  PHP's daemon capabilities are lacking so a queue supplements this
missing piece with a cron job working through the queue.

This is particular important when you want to put a lot of load but it can
be spread out.  For example, changing the schema of a thousand sites or
re-indexing search tables for all sites in a multi-site configuration.

Is this functionality something that people would like to see in core?
> Search module has a 'queue' of unindexed content, would it benefit
> from having an abstracted queue?


I'd recommend the following features.

1) Error states for the queue.  If a job fails it will set an error level.
This is particularly useful for queue logging.
2) A way to tell if queue jobs are still running. This would need to be set
for each server instance empirically.  If your queued job takes longer than
the cron cycle, then you can get escalation issues.  You'll want to know
wether your server can process 1, 2, 3, or 10 jobs in the queue
simultaneously.
3) A global counter for multi-site configurations.  If there is a queue, you
probably want it to be aware of the resources being expended on other Drupal
sites on the same server.

Cheers,
Kieran

--
> Neil Drumm
> http://delocalizedham.com
>



-- 
To strive, to seek, to find, and not to yield.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.drupal.org/pipermail/development/attachments/20070320/ea324b60/attachment-0001.htm