[drupal-devel] [wanted] mail queue
Anyone with a mail background that would like to investigate http:// drupal.org/node/28604 ? The idea is to create a mail.inc for core that implements a mail queue, as well as provides common functionality such as converting HTML to text. It would be a centralized building block for modules like notify.module, project.module, subscription.module, simplenews.module, etc. -- Dries Buytaert :: http://www.buytaert.net/
On Wed, 17 Aug 2005, Dries Buytaert wrote:
Anyone with a mail background that would like to investigate http:// drupal.org/node/28604 ?
Maybe.
The idea is to create a mail.inc for core that implements a mail queue, as well as provides common functionality such as converting HTML to text.
It would be a centralized building block for modules like notify.module, project.module, subscription.module, simplenews.module, etc.
I've spent some time discussing mail and Drupal with Kieran and I have also been working on the massmailer module. Here are my thoughts on what I think we need: 1) Front ends. that would be any module that wishes for some reason to send single or mass mail. 2) API. A module that takes mail from frontends and hands it off to backends. It has APIs to get mail and to provide (to 1)) and collect (from 3)) feedback on mails sent (think: bounces, failures). 3) backends. Anything that is capable of sending mail, either directly or through the system's MTA. A backend can expose a certain set of features. If the backend does not implement a certain feature, the API would need to fall back on something else. The current massmailer module is a merger of 1) and 2) while relying on phplist as a backend (addressed through a phplist.module). In principle more backends could have been written, but nobody did that. The backend is also not complete (no bounce handling, although phplist can do that). What should be in Drupal core? - frontends: user.module, contact.module, simplenews? - API - user_mail as a backend. Maybe some ligthweight library. This architecture has the benefit that it is very flexible and allows everybody to swap in his pet mailer. I have on purpose left out subjects of mail formatting as they are not really a problem of mail sending. There could be an extra .inc file for formatting, but I don't see this as a core issue. BTW: The fact that massmailer was so buggy seems to indicate that the demand for massmailing beyond user_mail()'s capabilities is maybe overestimated. ;-> Cheers, Gerhard
BTW: The fact that massmailer was so buggy seems to indicate that the demand for massmailing beyond user_mail()'s capabilities is maybe overestimated. ;->
I think rather that it is more complex and therefore not accessible to hobby Drupalists. I personally think demand is large for mailing lists, newsletters and the like, but that it is too hard. -Robert
On Wed, 17 Aug 2005, Robert Douglass wrote:
BTW: The fact that massmailer was so buggy seems to indicate that the demand for massmailing beyond user_mail()'s capabilities is maybe overestimated. ;->
I think rather that it is more complex and therefore not accessible to hobby Drupalists.
True massmailing (as in tens of thousands of mails) will _never_ be available to hobby Drupalists with el cheapo hosting. Why? - technical difficulties - any sane webhoster will deny sending mass mails to cheap customers for fear of getting blacklisted. And if you just need to send a few hundred mails, you probably will do fine with user_mail() and a cronjob.
I personally think demand is large for mailing lists, newsletters and the like, but that it is too hard.
Proper mailing _is_ hard. Cheers, Gerhard
On 17 Aug 2005, at 12:38, Gerhard Killesreiter wrote:
BTW: The fact that massmailer was so buggy seems to indicate that the demand for massmailing beyond user_mail()'s capabilities is maybe overestimated. ;->
I think rather that it is more complex and therefore not accessible to hobby Drupalists.
True massmailing (as in tens of thousands of mails) will _never_ be available to hobby Drupalists with el cheapo hosting.
Why?
- technical difficulties - any sane webhoster will deny sending mass mails to cheap customers for fear of getting blacklisted.
And if you just need to send a few hundred mails, you probably will do fine with user_mail() and a cronjob.
True, and that is probably what the default/initial backend should look like. Right now, all these modules implement their own "cronjob + user_mail" function. Let's make this available as a service. In addition we could provide a centralized way of dealing with HTML mail, a mechanism to throttle the rate of outgoing e-mails, a mechanism to convert HTML mails to plain text, and - in future - maybe a mechanism to track and report back bounces. As you said, proper mailing is hard. Hence the need to make this a specialized service and to extract this from the individual modules. Hence my call for people with experience, or for people willing to learn more about this. I don't expect this to be implemented overnight, but it would be nice if some people could explore our options, see what other projects have done, prototype some code, etc. -- Dries Buytaert :: http://www.buytaert.net/
On Wed, 17 Aug 2005, Dries Buytaert wrote:
On 17 Aug 2005, at 12:38, Gerhard Killesreiter wrote:
And if you just need to send a few hundred mails, you probably will do fine with user_mail() and a cronjob.
True, and that is probably what the default/initial backend should look like.
Glad we agree.
Right now, all these modules implement their own "cronjob + user_mail" function. Let's make this available as a service.
Ok
In addition we could provide a centralized way of dealing with HTML mail,
That is really unrelated and nothing I will work on beyond htmlstriptags(). :)
a mechanism to throttle the rate of outgoing e-mails, a mechanism to convert HTML mails to plain text,
Can the filter sytem be used for this?
and - in future - maybe a mechanism to track and report back bounces.
This is in fact rather easy to implement if you use mailhandler to access the designated bounces mailbox. The hard part is to figure out which address actually bounced. :-/
As you said, proper mailing is hard. Hence the need to make this a specialized service and to extract this from the individual modules. Hence my call for people with experience, or for people willing to learn more about this. I don't expect this to be implemented overnight, but it would be nice if some people could explore our options, see what other projects have done, prototype some code, etc.
If people abstain from nagging about html mail I could work on that after the revisions patch is done. Cheers, Gerhard
On 17-Aug-05, at 7:23 AM, Gerhard Killesreiter wrote:
a mechanism to throttle the rate of outgoing e-mails, a mechanism to convert HTML mails to plain text,
Can the filter sytem be used for this?
+1 ... kind of a "reverse markdown/textile" filter - would be helpful in general. for those that don't know - prefixing any idea with "reverse" makes it new and exciting ;) -- James Walker :: http://walkah.net/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 18 Aug 2005, at 7:11 PM, James Walker wrote:
+1 ... kind of a "reverse markdown/textile" filter - would be helpful in general.
for those that don't know - prefixing any idea with "reverse" makes it new and exciting ;)
Which is why we need to start playing reverse patch bingo. - -- Adrian Rossouw Drupal developer and Bryght Guy http://drupal.org | http://bryght.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFDDZOlgegMqdGlkasRAkitAKCu9VFd/EvYFkkhUjjuTpYEhszHxQCeKJPr JPrEG8dzgkvEqeMmkhU48XM= =0JEf -----END PGP SIGNATURE-----
As you said, proper mailing is hard. Hence the need to make this a specialized service and to extract this from the individual modules. Hence my call for people with experience, or for people willing to learn more about this. I don't expect this to be implemented overnight, but it would be nice if some people could explore our options, see what other projects have done, prototype some code, etc.
I agree we need a separate mail.inc. This is partly just because mailing needs go far beyond what the user module needs. The user module should be one of a number of front ends. Some potential approaches are implemented in the mail.module, and that might be a better starting point for discussions than the current user_mail() function. Mail.module implements a custom mail_send() function, which handles mail format, priority, attachments, character set, encoding, etc. as well as testing for send success. The sending itself is handled by a LGPL PHP mailing class, activeMailLib, which was introduced into the module through a patch by Dublin Drupaller in order to implement attachment support and other functionality. I would look to a mail.inc component to handle: 1. Mail sending. Accept a $mail argument with the following properties: * to (string or array) * from (string) * format (string: html or plain text) * subject (string) * priority (int) * receipt (boolean) * character set * encode * files/attachments (array) 2. System-wide mail-related permissions and configuration options a. permissions, possibly: 'email users' 'email users in own role' 'email nonusers' b. default sending options (format, character set, etc.). In mail.module, this is currently implemented through mail_settings(), which provides system defaults, and mail_form_elements($node), which provides a node editing form group for overriding the system defaults. 3. Queuing 4. Bounce tracking and handling To implement this approach, we could: * start with applicable mail.module functions * for mail handling, use activeMailLib (or another candidate), or (preferably) adapt it to Drupal coding approaches. Queuing and bounce tracking/handling would be added later. Thoughts? Nedjo
BTW: The fact that massmailer was so buggy seems to indicate that the demand for massmailing beyond user_mail()'s capabilities is maybe overestimated. ;->
IMO subscriptions / notify etc. is popular and all of them require a proper mass mailer backend. By "mass mailer" I mean something that can handle a mail queue (most likely a database table) and send it off . I know that "traditionally" mass mailing means one letter sent to N addresses.
On 17 Aug 2005, at 10:46, Dries Buytaert wrote:
Anyone with a mail background that would like to investigate http:// drupal.org/node/28604 ?
The idea is to create a mail.inc for core that implements a mail queue, as well as provides common functionality such as converting HTML to text.
It would be a centralized building block for modules like notify.module, project.module, subscription.module, simplenews.module, etc.
Lots of people showed interest in working on this, though as far as I know no one stepped forward to take a lead. I suggest someone creates a _minimal_ mail.inc first. As soon that is in place, everyone can submit patches to extend the mail queue functionality. A minimal mail.inc implements a mail queue, an mail-function, a cron- function, and a setting to throttle the rate of outgoing mails. The mail-function' should have two modes: 'send mail later', 'send mail now'. Regardless of this mode, the accounting should be correct. We can worry about HTML-mail, embedding images, bounces in subsequent rounds. Please raise your hand if you commit to working on this in the next few weeks/days. Thanks. -- Dries Buytaert :: http://www.buytaert.net/
On 24-Aug-05, at 11:37 PM, Dries Buytaert wrote:
On 17 Aug 2005, at 10:46, Dries Buytaert wrote:
Anyone with a mail background that would like to investigate http://drupal.org/node/28604 ?
The idea is to create a mail.inc for core that implements a mail queue, as well as provides common functionality such as converting HTML to text.
It would be a centralized building block for modules like notify.module, project.module, subscription.module, simplenews.module, etc.
Lots of people showed interest in working on this, though as far as I know no one stepped forward to take a lead. I suggest someone creates a _minimal_ mail.inc first. As soon that is in place, everyone can submit patches to extend the mail queue functionality.
A minimal mail.inc implements a mail queue, an mail-function, a cron-function, and a setting to throttle the rate of outgoing mails. The mail-function' should have two modes: 'send mail later', 'send mail now'. Regardless of this mode, the accounting should be correct. We can worry about HTML-mail, embedding images, bounces in subsequent rounds.
Please raise your hand if you commit to working on this in the next few weeks/days. Thanks.
Jonathan is doing some work for us on (finally!) merging subscription + notify, and making it general enough to handle multiple forms of subscription (i.e. email subscription for which the mail.inc stuff is necessary, RSS subscription, SMS, Jabber, etc. etc.) I'll see if the mail stuff makes sense for him to do at the same time. Probably not the HTML-to-text, although there is a ton of code for that floating around (the mailhandler filter throws errors right now) -- sounds like a default email filter. -- Boris Mann http://www.bmannconsulting.com
Boris Mann wrote:
Jonathan is doing some work for us on (finally!) merging subscription + notify, and making it general enough to handle multiple forms of subscription (i.e. email subscription for which the mail.inc stuff is necessary, RSS subscription, SMS, Jabber, etc. etc.)
I'll see if the mail stuff makes sense for him to do at the same time. Probably not the HTML-to-text, although there is a ton of code for that floating around (the mailhandler filter throws errors right now) -- sounds like a default email filter.
Make sure and look at what Márton has done with the subscriptions module: http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/ee/ From what I've seen this is real high quality work and different modules can register their subscription needs with it. Chx can you elaborate? (Márton, are you this list too?) Robert
Jonathan is doing some work for us on (finally!) merging subscription + notify, and making it general enough to handle multiple forms of
In sandbox/ee you can find a subscriptions module from Elek Marton, an SoC participant. I am reviewing the code currently, and I know there are issues, but once those minor stuff are cleared, it'll a lot better than anything else available. It's really extendible and so on.
On 26 Aug 2005, at 02:53, Boris Mann wrote:
I'll see if the mail stuff makes sense for him to do at the same time. Probably not the HTML-to-text
Already available, I added a module to contrib to provide a HTML to text service: http://drupal.org/project/html2txt Note: this does more than just strip out HTML - markup is replaced with text formatting equivalents, e.g. bullet points on unordered lists get replaced with '*.', and links can be collected for inclusion as footnotes. Best regards, Robert
participants (9)
-
Adrian Rossouw -
Boris Mann -
Dries Buytaert -
Gerhard Killesreiter -
James Walker -
Karoly Negyesi -
Nedjo Rogers -
Robert Castelo -
Robert Douglass