[drupal-devel] [feature] Separate mail backend
killes
drupal-devel at drupal.org
Tue Sep 20 15:04:48 UTC 2005
Issue status update for
http://drupal.org/node/28604
Post a follow up:
http://drupal.org/project/comments/add/28604
Project: Drupal
Version: cvs
Component: base system
Category: feature requests
Priority: normal
-Assigned to: Anonymous
+Assigned to: killes at www.drop.org
Reported by: Dries
Updated by: killes at www.drop.org
Status: patch (code needs review)
@vwx: How did you send the mails? direcly or through the queue? can you
try to queue the messages and use the throttle setting?
Also, I believe we do neeed aid. There might be the same address twice
in the same batch of mails.
How the code should be split up depends on Dries, I guess. Personally
I'd put the module functions into system.module and the rest into a
small .inc that is lazily loaded like the .incs for xmlrpc. I don't
think we need a way to look at the queue in core. Do you eve look at
sendmail's mailqueue? I don't.
@noid: this issue is not about simplenews, but we do have a throttle
setting.
killes at www.drop.org
Previous comments:
------------------------------------------------------------------------
Wed, 10 Aug 2005 12:35:37 +0000 : Dries
It was suggested that we separate the mail backend from the front-end.
We might want to include the mail backend in core as includes/mail.inc.
Then, other modules like massmailer, subscriptions, notify, pm (private
messages), project (project issues), contact, etc. could reuse this
component. The mail.inc file would implement some kind of mail queue
functionality, and modules just add a mail to the mail queue using a
simple /mail API/. In future, the mail backend could deal with bounces
and report back proper status codes. Moreover, mail.inc would be
pluggable so it could be replaced by a more powerful one, or one that
is build specifically for the underlying mail transport (SMTP server).
Furthermore, this would solve issues with how many mails get send
within a specified interval. Like, when more than one module sends out
e-mails it is hard to enforce limitations/restrictions imposed by the
hosting company.
I'm posting this here so you can keep this in mind when working on
simplenews, and to solicit for feedback. Chances are we'll setup an
IRC meeting to discuss this further. Details to follow.
------------------------------------------------------------------------
Wed, 10 Aug 2005 16:15:16 +0000 : Amazon
A good place to start would be to look at existing Mail API's
Pear Mail: http://pear.php.net/package/Mail
Pear Mail tutorial:
http://www.zend.com/pear/tutorials/Mail.php#Heading2
------------------------------------------------------------------------
Wed, 10 Aug 2005 20:21:57 +0000 : walkah
um. +1 and then some. I think this is good... and some good references
from Amazon. I'll throw in one other :
http://phpmailer.sf.net/ - has great mime handling (i've run into
issues with PEAR::Mail's mime handilng in the past, but that was around
2 years ago, and things may have gotten better since then).
Oh - and you don't mention it - but support for multipart / mime
messages would be lovely :)
I'd be interested in / happy to sit in on any IRC discussion as well.
------------------------------------------------------------------------
Wed, 10 Aug 2005 21:40:34 +0000 : robertDouglass
phpmailer++
I've used it and like how easy it is to send both text and html mail.
Did I say html mail? You bet! It's time that we had the option.
------------------------------------------------------------------------
Thu, 11 Aug 2005 05:47:01 +0000 : cyberchucktx
I'm definitely interested.
Hopefully by adding a post to this I'll get notified on new postings
(?)
If not I may miss the IRC ..
I've been doing a lot of work with safe_mode PHPLIST (have posted to
the drupal site somewhee, can dig out the reference if anyone is
interested).
Charlie in TX
------------------------------------------------------------------------
Wed, 17 Aug 2005 08:42:34 +0000 : Dries
It was suggested that mail.inc also provides a HTML to text services.
Lots of modules (newsletter, notify, project) try converting HTML to
text. Having a reusable function makes sense, and will lead to better
conversion routines. XSLT anyone?
------------------------------------------------------------------------
Wed, 17 Aug 2005 12:31:27 +0000 : Grugnog2
+1 for phpmailer
I found it very easy to use - and most importantly for me - it supports
SMTP authentication. As many ISP's will not let you send mail without
using SMTP-auth nowadays it may be worth this feature is incorporated
into the mail.inc API.
- Grug
------------------------------------------------------------------------
Wed, 17 Aug 2005 15:08:43 +0000 : Robert Castelo
For about a year now I've been working on an email newsletter and
announcement module [1]. There's a version in my sandbox commited about
2 months ago which works fairly well. Dan Robinson and Varr Willis at
CivicActions, Moshe Weitzman plus a few others have been testing it,
and I've had lots of positive feedback and feature and bug reports from
them.
A few months ago a realised that there's not much point in putting so
much effort into creating all this functionality, only for it to be
locked away in my module. Better to
Better to create discreet component modules which provide a particular
service, such as bounced email handling, but which can be used by any
other module that also needs that service.
For the last two months I've been working to split functionality into
component services modules and make these services available to all
modules.
One of the biggest challenges was to make these component modules
independent of each other. The only area where I haven't managed this
is some of the database calls - but thanks to a chat with chx I
realised that db_rewrite_sql could be used to handle this very nicely.
This is what I have:
* bounced_email.module - process bounced emails
* html2text.module - convert HTML to plain text equivalent (e.g. list
item becomes "* item")
* identity_hash.module - manages full and partial loggin based on hash
which can be used in email links
* publication.module - defines and packages content of publication,
which could be any kind of publication
* schedule.module - defines and manages schedules, e.g. email sending
schedule
* subscribed.module - manages subscriptions to publications
* templates.module - manage and define templates
What's great is that the component modules are not limited to email,
they could be used to quickly create RSS newsletters, PDF newsletters,
text message newsletters, or even personalised/filtered website
sections.
I haven't made the new component modules available anywhere yet, but
I'll be happy to upload them to contrib and let others get involved.
What would be the best way to commit them - as a single directory? or
each component module in it's own directory?
[1]
http://www.cortextcommunications.com/development/newsletter/features
------------------------------------------------------------------------
Thu, 15 Sep 2005 02:05:18 +0000 : vwX
Attachment: http://drupal.org/files/issues/mailqueue.tar.gz (3.71 KB)
This is a very basic mail queue system I would like to contribute.
There are some changes to other modules that will have to be made such
as changing user.module to use this instead of directly calling the php
mail function and moving mime_header_encoding to mail.inc.
------------------------------------------------------------------------
Thu, 15 Sep 2005 15:25:11 +0000 : killes at www.drop.org
Note: I am moving this to the main project.
VwX: What I see looks very nice for a start. Some polishing is
certainly needed, but I'd like to try this. If I only had the time...
------------------------------------------------------------------------
Fri, 16 Sep 2005 22:10:19 +0000 : killes at www.drop.org
Ok, I've been doing some clean-up on this one and made contact.module
use it. The module and the .inc file work.
I've put the files into my sandbox for now. I plan to work on them over
the weekend (unless Dries tells me they are too late for 4.7).
http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/killes/mail/
But I could use some input:
- the sql file defines a column to store the module that generated the
mail. Do we want this? If yes, we need to fix it.
- Do we need a throttle for only sending x mails per cron run?
- can somebody try the files with php5?
- what else do we need?
Please give input, links and what not.
------------------------------------------------------------------------
Sat, 17 Sep 2005 20:34:04 +0000 : Dries
Some of my comments might be false. If so, sorry. I only looked at it
for 5 minutes:
* I wonder why we need per-module settings. I can't think of concrete
use cases.
* For consistency, mailid needs to be mid.
* I don't see an option to send mails immediately?
* I don't see any "messages per hour" accounting?
* I don't understand why mail.inc should be node system aware?
* mailid = '%s' should be mailid = %d.
* Incorrect use of t(): don't translate user input.
* The coding style needs work; inconsistent variable names,
inconsistent spacing, etc.
* Some PHPDoc is rather cryptic: /Updates table mail queue address
table sets sent flag to 1 for a queued msg./.
------------------------------------------------------------------------
Sun, 18 Sep 2005 04:28:11 +0000 : killes at www.drop.org
I've fixed a number of things. needs testing.
------------------------------------------------------------------------
Mon, 19 Sep 2005 02:28:25 +0000 : killes at www.drop.org
Ok, changed and tested.
Some more changes, taken some concepts from massmailer.module.
Included hacked copies of simplenews and contact module for testing
purposes.
We need to discuss if this should be a module or a part od
system.module.
depending on the outcome we need to put the functions in mail.module
and mail.inc into different files.
Testers welcome.
------------------------------------------------------------------------
Mon, 19 Sep 2005 06:31:04 +0000 : Peter Apockotos
Dries,
Are you talking about something like I request here
http://drupal.org/node/28079 ?
------------------------------------------------------------------------
Mon, 19 Sep 2005 18:15:20 +0000 : m3avrck
Well first off, just wanted to say I'm currently evaluating this and
been sending regular feedback to killes for improvements. So far so
good on PHP5 machines :-)
But in terms of where should this code go, I think mail.module should
be eliminated and those functions should be put in mail.inc which
*should* be seperate from common.inc. This way, as Drupal grows and
more and more modules depend upon mail functions, they will be in their
unqiue spot which can easily be patched and extended instead of
rummaging through the common.inc mess ;-)
As a result, the mail.module settings should be moved to Admin >
Settings with the mail setup there, since the basic Drupal "request new
password" etc framework requires mail functionality and having these
settings there makes the most sense, usability wise.
I think with this setup the code could be organized in the cleanest and
most straightforward method and would be an elegant refinement in mail
handling for 4.7... no new features, just cleaned up and seperate for
other modules to make use of, instead of duplicating code and time.
This also has the benefit of bringing in more patches to the Drupal
core and how it handles mail and making it rock solid.
Just my 2 cents.
------------------------------------------------------------------------
Tue, 20 Sep 2005 03:14:25 +0000 : vwX
Did some testing on Killes changes. Queued ~1000 addresses for one
email. When sending started receiving this message :
"
warning: mail(): Could not execute mail delivery program
'/usr/sbin/sendmail -t -i ' in
/home/webuser/mars/www/drupal/modules/mail.module on line 189.
"
at around message 258.
Looking further seems sendmail got hammered. This is a RH6.2 box
550MHz processor with 256MB memory and sendmail configured to use a
smarthost qmail server. I can share the script to test if anybody
wants to help. Will try a bit more after I do some tuning.
Now about the changes.
I think that the mail_queue_addresses table key structure needs to be
changed. Remove Aid and just use address. Also, I've always used a
primary key structure of parent table primary key + one or more
additional keys for a child table. Originally I had something like
this, but it is now one primary key and an index. I don't think this
is correct and I've been doing database work for 15 odd years.
Also, I agree with what m3avrck says. Orginally when I submitted the
code I had most of the functionality in mail.inc and the module was
primarily for administrative functions. I also functionality to view
what was in the queue and force the sending of messages in the queue.
That has since been removed and I think it should be re-added.
------------------------------------------------------------------------
Tue, 20 Sep 2005 11:39:42 +0000 : noid
Sorry guys for interrupting your discussion, but just wanted to make
sure that batch processing ( that is, to help guys like me whose host
limits the sending of emails to say, 200 per hour) is still not
implemented in Simplenews? From my shallow understanding this is what
you are working on right now? Raised this issue which Dries marked as
duplicate here: http://drupal.org/node/28311 .
Just wanted to make sure that I didn't miss the release of this
feature. Thanks in advance! :)
More information about the drupal-devel
mailing list