Issue status update for http://drupal.org/node/28604 Post a follow up: http://drupal.org/project/comments/add/28604 Project: Drupal Version: cvs Component: base system Category: feature requests Priority: normal -Assigned to: Anonymous +Assigned to: killes@www.drop.org Reported by: Dries Updated by: killes@www.drop.org Status: patch (code needs review) @vwx: How did you send the mails? direcly or through the queue? can you try to queue the messages and use the throttle setting? Also, I believe we do neeed aid. There might be the same address twice in the same batch of mails. How the code should be split up depends on Dries, I guess. Personally I'd put the module functions into system.module and the rest into a small .inc that is lazily loaded like the .incs for xmlrpc. I don't think we need a way to look at the queue in core. Do you eve look at sendmail's mailqueue? I don't. @noid: this issue is not about simplenews, but we do have a throttle setting. killes@www.drop.org Previous comments: ------------------------------------------------------------------------ Wed, 10 Aug 2005 12:35:37 +0000 : Dries It was suggested that we separate the mail backend from the front-end. We might want to include the mail backend in core as includes/mail.inc. Then, other modules like massmailer, subscriptions, notify, pm (private messages), project (project issues), contact, etc. could reuse this component. The mail.inc file would implement some kind of mail queue functionality, and modules just add a mail to the mail queue using a simple /mail API/. In future, the mail backend could deal with bounces and report back proper status codes. Moreover, mail.inc would be pluggable so it could be replaced by a more powerful one, or one that is build specifically for the underlying mail transport (SMTP server). Furthermore, this would solve issues with how many mails get send within a specified interval. Like, when more than one module sends out e-mails it is hard to enforce limitations/restrictions imposed by the hosting company. I'm posting this here so you can keep this in mind when working on simplenews, and to solicit for feedback. Chances are we'll setup an IRC meeting to discuss this further. Details to follow. ------------------------------------------------------------------------ Wed, 10 Aug 2005 16:15:16 +0000 : Amazon A good place to start would be to look at existing Mail API's Pear Mail: http://pear.php.net/package/Mail Pear Mail tutorial: http://www.zend.com/pear/tutorials/Mail.php#Heading2 ------------------------------------------------------------------------ Wed, 10 Aug 2005 20:21:57 +0000 : walkah um. +1 and then some. I think this is good... and some good references from Amazon. I'll throw in one other : http://phpmailer.sf.net/ - has great mime handling (i've run into issues with PEAR::Mail's mime handilng in the past, but that was around 2 years ago, and things may have gotten better since then). Oh - and you don't mention it - but support for multipart / mime messages would be lovely :) I'd be interested in / happy to sit in on any IRC discussion as well. ------------------------------------------------------------------------ Wed, 10 Aug 2005 21:40:34 +0000 : robertDouglass phpmailer++ I've used it and like how easy it is to send both text and html mail. Did I say html mail? You bet! It's time that we had the option. ------------------------------------------------------------------------ Thu, 11 Aug 2005 05:47:01 +0000 : cyberchucktx I'm definitely interested. Hopefully by adding a post to this I'll get notified on new postings (?) If not I may miss the IRC .. I've been doing a lot of work with safe_mode PHPLIST (have posted to the drupal site somewhee, can dig out the reference if anyone is interested). Charlie in TX ------------------------------------------------------------------------ Wed, 17 Aug 2005 08:42:34 +0000 : Dries It was suggested that mail.inc also provides a HTML to text services. Lots of modules (newsletter, notify, project) try converting HTML to text. Having a reusable function makes sense, and will lead to better conversion routines. XSLT anyone? ------------------------------------------------------------------------ Wed, 17 Aug 2005 12:31:27 +0000 : Grugnog2 +1 for phpmailer I found it very easy to use - and most importantly for me - it supports SMTP authentication. As many ISP's will not let you send mail without using SMTP-auth nowadays it may be worth this feature is incorporated into the mail.inc API. - Grug ------------------------------------------------------------------------ Wed, 17 Aug 2005 15:08:43 +0000 : Robert Castelo For about a year now I've been working on an email newsletter and announcement module [1]. There's a version in my sandbox commited about 2 months ago which works fairly well. Dan Robinson and Varr Willis at CivicActions, Moshe Weitzman plus a few others have been testing it, and I've had lots of positive feedback and feature and bug reports from them. A few months ago a realised that there's not much point in putting so much effort into creating all this functionality, only for it to be locked away in my module. Better to Better to create discreet component modules which provide a particular service, such as bounced email handling, but which can be used by any other module that also needs that service. For the last two months I've been working to split functionality into component services modules and make these services available to all modules. One of the biggest challenges was to make these component modules independent of each other. The only area where I haven't managed this is some of the database calls - but thanks to a chat with chx I realised that db_rewrite_sql could be used to handle this very nicely. This is what I have: * bounced_email.module - process bounced emails * html2text.module - convert HTML to plain text equivalent (e.g. list item becomes "* item") * identity_hash.module - manages full and partial loggin based on hash which can be used in email links * publication.module - defines and packages content of publication, which could be any kind of publication * schedule.module - defines and manages schedules, e.g. email sending schedule * subscribed.module - manages subscriptions to publications * templates.module - manage and define templates What's great is that the component modules are not limited to email, they could be used to quickly create RSS newsletters, PDF newsletters, text message newsletters, or even personalised/filtered website sections. I haven't made the new component modules available anywhere yet, but I'll be happy to upload them to contrib and let others get involved. What would be the best way to commit them - as a single directory? or each component module in it's own directory? [1] http://www.cortextcommunications.com/development/newsletter/features ------------------------------------------------------------------------ Thu, 15 Sep 2005 02:05:18 +0000 : vwX Attachment: http://drupal.org/files/issues/mailqueue.tar.gz (3.71 KB) This is a very basic mail queue system I would like to contribute. There are some changes to other modules that will have to be made such as changing user.module to use this instead of directly calling the php mail function and moving mime_header_encoding to mail.inc. ------------------------------------------------------------------------ Thu, 15 Sep 2005 15:25:11 +0000 : killes@www.drop.org Note: I am moving this to the main project. VwX: What I see looks very nice for a start. Some polishing is certainly needed, but I'd like to try this. If I only had the time... ------------------------------------------------------------------------ Fri, 16 Sep 2005 22:10:19 +0000 : killes@www.drop.org Ok, I've been doing some clean-up on this one and made contact.module use it. The module and the .inc file work. I've put the files into my sandbox for now. I plan to work on them over the weekend (unless Dries tells me they are too late for 4.7). http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/killes/mail/ But I could use some input: - the sql file defines a column to store the module that generated the mail. Do we want this? If yes, we need to fix it. - Do we need a throttle for only sending x mails per cron run? - can somebody try the files with php5? - what else do we need? Please give input, links and what not. ------------------------------------------------------------------------ Sat, 17 Sep 2005 20:34:04 +0000 : Dries Some of my comments might be false. If so, sorry. I only looked at it for 5 minutes: * I wonder why we need per-module settings. I can't think of concrete use cases. * For consistency, mailid needs to be mid. * I don't see an option to send mails immediately? * I don't see any "messages per hour" accounting? * I don't understand why mail.inc should be node system aware? * mailid = '%s' should be mailid = %d. * Incorrect use of t(): don't translate user input. * The coding style needs work; inconsistent variable names, inconsistent spacing, etc. * Some PHPDoc is rather cryptic: /Updates table mail queue address table sets sent flag to 1 for a queued msg./. ------------------------------------------------------------------------ Sun, 18 Sep 2005 04:28:11 +0000 : killes@www.drop.org I've fixed a number of things. needs testing. ------------------------------------------------------------------------ Mon, 19 Sep 2005 02:28:25 +0000 : killes@www.drop.org Ok, changed and tested. Some more changes, taken some concepts from massmailer.module. Included hacked copies of simplenews and contact module for testing purposes. We need to discuss if this should be a module or a part od system.module. depending on the outcome we need to put the functions in mail.module and mail.inc into different files. Testers welcome. ------------------------------------------------------------------------ Mon, 19 Sep 2005 06:31:04 +0000 : Peter Apockotos Dries, Are you talking about something like I request here http://drupal.org/node/28079 ? ------------------------------------------------------------------------ Mon, 19 Sep 2005 18:15:20 +0000 : m3avrck Well first off, just wanted to say I'm currently evaluating this and been sending regular feedback to killes for improvements. So far so good on PHP5 machines :-) But in terms of where should this code go, I think mail.module should be eliminated and those functions should be put in mail.inc which *should* be seperate from common.inc. This way, as Drupal grows and more and more modules depend upon mail functions, they will be in their unqiue spot which can easily be patched and extended instead of rummaging through the common.inc mess ;-) As a result, the mail.module settings should be moved to Admin > Settings with the mail setup there, since the basic Drupal "request new password" etc framework requires mail functionality and having these settings there makes the most sense, usability wise. I think with this setup the code could be organized in the cleanest and most straightforward method and would be an elegant refinement in mail handling for 4.7... no new features, just cleaned up and seperate for other modules to make use of, instead of duplicating code and time. This also has the benefit of bringing in more patches to the Drupal core and how it handles mail and making it rock solid. Just my 2 cents. ------------------------------------------------------------------------ Tue, 20 Sep 2005 03:14:25 +0000 : vwX Did some testing on Killes changes. Queued ~1000 addresses for one email. When sending started receiving this message : " warning: mail(): Could not execute mail delivery program '/usr/sbin/sendmail -t -i ' in /home/webuser/mars/www/drupal/modules/mail.module on line 189. " at around message 258. Looking further seems sendmail got hammered. This is a RH6.2 box 550MHz processor with 256MB memory and sendmail configured to use a smarthost qmail server. I can share the script to test if anybody wants to help. Will try a bit more after I do some tuning. Now about the changes. I think that the mail_queue_addresses table key structure needs to be changed. Remove Aid and just use address. Also, I've always used a primary key structure of parent table primary key + one or more additional keys for a child table. Originally I had something like this, but it is now one primary key and an index. I don't think this is correct and I've been doing database work for 15 odd years. Also, I agree with what m3avrck says. Orginally when I submitted the code I had most of the functionality in mail.inc and the module was primarily for administrative functions. I also functionality to view what was in the queue and force the sending of messages in the queue. That has since been removed and I think it should be re-added. ------------------------------------------------------------------------ Tue, 20 Sep 2005 11:39:42 +0000 : noid Sorry guys for interrupting your discussion, but just wanted to make sure that batch processing ( that is, to help guys like me whose host limits the sending of emails to say, 200 per hour) is still not implemented in Simplenews? From my shallow understanding this is what you are working on right now? Raised this issue which Dries marked as duplicate here: http://drupal.org/node/28311 . Just wanted to make sure that I didn't miss the release of this feature. Thanks in advance! :)