Re: [development] aggregator2 update path complete!
(Half kidding. It would be great if all who work on aggregator stuff unify their efforts on such an API).
Already there. See http://groups.drupal.org/node/3528 (or the original post at http://drupal.org/node/130942). Aron Novak's SoC project also addresses this, turning Aggregator module into an API with standard library for parsing, but opening up a pluggable architecture for other modules to replace/extend. I believe that all the aggregator-type module authors are on board with the new API concept. If you haven't read the proposed API, now is a good time... - Ken Rickard agentrickard
I've taken a look at the proposed API, what exactly is it helping us do? I saw the groups, but the only thing that I understood is that contributing to them would be advocating a need for an aggregation API, which frankly, I don't think is needed at all (I'm open to other opinions though if they're valid and convincing). Would the current 3 or 4 aggregation modules have been justified if they utilized an aggregation API? There is, however, one thing that I did see as very wrong. A module developer can't just throw away his work, create an alternative module, and leave things behind as if they've never been. Creating a module that everyone depends on is an ethical responsibility that needs to be transferred to someone who can manage it when the original author can't work on it anymore. What has been achieved with leech could have also been achieved by updating aggregator2 to become leech (albeit retaining its aggregator2 name) without neglecting its users. I'm trying to provide positive criticism here. Mistakes are done all the time, it's simply human nature. But let's make sure they aren't repeated, shall we? IMHO, we should make it compulsive that every module developer post to the mailing list before setting on a new module. This may prevent the catastrophe of so much repeated modules and repeated code. On 4/29/07, Ken Rickard <agentrickard@gmail.com> wrote:
(Half kidding. It would be great if all who work on aggregator stuff unify their efforts on such an API).
Already there. See http://groups.drupal.org/node/3528 (or the original post at http://drupal.org/node/130942).
Aron Novak's SoC project also addresses this, turning Aggregator module into an API with standard library for parsing, but opening up a pluggable architecture for other modules to replace/extend.
I believe that all the aggregator-type module authors are on board with the new API concept.
If you haven't read the proposed API, now is a good time...
- Ken Rickard agentrickard
[snip] On Mon, Apr 30, 2007 at 01:09:16AM +0300, Ashraf Amayreh wrote:
IMHO, we should make it compulsive that every module developer post to the mailing list before setting on a new module. This may prevent the catastrophe of so much repeated modules and repeated code.
There's a point I want to say. Writing your own module/code is sometimes a lot easier than spending your time to understand the current code. I'm not sure what lead to having all those aggregator modules around. It even becomes worse when you find that most of them are either unmaintained, not working for you or your setup or not ported to the latest drupal. Then you end up writing your own. I had this problem with the stock aggregator module bringing my server to its knees. No time to push changes to core aggregator. The solution ? I have my own 100th aggregator module (Yes, I'm serious) and I'm thinking about releasing it since I know I'll maintain it and keep it up to date. I guess we need to work on the core one to fix it. My 0.02 Euros! -- GPG-Key: 0xA3FD0DF7 - 9F73 032E EAC9 F7AD 951F 280E CB66 8E29 A3FD 0DF7 Debian User and Developer. Homepage: www.foolab.org
Post what it is you you think is lacking in the current aggregation modules and let's all collaborate. My aggregation module would welcome a patch anytime :-p On 4/30/07, Mohammed Sameer <msameer@foolab.org> wrote:
[snip] On Mon, Apr 30, 2007 at 01:09:16AM +0300, Ashraf Amayreh wrote:
IMHO, we should make it compulsive that every module developer post to the mailing list before setting on a new module. This may prevent the catastrophe of so much repeated modules and repeated code.
There's a point I want to say. Writing your own module/code is sometimes a lot easier than spending your time to understand the current code. I'm not sure what lead to having all those aggregator modules around. It even becomes worse when you find that most of them are either unmaintained, not working for you or your setup or not ported to the latest drupal. Then you end up writing your own. I had this problem with the stock aggregator module bringing my server to its knees. No time to push changes to core aggregator. The solution ? I have my own 100th aggregator module (Yes, I'm serious) and I'm thinking about releasing it since I know I'll maintain it and keep it up to date.
I guess we need to work on the core one to fix it.
My 0.02 Euros!
-- GPG-Key: 0xA3FD0DF7 - 9F73 032E EAC9 F7AD 951F 280E CB66 8E29 A3FD 0DF7 Debian User and Developer. Homepage: www.foolab.org
2 main things are causing problems with the aggregation module: 1) I am not sure whether you will be maintaining it or it'll be dead after a while like the rest. 2) You are depending on php5. I'm still running 4 and not planning to move now. We really have a mess and I guess the only way to solve it is a core aggregator rework. I don't have much time for that and I don't think I have much experience. What do you think ? On Mon, Apr 30, 2007 at 03:53:11AM +0300, Ashraf Amayreh wrote:
Post what it is you you think is lacking in the current aggregation modules and let's all collaborate. My aggregation module would welcome a patch anytime :-p
On 4/30/07, Mohammed Sameer <[1]msameer@foolab.org> wrote:
[snip] On Mon, Apr 30, 2007 at 01:09:16AM +0300, Ashraf Amayreh wrote:
IMHO, we should make it compulsive that every module developer post to the mailing list before setting on a new module. This may prevent the catastrophe of so much repeated modules and repeated code. There's a point I want to say. Writing your own module/code is sometimes a lot easier than spending your time to understand the current code. I'm not sure what lead to having all those aggregator modules around. It even becomes worse when you find that most of them are either unmaintained, not working for you or your setup or not ported to the latest drupal. Then you end up writing your own. I had this problem with the stock aggregator module bringing my server to its knees. No time to push changes to core aggregator. The solution ? I have my own 100th aggregator module (Yes, I'm serious) and I'm thinking about releasing it since I know I'll maintain it and keep it up to date. I guess we need to work on the core one to fix it. My 0.02 Euros! -- GPG-Key: 0xA3FD0DF7 - 9F73 032E EAC9 F7AD 951F 280E CB66 8E29 A3FD 0DF7 Debian User and Developer. Homepage: [2]www.foolab.org
References
1. mailto:msameer@foolab.org 2. http://www.foolab.org/
-- GPG-Key: 0xA3FD0DF7 - 9F73 032E EAC9 F7AD 951F 280E CB66 8E29 A3FD 0DF7 Debian User and Developer. Homepage: www.foolab.org
That is the general guideline. But, of course, you're guilty of that as well :P SimpleFeed and others were in progress when you came along.
Was leech and simplefeed announced in the mailing list? I'm quite new to the lists. If leech was, I'm surprised you agreed to it instead of asking the developer to upgrade his aggregator2 module. As to simplefeed, I really don't know too much about its history to say weather its development was warranted or not, maybe it was, maybe not. I will say however that I'm guilty to the core with that in lucid_menu, should have announced it first. I'm quite over the curve where I think I'm always right, just right 99.999%of the time (i wish!) :-P
No, hence why all the other folks are joining forces to work on seeing if there can be some core upgrades. If you don't want to help with core, that's fine.
Hey, don't get me wrong man. What I'm saying is that I need to understand the need for an API, not that I don't want to help or anything. If the idea is sound then you can count me in as long as I have the time for it. I'm well aware that there was much effort spent towards this end as well as an SoC, I'm not really in the business of undermining people's efforts. But I can already detect some flaws here, if you simply allow overriding then developers could override everything and you'll be left with the same amount of code to find security issues in and so forth. Am I correct? I just don't want us to spend so much effort in something that we may find was useless (or worse) later on. What I propose is to simply pick a module that would replace the core one, create update paths from all other modules to it, and then stop all aggregation module development. The one that should be picked is the one most well designed. That's it. Is anyone else with me on this? We could request that all aggregation module developers who think their modules are candidate to step up and nominate them, then we would all get in and discuss this until we agree on one.
I am not sure whether you will be maintaining it or it'll be dead after a while like the rest.
It will not, unless I happen to get infected by the RDS (Remote Death Syndrome lol), then you'll have to pardon me :-P. Other reason is if I don't have enough money to maintain my net connection, which in light of my current conditions, is likely lol, but hey, what did they create non-encrypted wireless Internet connections for? lol
You are depending on php5. I'm still running 4 and not planning to move now.
My use of php5 functions is limited to three locations. Using simpleXML (search the net and you'll see that backports to php4 are available), pretty few try catch statements, which are also replaceable, and some minimal functions like file_put_contents and the like, they can be rewritten or (was it PEAR?)'s compatibility layer could be used here. As a matter of fact, now that I think about it, maybe I'll start a port to PHP 4 soon. Seems simple enough. Really looking forward to everyone's views on this. Had to scratch my head a bit to come up with what I think is an alternative plan. As always, feel free to bash at my suggestions as much as you like, that's why I'm writing them ;-) On 4/30/07, Mohammed Sameer <msameer@foolab.org> wrote:
2 main things are causing problems with the aggregation module: 1) I am not sure whether you will be maintaining it or it'll be dead after a while like the rest. 2) You are depending on php5. I'm still running 4 and not planning to move now.
We really have a mess and I guess the only way to solve it is a core aggregator rework. I don't have much time for that and I don't think I have much experience.
What do you think ?
On Mon, Apr 30, 2007 at 03:53:11AM +0300, Ashraf Amayreh wrote:
Post what it is you you think is lacking in the current aggregation
modules
and let's all collaborate. My aggregation module would welcome a patch anytime :-p
On 4/30/07, Mohammed Sameer <[1]msameer@ foolab.org> wrote:
[snip] On Mon, Apr 30, 2007 at 01:09:16AM +0300, Ashraf Amayreh wrote:
IMHO, we should make it compulsive that every module developer post to the mailing list before setting on a new module. This may prevent the catastrophe of so much repeated modules and repeated code. There's a point I want to say. Writing your own module/code is sometimes a lot easier than spending your time to understand the current code. I'm not sure what lead to having all those aggregator modules around. It even becomes worse when you find that most of them are either unmaintained, not working for you or your setup or not ported to the latest drupal. Then you end up writing your own. I had this problem with the stock aggregator module bringing my server to its knees. No time to push changes to core aggregator. The solution ? I have my own 100th aggregator module (Yes, I'm serious) and I'm thinking about releasing it since I know I'll maintain it and keep it up to date. I guess we need to work on the core one to fix it. My 0.02 Euros! -- GPG-Key: 0xA3FD0DF7 - 9F73 032E EAC9 F7AD 951F 280E CB66 8E29 A3FD 0DF7
Debian User and Developer. Homepage: [2]www.foolab.org
References
1. mailto:msameer@foolab.org 2. http://www.foolab.org/
-- GPG-Key: 0xA3FD0DF7 - 9F73 032E EAC9 F7AD 951F 280E CB66 8E29 A3FD 0DF7 Debian User and Developer. Homepage: www.foolab.org
On 30 Apr 2007, at 02:53, Ashraf Amayreh wrote:
Post what it is you you think is lacking in the current aggregation modules and let's all collaborate. My aggregation module would welcome a patch anytime :-p
My aggregator module too ... ;-) Personally, I think we need to fix this in core but hey, we've been saying that for 2 years now, and so far, I've seen very few aggregator module patches. Boris did a good job outlining some of the work it takes, it's just a matter of implementing some of these proposals. Anyone willing to work on this? We can do this one patch at a time so it not particularly complicated or time-consuming ... -- Dries Buytaert :: http://www.buytaert.net/
On 5/1/07, Dries Buytaert <dries.buytaert@gmail.com> wrote:
My aggregator module too ... ;-) Personally, I think we need to fix this in core but hey, we've been saying that for 2 years now, and so far, I've seen very few aggregator module patches. Boris did a good job outlining some of the work it takes, it's just a matter of implementing some of these proposals. Anyone willing to work on this? We can do this one patch at a time so it not particularly complicated or time-consuming ...
I wanted to point out that Bill from Achieve Internet updated a bunch and there are three ready for review already, including fixing PHP notices, the return (!!) of "blog this", and entity / tag stripping of titles. See http://groups.drupal.org/node/3538 for a few more to look at. All I'm doing is some bug gardening on the issue queue: feel free to do some review of issues (I've got some favorites in there from '05!) and then summarizing on the wiki. Feel free to jump in, it's a good way to power level your core commit points :P -- Boris Mann Office 604-682-2889 Skype borismann http://www.bryght.com
My module's architecture is already functioning as an API. Upon installation, it creates a vocabulary that contains terms such as (RSS20, RDF10, ATOM10). These terms are attached to the feed node type. When a user creates a feed, he has to specify one of these terms to go with the feed. He can create extra terms (this is where the expandability comes in!). Now that he created his feed. He needs to create a file associated with the term he chose, the file, such as RSS20.inc will be passed a simpleXML object and the feed object. My module handles reading the URL string and loading it into a simpleXML object, then passing it to the feed handler, in this case, RSS20.inc, the feed handler loops over the items and extracts specific fields such as title, body, time, image_url. This is where the XML specific extraction occurs, then it calls another (API?) function inside my module called "add item" and passes all of the extracted data to it. Among the passed data it can specify an unlimited number of (extracted) terms and even new (extracted) vocabularies to group this item under. If the term already exists, it will go under it, else, it will create it and go under it (auto-taxonomy). My API function can also be passed one image URL per item, automatically extract it, insert it as an image node, and attach it to the item node. What if we have a custom XML formatted URL to parse? Some XML format called XYZ? We create a new term called XYZ, add it to my aggregation vocabulary, add a file in the feed_handler sub folder called XYZ.inc, my module retrieves the URL and loads it in a simpleXML object, passes it to XYZ.inc, which does the special extraction, and passes the results to my "add item" function which creates a node, and an image node if a URL was passed in by XML.inc, and wala, that's how easy it is to expand it. Nil code duplication, but I'm sure there are infinite ways to extend it. This provides the ability to parse any new formats using what I hope is a simple API. And is in need of everyone's help to make perfect. As a beginning, we can start by removing the PHP5 specific code (or to provide conditional alternatives to be precise?). My API/module is composed of the following methods : function aggregation_get_URL($url, $username = NULL, $password = NULL, $feed = NULL, $feed_etag = NULL, $feed_last_modified = NULL) function aggregation_get_XML($string, $feed = NULL) function _aggregation_<term-name>_parse($feed_XML, $feed) ** (present in file <term_name>.inc) making it a clean add-on. I thought of making this a new hook_aggregation_term_name or something??? Just some thoughts... ** function _aggregation_add_item($title, $body, $teaser, $original_author, $feed, $additional_taxonomies, $timestamp = NULL, $original_item_url = NULL, $guid = NULL, $image_array = NULL); What does the user who wishes to extend this module do? He adds one term and one file that holds one function. The only code he writes is specific format extraction code. Now to one important last issue, I would be extremely grateful for a sponsor to my module, weather it goes to core or not. I have 6 wonderful feature requests that make my fingers tingle just thinking about implementing them. But the times on my side, they are a changin', and not to the better I'm afraid. Regardless of weather my module is sponsored or not, I'm telling everyone who uses it that they should rest assured I will keep it in tip-top shape. It's become so solid that I've not received a bug-report for nearly two weeks, but rather 6 feature requests. Two of which are BIG. I've received a number of thank yous which has made this well worth the effort. If someone can help me by sponsoring it so I could actively continue development I would be extremely grateful. On 5/1/07, Boris Mann <boris@bryght.com> wrote:
On 5/1/07, Dries Buytaert <dries.buytaert@gmail.com> wrote:
My aggregator module too ... ;-) Personally, I think we need to fix this in core but hey, we've been saying that for 2 years now, and so far, I've seen very few aggregator module patches. Boris did a good job outlining some of the work it takes, it's just a matter of implementing some of these proposals. Anyone willing to work on this? We can do this one patch at a time so it not particularly complicated or time-consuming ...
I wanted to point out that Bill from Achieve Internet updated a bunch and there are three ready for review already, including fixing PHP notices, the return (!!) of "blog this", and entity / tag stripping of titles. See http://groups.drupal.org/node/3538 for a few more to look at.
All I'm doing is some bug gardening on the issue queue: feel free to do some review of issues (I've got some favorites in there from '05!) and then summarizing on the wiki.
Feel free to jump in, it's a good way to power level your core commit points :P
-- Boris Mann Office 604-682-2889 Skype borismann http://www.bryght.com
On 5/1/07, Ashraf Amayreh <mistknight@gmail.com> wrote:
My module's architecture is already functioning as an API. <snip>
That's great...would be good to see this activity happen on groups.drupal.org and work with all the people interested in aggregation to get this into core. No, this does not mean copy/pasting your module into core until you can start by extending what's already there. Hope that makes sense. Thanks, -- Boris Mann Office 604-682-2889 Skype borismann http://www.bryght.com
On 4/29/07, Ashraf Amayreh <mistknight@gmail.com> wrote:
I've taken a look at the proposed API, what exactly is it helping us do? I saw the groups, but the only thing that I understood is that contributing to them would be advocating a need for an aggregation API, which frankly, I don't think is needed at all (I'm open to other opinions though if they're valid and convincing). Would the current 3 or 4 aggregation modules have been justified if they utilized an aggregation API?
No, hence why all the other folks are joining forces to work on seeing if there can be some core upgrades. If you don't want to help with core, that's fine.
IMHO, we should make it compulsive that every module developer post to the mailing list before setting on a new module. This may prevent the catastrophe of so much repeated modules and repeated code.
That is the general guideline. But, of course, you're guilty of that as well :P SimpleFeed and others were in progress when you came along. -- Boris Mann Office 604-682-2889 Skype borismann http://www.bryght.com
Sunday, April 29, 2007, 6:09:16 PM, you wrote:
I've taken a look at the proposed API, what exactly is it helping us do? I saw the groups, but the only thing that I understood is that contributing to them would be advocating a need for an aggregation API, which frankly, I don't think is needed at all (I'm open to other opinions though if they're valid and convincing). Would the current 3 or 4 aggregation modules have been justified if they utilized an aggregation API?
I guess most people started a new aggregator module because they needed one or two extra features, but the existing core aggregator was too hard to extend or didn't convince as a good starting ground for new development. The new aggregation solution should address that: It should provide some clean and useful basic functionality and be very extensible through an API. I think all of us aggegrator developers should be able to discontinue their modules by Drupal7 and implement specific functionality as leaner add ons for a common solution.
There is, however, one thing that I did see as very wrong. A module developer can't just throw away his work, create an alternative module, and leave things behind as if they've never been. Creating a module that everyone depends on is an ethical responsibility that needs to be transferred to someone who can manage it when the original author can't work on it anymore. What has been achieved with leech could have also been achieved by updating aggregator2 to become leech (albeit retaining its aggregator2 name) without neglecting its users. I'm trying to provide positive criticism here. Mistakes are done all the time, it's simply human nature. But let's make sure they aren't repeated, shall we?
IMHO, we should make it compulsive that every module developer post to the mailing list before setting on a new module. This may prevent the catastrophe of so much repeated modules and repeated code.
On 4/29/07, Ken Rickard <agentrickard@gmail.com> wrote: >> (Half kidding. It would be great if all who work on aggregator stuff
unify their efforts on such an API).
Already there. See http://groups.drupal.org/node/3528 (or the original post at http://drupal.org/node/130942).
Aron Novak's SoC project also addresses this, turning Aggregator module into an API with standard library for parsing, but opening up a pluggable architecture for other modules to replace/extend.
I believe that all the aggregator-type module authors are on board with the new API concept.
If you haven't read the proposed API, now is a good time...
- Ken Rickard agentrickard
-- Alexander Barth Development Seed http://www.developmentseed.org http://www.developmentseed.org/blog lx_barth(skype) alex_b(drupal.org) Tel. 202.250.3633 Fax. 806.214.6218
participants (6)
-
Alexander Barth -
Ashraf Amayreh -
Boris Mann -
Dries Buytaert -
Ken Rickard -
Mohammed Sameer