Scalable Internal Activity Stream Module
Greetings Drupalistas, I'm working on what feels like the hundredth (but is really the 4th or 5th) project that includes some variety of Facebook-like "Activity Stream" for a Drupal-based community. Having tackled this problem in a number of different ways in the past couple years -- none of which I've really ever loved -- I'm tempted to launch a new module project to solve this thing once and for all. My primary concerns are modularity -- such that anything can potentially be an Activity -- and scalability to work with 100s of 1000s of actions and users. After some initial review, I found both this existing module: http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/activitystream/ However, this seems like a different concept since A) it's focused on external sources and B) it's based on turning each activity into a node. Basically, it seems better suited for an individual site aggregating internet-wide activity than a community reporting on itself. On the upside, this module (and many others, e.g. userpoints) show a nice way to handle things modularly. I'm not too worried about that, really. But I am worried about scaling, and the architecture of activitystream got me thinking about whether or not the activity-as- node architecture was workable or not. I'd love opinions here. IN FAVOR: Nodes are functional. Facebook already lets you comment on every little thing that goes on. This is fun! It would be good for this module to do that too. It also makes future integration with notification/messaging updates possible, as well as every other wonderful thing nodes can do. AGAINST: This will mean huge amounts of nodes, bloating the table. It also means more overhead when logging activity (node_save vs a single optimized db_query). I'm also skeptical that the core node table structure has the right stuff to be queried with maximum efficiency (e.g. nothing to group similar queries by unless I make a ton of node types, etc). There's also the question of unwanted node functions. We don't really want anyone to edit activities. We also don't want them to start showing up in search queries. I could see a possible solution in maintaining an optimized index table for queries, as well as nodes for functionality, individual page views, etc. The bloat problem could conceivably be solved by giving activity nodes (and index entries) a maximum TTL ala watchdog and other big tables. I've already got a table/query design down for indexing that seems to scale very well to 200k activity entries grouped over 20 types and 5000 users. I suppose the next step is to do some testing around what the overall effects are of having short-lived nodes, and whether or not the other edge cases can be solved. I'm wondering if anyone has done any of their own thinking along these lines and has any comments to add. Happy New Year! -josh ------------------------------------------ Josh Koenig, Partner, CTO http://www.chapterthree.com AOL IM: chap3josh 1-888-496-3238
Read this (very) long discussion on g.d.o on activity streams: http://groups.drupal.org/node/15088 Heartbeat is the module that's resulted from that discussion and looks promising. http://drupal.org/project/heartbeat I'd love to hear more about your thoughts on this subject as I'm about to implement Heartbeat on a site I'm working on. Kyle Research Assistant eBusiness Center @ BYU kyle.mathews2000.com/blog On Wed, Dec 31, 2008 at 12:08 PM, Daniel F. Kudwien <news@unleashedmind.com>wrote:
Greetings Drupalistas,
Happy New Year! -josh
Happy New Year for you, too! *g*
Daniel
Excellent! I knew this had to have already happened. :) I'll dig into this and at least post a link here if I get into the discussion, etc. If it all looks good I may bang out a 5.x backport. cheers -j
Read this (very) long discussion on g.d.o on activity streams: http://groups.drupal.org/node/15088
Heartbeat is the module that's resulted from that discussion and looks promising. http://drupal.org/project/heartbeat
I'd love to hear more about your thoughts on this subject as I'm about to implement Heartbeat on a site I'm working on.
Kyle
Research Assistant eBusiness Center @ BYU kyle.mathews2000.com/blog
On Wed, Dec 31, 2008 at 12:08 PM, Daniel F. Kudwien <news@unleashedmind.com
wrote: Greetings Drupalistas,
Happy New Year! -josh
Happy New Year for you, too! *g*
Daniel
------------------------------------------ Josh Koenig, Partner, CTO http://www.chapterthree.com AOL IM: chap3josh 1-888-496-3238
That's what I love about Drupal -- every time I think to write some code, I turn around and someone's already done it for me. :) Kyle Research Assistant eBusiness Center @ BYU kyle.mathews2000.com/blog On Wed, Dec 31, 2008 at 12:22 PM, Josh Koenig <josh@chapterthree.com> wrote:
Excellent!
I knew this had to have already happened. :)
I'll dig into this and at least post a link here if I get into the discussion, etc. If it all looks good I may bang out a 5.x backport.
cheers -j
Read this (very) long discussion on g.d.o on activity streams:
http://groups.drupal.org/node/15088
Heartbeat is the module that's resulted from that discussion and looks promising. http://drupal.org/project/heartbeat
I'd love to hear more about your thoughts on this subject as I'm about to implement Heartbeat on a site I'm working on.
Kyle
Research Assistant eBusiness Center @ BYU kyle.mathews2000.com/blog
On Wed, Dec 31, 2008 at 12:08 PM, Daniel F. Kudwien < news@unleashedmind.com> wrote:
Greetings Drupalistas,
Happy New Year! -josh
Happy New Year for you, too! *g*
Daniel
------------------------------------------ Josh Koenig, Partner, CTO http://www.chapterthree.com AOL IM: chap3josh 1-888-496-3238
Heh, sometimes that works. Sometimes is just means there's some code. ;) Heartbeat seems interesting, though it's dependence on Rules makes it hard to backport to Drupal 5 (which I'll need) and it's got some internal classes that I don't entirely grok yet. It also doesn't make use of nodes, speaking to my initial question. I think I'll repost to the Social Networking group and see what I can stir up. :) -j
That's what I love about Drupal -- every time I think to write some code, I turn around and someone's already done it for me. :)
Kyle
Research Assistant eBusiness Center @ BYU kyle.mathews2000.com/blog
On Wed, Dec 31, 2008 at 12:22 PM, Josh Koenig <josh@chapterthree.com> wrote:
Excellent!
I knew this had to have already happened. :)
I'll dig into this and at least post a link here if I get into the discussion, etc. If it all looks good I may bang out a 5.x backport.
cheers -j
Read this (very) long discussion on g.d.o on activity streams: http://groups.drupal.org/node/15088
Heartbeat is the module that's resulted from that discussion and looks promising. http://drupal.org/project/heartbeat
I'd love to hear more about your thoughts on this subject as I'm about to implement Heartbeat on a site I'm working on.
Kyle
Research Assistant eBusiness Center @ BYU kyle.mathews2000.com/blog
On Wed, Dec 31, 2008 at 12:08 PM, Daniel F. Kudwien <news@unleashedmind.com
wrote: Greetings Drupalistas,
Happy New Year! -josh
Happy New Year for you, too! *g*
Daniel
------------------------------------------ Josh Koenig, Partner, CTO http://www.chapterthree.com AOL IM: chap3josh 1-888-496-3238
------------------------------------------ Josh Koenig, Partner, CTO http://www.chapterthree.com AOL IM: chap3josh 1-888-496-3238
Activity module was created for just this purpose. http://drupal.org/project/activity It doesn't utilize the node system, however, and therefor you can't comment upon recorded activity, which was part of your request. I also have heard that you can have this sort of functionality with workflow-ng in D5, but again, not sure that you can comment on activity there either. Just throwing some ideas out there for ya. Would love to hear what you eventually decide upon! ~jerad On Wed, Dec 31, 2008 at 2:48 PM, Josh Koenig <josh@chapterthree.com> wrote:
Heh, sometimes that works. Sometimes is just means there's some code. ;)
Heartbeat seems interesting, though it's dependence on Rules makes it hard to backport to Drupal 5 (which I'll need) and it's got some internal classes that I don't entirely grok yet.
It also doesn't make use of nodes, speaking to my initial question.
I think I'll repost to the Social Networking group and see what I can stir up. :)
-j
That's what I love about Drupal -- every time I think to write some code,
I turn around and someone's already done it for me. :)
Kyle
Research Assistant eBusiness Center @ BYU kyle.mathews2000.com/blog
On Wed, Dec 31, 2008 at 12:22 PM, Josh Koenig <josh@chapterthree.com> wrote:
Excellent!
I knew this had to have already happened. :)
I'll dig into this and at least post a link here if I get into the discussion, etc. If it all looks good I may bang out a 5.x backport.
cheers -j
Read this (very) long discussion on g.d.o on activity streams: http://groups.drupal.org/node/15088
Heartbeat is the module that's resulted from that discussion and looks promising. http://drupal.org/project/heartbeat
I'd love to hear more about your thoughts on this subject as I'm about to implement Heartbeat on a site I'm working on.
Kyle
Research Assistant eBusiness Center @ BYU kyle.mathews2000.com/blog
On Wed, Dec 31, 2008 at 12:08 PM, Daniel F. Kudwien < news@unleashedmind.com> wrote:
Greetings Drupalistas,
Happy New Year! -josh
Happy New Year for you, too! *g*
Daniel
------------------------------------------ Josh Koenig, Partner, CTO http://www.chapterthree.com AOL IM: chap3josh 1-888-496-3238
------------------------------------------ Josh Koenig, Partner, CTO http://www.chapterthree.com AOL IM: chap3josh 1-888-496-3238
If this is anything like the Tracker, you'll need to denormalize and index the data to get a scalable solution.
participants (5)
-
Daniel F. Kudwien -
David Timothy Strauss -
Jerad Bitner -
Josh Koenig -
Kyle Mathews