115,000 records -- nodes or custom
Hi. I was hoping for some advice. I recently took over an existing Drupal site for a nonprofit. The client currently has a searchable database of a 115,000 records written in cold fusion in a separate database on a separate server. They would like me to bring it into their Drupal 6 site or at least onto the same server. I can do this in two ways: 1. As a custom module running off it's own db. 2. Or I can make it a content type and use Views for the interface. My preference is to just use CCK/Views for maximum flexibility -- but I question whether or not it's wise to add a 115k records to the node and other tables and what type of impact this might have on overall site performance. (Any other downsides?) Anyone have any advice/suggestions? If it was your project would you code it separately or use CCK/Views. Thanks, Sam Sam Cohen, Principal New Media Solutions <http://new-media-solutions.com> Drupal Training & Services <http://twitter.com/samcohen>
Let's see, Drupal.org has more than 10 times the nodes plus many more comments. It runs D6 and uses Views. And it is usually fast enough. Nancy Injustice anywhere is a threat to justice everywhere. -- Dr. Martin L. King, Jr.
________________________________ From: Sam Cohen
The client currently has a searchable database of a 115,000 records written in cold fusion in a separate database on a separate server.
They would like me to bring it into their Drupal 6 site or at least onto the same server.
I can do this in two ways: 1. As a custom module running off it's own db. 2. Or I can make it a content type and use Views for the interface.
My preference is to just use CCK/Views for maximum flexibility -- but I question whether or not it's wise to add a 115k records to the node and other tables and what type of impact this might have on overall site performance. (Any other downsides?)
Anyone have any advice/suggestions? If it was your project would you code it separately or use CCK/Views.
Drupal.org is also running on a cluster of servers that have been scaled over a longer period of time by people that know what they're doing. On Mar 8, 2012 8:08 AM, "Ms. Nancy Wichmann" <nan_wich@bellsouth.net> wrote:
Let's see, Drupal.org has more than 10 times the nodes plus many more comments. It runs D6 and uses Views. And it is usually fast enough.
*Nancy*
Injustice anywhere is a threat to justice everywhere. -- Dr. Martin L. King, Jr.
------------------------------ *From:* Sam Cohen
The client currently has a searchable database of a 115,000 records written in cold fusion in a separate database on a separate server.
They would like me to bring it into their Drupal 6 site or at least onto the same server.
I can do this in two ways: 1. As a custom module running off it's own db. 2. Or I can make it a content type and use Views for the interface.
My preference is to just use CCK/Views for maximum flexibility -- but I question whether or not it's wise to add a 115k records to the node and other tables and what type of impact this might have on overall site performance. (Any other downsides?)
Anyone have any advice/suggestions? If it was your project would you code it separately or use CCK/Views.
I run a Drupal 5 site with just under a half million nodes on a single $50 VPS. As long as you don't run any stupid queries, you're fine. That said, you can integrate Views to any arbitrary SQL table. You just need to write the views_data hook for it. There is, or was at least, a module to help automate that process. The deciding factor for node-or-not is not the volume of data, but how it will be maintained. Do you want each of those records to have its own view page, edit page, workflow, etc. within Drupal? Make 'em a node. Is it legacy read only data that will never be updated by a human? Probably not worth the effort to make them nodes. --Larry Garfield On 3/8/12 11:35 AM, Cameron Eagans wrote:
Drupal.org is also running on a cluster of servers that have been scaled over a longer period of time by people that know what they're doing.
On Mar 8, 2012 8:08 AM, "Ms. Nancy Wichmann" <nan_wich@bellsouth.net <mailto:nan_wich@bellsouth.net>> wrote:
Let's see, Drupal.org has more than 10 times the nodes plus many more comments. It runs D6 and uses Views. And it is usually fast enough. /*Nancy*/ Injustice anywhere is a threat to justice everywhere. -- Dr. Martin L. King, Jr.
------------------------------------------------------------------------ *From:* Sam Cohen
The client currently has a searchable database of a 115,000 records written in cold fusion in a separate database on a separate server.
They would like me to bring it into their Drupal 6 site or at least onto the same server.
I can do this in two ways: 1. As a custom module running off it's own db. 2. Or I can make it a content type and use Views for the interface.
My preference is to just use CCK/Views for maximum flexibility -- but I question whether or not it's wise to add a 115k records to the node and other tables and what type of impact this might have on overall site performance. (Any other downsides?)
Anyone have any advice/suggestions? If it was your project would you code it separately or use CCK/Views.
http://drupal.org/project/data is one such module. And yay, it has a D7 upgrade now. I've been waiting for something lightweight with good Feeds synergy to get upgraded. On Thu, Mar 8, 2012 at 10:05 AM, Larry Garfield <larry@garfieldtech.com>wrote:
There is, or was at least, a module to help automate that process.
CCK / Views is definitely a safer bet. For performance, you can go for pressflow - http://pressflow.org/ which is mainly used for enhanced performance. Also, you can use some external caching like Memcache, Varnish to further enhance performance. On Fri, Mar 9, 2012 at 1:51 AM, Adam B. Ross <grayside@gmail.com> wrote:
http://drupal.org/project/data is one such module.
And yay, it has a D7 upgrade now. I've been waiting for something lightweight with good Feeds synergy to get upgraded.
On Thu, Mar 8, 2012 at 10:05 AM, Larry Garfield <larry@garfieldtech.com>wrote:
There is, or was at least, a module to help automate that process.
-- Regards, Vaibhav Jain
Sigh. Methinks Vaibhav Jain should understand caching a little better before making statements like that. @op, go with Larry Garfield's advice. It's a solid plan. -- Cameron Eagans http://cweagans.net On Thu, Mar 8, 2012 at 20:40, Vaibhav Jain <in.vaibhavjain@gmail.com> wrote:
CCK / Views is definitely a safer bet. For performance, you can go for pressflow - http://pressflow.org/ which is mainly used for enhanced performance. Also, you can use some external caching like Memcache, Varnish to further enhance performance.
On Fri, Mar 9, 2012 at 1:51 AM, Adam B. Ross <grayside@gmail.com> wrote:
http://drupal.org/project/data is one such module.
And yay, it has a D7 upgrade now. I've been waiting for something lightweight with good Feeds synergy to get upgraded.
On Thu, Mar 8, 2012 at 10:05 AM, Larry Garfield <larry@garfieldtech.com>wrote:
There is, or was at least, a module to help automate that process.
-- Regards, Vaibhav Jain
On Thu, Mar 8, 2012 at 11:22 PM, Cameron Eagans <cweagans@gmail.com> wrote:
Sigh. Methinks Vaibhav Jain should understand caching a little better before making statements like that. @op, go with Larry Garfield's advice. It's a solid plan.
But not mentioning it will never give the OP a hint to go learn. The issue is complex with many possible solutions. Depending on how the third party table is being updated it may make sense to give a view to the third party table. If his client wants Drupal to update the third party table then several things could take place. If the table is never to be updated then several other things could take place. -- Earnie -- https://sites.google.com/site/earnieboyd
Keep in mind that most of that has less to do with overall performance and almost nothing to do with data size, but is a factor of the load, i.e. traffic that Drupal.org receives. Cameron Eagans wrote:
Drupal.org is also running on a cluster of servers that have been scaled over a longer period of time by people that know what they're doing.
On Mar 8, 2012 8:08 AM, "Ms. Nancy Wichmann" <nan_wich@bellsouth.net <mailto:nan_wich@bellsouth.net>> wrote:
Let's see, Drupal.org has more than 10 times the nodes plus many more comments. It runs D6 and uses Views. And it is usually fast enough.
/*Nancy*/
Injustice anywhere is a threat to justice everywhere. -- Dr. Martin L. King, Jr.
------------------------------------------------------------------------ *From:* Sam Cohen
The client currently has a searchable database of a 115,000 records written in cold fusion in a separate database on a separate server.
They would like me to bring it into their Drupal 6 site or at least onto the same server.
I can do this in two ways: 1. As a custom module running off it's own db. 2. Or I can make it a content type and use Views for the interface.
My preference is to just use CCK/Views for maximum flexibility -- but I question whether or not it's wise to add a 115k records to the node and other tables and what type of impact this might have on overall site performance. (Any other downsides?)
Anyone have any advice/suggestions? If it was your project would you code it separately or use CCK/Views.
-- __________________ Michael Prasuhn 503.512.0822 office mike@mikeyp.net
FWIW, Drupal.org does not make heavy use of CCK (change nodes, and a few other types are some of the first content types built mostly on CCK). Projects and especially the issue queue make use of custom tables, and highly tuned views, using multiple instances of denormalization for performance. Ms. Nancy Wichmann wrote:
Let's see, Drupal.org has more than 10 times the nodes plus many more comments. It runs D6 and uses Views. And it is usually fast enough.
/*Nancy*/
Injustice anywhere is a threat to justice everywhere. -- Dr. Martin L. King, Jr.
------------------------------------------------------------------------ *From:* Sam Cohen
The client currently has a searchable database of a 115,000 records written in cold fusion in a separate database on a separate server.
They would like me to bring it into their Drupal 6 site or at least onto the same server.
I can do this in two ways: 1. As a custom module running off it's own db. 2. Or I can make it a content type and use Views for the interface.
My preference is to just use CCK/Views for maximum flexibility -- but I question whether or not it's wise to add a 115k records to the node and other tables and what type of impact this might have on overall site performance. (Any other downsides?)
Anyone have any advice/suggestions? If it was your project would you code it separately or use CCK/Views.
-- __________________ Michael Prasuhn 503.512.0822 office mike@mikeyp.net
Sam, I would opt for option 2, it's the most flexible in the long run. We have one client with 65.000 nodes (25 fields each) and they are using views + exposed filters to search them, but it's an internal site so only a couple of users, the site is still quick (results are shown in less than a second). If you expect a lot of users searching, you might better look into apacheSolr to increase the speed. Cheers Peter From: development-bounces@drupal.org [mailto:development-bounces@drupal.org] On Behalf Of Sam Cohen Sent: donderdag 8 maart 2012 15:51 To: development@drupal.org Subject: [development] 115,000 records -- nodes or custom Hi. I was hoping for some advice. I recently took over an existing Drupal site for a nonprofit. The client currently has a searchable database of a 115,000 records written in cold fusion in a separate database on a separate server. They would like me to bring it into their Drupal 6 site or at least onto the same server. I can do this in two ways: 1. As a custom module running off it's own db. 2. Or I can make it a content type and use Views for the interface. My preference is to just use CCK/Views for maximum flexibility -- but I question whether or not it's wise to add a 115k records to the node and other tables and what type of impact this might have on overall site performance. (Any other downsides?) Anyone have any advice/suggestions? If it was your project would you code it separately or use CCK/Views. Thanks, Sam Sam Cohen, Principal New Media Solutions<http://new-media-solutions.com> Drupal Training & Services
2nd option definitely. Just do it with caution, test for performance early enough, check for slow queries, etc Drupal can scale, you just have to think and understand what are you doing. Why not D7? JAnez Dne 8. mar. 2012 15:57 je "Sam Cohen" <sam@samcohen.com> napisal/-a:
Hi.
I was hoping for some advice. I recently took over an existing Drupal site for a nonprofit.
The client currently has a searchable database of a 115,000 records written in cold fusion in a separate database on a separate server.
They would like me to bring it into their Drupal 6 site or at least onto the same server.
I can do this in two ways:
1. As a custom module running off it's own db.
2. Or I can make it a content type and use Views for the interface.
My preference is to just use CCK/Views for maximum flexibility -- but I question whether or not it's wise to add a 115k records to the node and other tables and what type of impact this might have on overall site performance. (Any other downsides?)
Anyone have any advice/suggestions? If it was your project would you code it separately or use CCK/Views.
Thanks, Sam
Sam Cohen, Principal New Media Solutions <http://new-media-solutions.com> Drupal Training & Services <http://twitter.com/samcohen>
participants (10)
-
Adam B. Ross -
Cameron Eagans -
Earnie Boyd -
Janez Urevc -
Larry Garfield -
Michael Prasuhn -
Ms. Nancy Wichmann -
Peter Droogmans -
Sam Cohen -
Vaibhav Jain