[development] Huge Database Operation

[Kamal Palei]

Thanks Larry.

Please see response inline. I am not an expert, in learning stage, so please
bear with  me if I am asking very basic questions.

Regards,
Austin



On Mon, Mar 7, 2011 at 4:14 AM, Larry Garfield <larry at garfieldtech.com>wrote:

> Is this data that must be managed through Drupal directly?  If so, what
> level
> of flexibility do you need with it?
>
>
[Austin]
I do not mind if Drupal can manage data directly. Here the only concern is
performance. While adding data, it should be faster and more importantly,
while seraching one should be able to search quickly (given huge number of
record).


> Is this data coming in from an existing 3rd party source?  If so, why does
> it
> need to live in Drupal?  Can you get away with leveraging the existing data
> source?
>


[Austin]
Data is not from 3rd party. Over the period of time, I really hope number of
users will increase.


>
> Assuming your number is in fact 70 million records (if you have that many
> resumes I really have to wonder who your recruiter is... <g>), then it
> really
> hinges on what "skill set" means.  A properly tuned SQL table should handle
> that many records if you add the right indexes and give it enough RAM, but
> if
> skill set is a very complex concept then that may not work.  If skill set
> is
> an unpredictable structure, you may be better off looking at a Document DB
> such as MongoDB or Cassandra.
>
>
[Austin]
Skill set is basically comma (",") separated values. Example - PHP, SQL,
Drupal, VoIP, SIP, HTTP etc.
This is the field that will be used during searching of records instead of
using resume just to have improved performance.



> The degree to which you need to manage that data through Drupal rather than
> just search it is also a major factor.
>
> [Austin]
The data addition, delete and search are the major operations.
If you can think any other data operations (which I might be missing) ,
really appreciate if you can share.

Is there any way, where I can compress the resume and store as zip file,
when somebody really wants it unzip and give it.
This may be usefull to save memory to great extent.



> So "needs more info".  Also be aware that DrupalCon is this week (the big
> Drupal dev conference), so most people on the list will be quite busy this
> week.  You may not get a very rapid reply. :-)
>
> --Larry Garfield
>
> On Sunday, March 06, 2011 4:20:47 pm Austin Einter wrote:
> > Hi
> > I am looking at storing huge data in database (approximately 70,000,00
> > records).
> > Each record consists of
> >
> > 1. Name
> > 2. Contact number
> > 3. Resume (may vary from 50KB to 150KB)
> > 4. Skill set
> >
> > And when a user(having role based permission) wants to search database
> and
> > retreive those user's record having certain skill set, should able to
> > perform as quick as possible and show it in a view.
> >
> > Please suggest me if any such modules are available or need to be
> > developed.
> >
> > For front end interface I am looking at webform. Do you have a better
> > suggestion please recomond.
> >
> > Is there any books/links available , which can guide for efficient
> > solution, please let me know.
> >
> > Best Regards
> > Austin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.drupal.org/pipermail/development/attachments/20110307/798d74a2/attachment.html