[development] Huge Database Operation

[Larry Garfield]

Is this data that must be managed through Drupal directly?  If so, what level 
of flexibility do you need with it?

Is this data coming in from an existing 3rd party source?  If so, why does it 
need to live in Drupal?  Can you get away with leveraging the existing data 
source?

Assuming your number is in fact 70 million records (if you have that many 
resumes I really have to wonder who your recruiter is... <g>), then it really 
hinges on what "skill set" means.  A properly tuned SQL table should handle 
that many records if you add the right indexes and give it enough RAM, but if 
skill set is a very complex concept then that may not work.  If skill set is 
an unpredictable structure, you may be better off looking at a Document DB 
such as MongoDB or Cassandra.

The degree to which you need to manage that data through Drupal rather than 
just search it is also a major factor.  

So "needs more info".  Also be aware that DrupalCon is this week (the big 
Drupal dev conference), so most people on the list will be quite busy this 
week.  You may not get a very rapid reply. :-)

--Larry Garfield

On Sunday, March 06, 2011 4:20:47 pm Austin Einter wrote:
> Hi
> I am looking at storing huge data in database (approximately 70,000,00
> records).
> Each record consists of
> 
> 1. Name
> 2. Contact number
> 3. Resume (may vary from 50KB to 150KB)
> 4. Skill set
> 
> And when a user(having role based permission) wants to search database and
> retreive those user's record having certain skill set, should able to
> perform as quick as possible and show it in a view.
> 
> Please suggest me if any such modules are available or need to be
> developed.
> 
> For front end interface I am looking at webform. Do you have a better
> suggestion please recomond.
> 
> Is there any books/links available , which can guide for efficient
> solution, please let me know.
> 
> Best Regards
> Austin