[drupal-devel] file attachment storage
I'm just tossing this up for discussion, but is it of any concern for scaling that all attachments get dropped in the same directory? Might it be better for them to be divided up by module so book attachments get filed into a book subdirectory of the files directory?
On Monday 27 June 2005 14:41, David Norman wrote:
I'm just tossing this up for discussion, but is it of any concern for scaling that all attachments get dropped in the same directory? Might it be better for them to be divided up by module so book attachments get filed into a book subdirectory of the files directory?
I have looked into this, and it is not a problem. ext2fs needs a special setting if you a real lot of files, but that's all. Every filesystem (which is not FAT...) is happy with something like a million files according to Google.
...and what about the people using FAT? Just tell them to use a real OS for load? Karoly Negyesi wrote:
I have looked into this, and it is not a problem.
ext2fs needs a special setting if you a real lot of files, but that's all. Every filesystem (which is not FAT...) is happy with something like a million files according to Google.
Karoly Negyesi wrote:
On Monday 27 June 2005 14:41, David Norman wrote:
I'm just tossing this up for discussion, but is it of any concern for scaling that all attachments get dropped in the same directory? Might it be better for them to be divided up by module so book attachments get filed into a book subdirectory of the files directory?
I have looked into this, and it is not a problem.
ext2fs needs a special setting if you a real lot of files, but that's all. Every filesystem (which is not FAT...) is happy with something like a million files according to Google.
This can be a problem, even on unix style file systems. The file system itself is fine and dandy, but get over a couple thousand files in one directory and things like 'ls' and 'rm *' and other directory management commands start having problems. At work we have some systems that dump out individual log files per service execution (vendor code not ours) and we where forced to write custom scripts to manage these log directories. I have an alternative file system management module and an equivalent of upload module which uses it that allows the system admin to specify the max number of files in a directory. Look in modules under filemanager and attachment. These modules allow multiple files with the same name as well as allow modules to have both private (permissions enforced with Drupal) and public (no php code involved in the download) concurrently. Also the attachment module does not suffer from simultaneous upload race problems (http://drupal.org/node/11060) nor does it rely on the users logging out to free up space used by uploaded files that where not submitted. Craig
Just for the record, I've long wished that Craig's code had been the code that 'won' the upload.module contest the last time this issue went around. Any chances we could open this debate in earnest with the possible goal of replacing upload.module? (putting on my body armour now) cheers, Robert
I have an alternative file system management module and an equivalent of upload module which uses it that allows the system admin to specify the max number of files in a directory. Look in modules under filemanager and attachment. These modules allow multiple files with the same name as well as allow modules to have both private (permissions enforced with Drupal) and public (no php code involved in the download) concurrently. Also the attachment module does not suffer from simultaneous upload race problems (http://drupal.org/node/11060) nor does it rely on the users logging out to free up space used by uploaded files that where not submitted.
Craig
On Monday 27 June 2005 09:17 am, Robert Douglass wrote:
Just for the record, I've long wished that Craig's code had been the code that 'won' the upload.module contest the last time this issue went around.
I had a lot of interest in Craig's code as well. It allowed modules to declare if they wanted to use public or private files rather than default to whatever the site uses. Matt
I think we will see more and more cases like this: http://drupal.org/node/26090 unfortunately I can't even begin to think about doing the work needed to upgrade our file API based on Craig's work or other ideas. I can only be a cheerleader for this one. Robert Matt Westgate wrote:
On Monday 27 June 2005 09:17 am, Robert Douglass wrote:
Just for the record, I've long wished that Craig's code had been the code that 'won' the upload.module contest the last time this issue went around.
I had a lot of interest in Craig's code as well. It allowed modules to declare if they wanted to use public or private files rather than default to whatever the site uses.
Matt
This is the type of concern I was trying to fish out, if it existed, when I started this topic. The issue of files staying in the files directory that never get submitted to associate to a final post seems important to me, too. Even if some filesystems can handle 1 million files, it doesn't seem like the correct way to do organize. I don't have a 50,000 file problem (yet), but would a patch be welcome on this (if I get the free time)? I'm thinking either each node would have its own directory in files, each module would, or both. I'm not familiar with the famous Craig patch, but is this in line with it? Option A: files/1/file.jpg files/1/another.png files/2/separatenode.gif Option B: files/blog.module/file.jpg files/project.module/patch.diff Option C (the ultimate in eliminating filename collisions): files/blog.module/12/wedding.png files/project.module/14/another.doc Robert Douglass wrote:
I think we will see more and more cases like this:
unfortunately I can't even begin to think about doing the work needed to upgrade our file API based on Craig's work or other ideas. I can only be a cheerleader for this one.
Robert
Matt Westgate wrote:
On Monday 27 June 2005 09:17 am, Robert Douglass wrote:
Just for the record, I've long wished that Craig's code had been the code that 'won' the upload.module contest the last time this issue went around.
I had a lot of interest in Craig's code as well. It allowed modules to declare if they wanted to use public or private files rather than default to whatever the site uses. Matt
The "famous Craig patch" is even better than that; a full functioning module that you can install and use right away to manage your uploads. You could probably also rewrite image module to use it instead of the current file API. http://drupal.org/project/filemanager it works with the attachment module to basically do what the upload module in core currently does. http://drupal.org/node/10245 -Robert
On a short term, would the fileapi patch not solve a lot of these problems? AFAIK with that patch we can do all this filemagik, with one module, instead of letting all that "stuff" over to the module developers (they are amongst the most lazy coders, IMO) Ber Op donderdag 30 juni 2005 19:38, schreef Robert Douglass:
The "famous Craig patch" is even better than that; a full functioning module that you can install and use right away to manage your uploads. You could probably also rewrite image module to use it instead of the current file API.
http://drupal.org/project/filemanager
it works with the attachment module to basically do what the upload module in core currently does.
-Robert Regards, Bèr -- [ Bèr Kessels | Drupal services www.webschuur.com ]
participants (6)
-
Bèr Kessels -
Craig Courtney -
David Norman -
Karoly Negyesi -
Matt Westgate -
Robert Douglass