Hi All, Thanks for all the replies, ideas and support. @Dries I'd like to do anything I can to make sure that this project has enough 'meat'. Even if this means just improving file handling capabilities of Drupal, adding AJAX functionality for uploads, whatever it takes. I do think that there are quite a few variations on automatic mirroring, such as mirroring to servers on a local network over FTP, and then presenting the downloads over HTTP. I think that there are some pretty complicated use cases in here, and I would like to cover all of them. @Anthony I was planning on putting directory scanning functionality in. It would be nice if files could just be uploaded through an HTML form but unfortunately that's somewhat limited. I think it would be possible using the cron functionality that Drupal already has. From that, it may be possible for PHP to possible mirror a file to other FTP locations so that a user only has to upload the file once. I really like the suggestion of having kind of 'reverse metalinks' if you want to call it that. That being said, it sounds pretty hard and sounds like it would mean implementing a metalink client with PHP. Maybe it would be possible to interface with aria2 and have it do all the hard lifting. To everyone, If anyone has any ideas I'd be happy to here them. I'd enjoy doing something that ultimately benefits Drupal the most, or adds some cool new functionality to Drupal. I don't have all the answers so anyone's input is appreciated. Thanks! Kyle Cunningham
I've become very interested in participating in the upcoming summer of code for Drupal and I have an idea about what I'd like to do with my proposal. I was just wondering what you all thought.
I was thinking of making a system that automatically mirrored files, scanned folders and otherwise made downloads available through the use of metalinks (http://www.metalinker.org/). So for instance, upload a file with Drupal and then that file is automatically sent to mirrors and a proper metalink file created (or a download page, I suppose there could be a number of preferences here).
So I was just wondering what everyone thought. Any criticisms or expansions to this idea? I'm aware that the isn't a full fledged proposal yet, but that is what I'm trying to get to eventually. Thanks for any input.
Hi Kyle,
(Hope I'm not butting in here but) thanks for the interest in metalink. I think your idea would be interesting! We use Drupal on http://metalink.packages.ro/ , but the metalink generation is done by a perl script and something automatic and integrated with Drupal would be great and a lot easier for most people!
A few things, I don't know if they're possible or how well they would fit in with Drupal: It would be nice to be able to scan a directory for new files and have .metalinks created, even if they were not uploaded within Drupal. A lot of the projects that use metalinks use them for large files like ISOs, so those are probably uploaded with ftp or synced from mirrors.
Also, there are no tools for uploading or repairing integrity with metalinks. For instance, if you want to upload a 4 gig ISO you could create a .metalink before hand listing the full file checksums and chunk checksums. If you upload the file, and there's an error in transfer, the chunk checksums could be used to scan the file and you could just upload the corrected chunk (you'd need to create this, or work with another tool I guess). Likewise, if there's corruption or an error in the file later, it could be automatically repaired from a small chunk from the other mirrors.
I don't know metalinker.org (and I don't have internet access right now) but is metalinker.org's goal to distribute the bandwidth and to help guarantee availability (i.e. distributed file cache). It sounds like a fun project, but I'm not sure it has enough 'meat' to keep you busy for 2 full months. Plus, it might be too vendor-specific.
Hi Dries, yes that's basically the goal! Guaranteeing availability and integrity. It's done with an XML format supported by 11 download programs (mostly open source but some proprietary) and used by projects like OpenOffice.org, cURL, openSUSE, and other distributions. They mainly list mirrors, p2p source, checksums, and other file information. It's not specific to any vendor. I wouldn't know how much meat the proposal has, but I'd be willing to work with anyone interested.
more info:
http://en.wikipedia.org/wiki/Metalink http://www.geospatialsemanticweb.com/2007/02/25/metalink-unifies-internet-do... http://slashdot.org/article.pl?sid=07/02/25/144209 http://www.freesoftwaremagazine.com/articles/using_metalinks http://www.osresources.com/3_18_en.html
-- (( Anthony Bryan )) Metalink [ http://www.metalinker.org ]