[development] Summer of Code Idea

[Anthony Bryan]

> I've become very interested in participating in the upcoming summer of
> code for Drupal and I have an idea about what I'd like to do with my
> proposal.  I was just wondering what you all thought.
>
> I was thinking of making a system that automatically mirrored files,
> scanned folders and otherwise made downloads available through the use of
> metalinks (http://www.metalinker.org/).  So for instance, upload a file
> with Drupal and then that file is automatically sent to mirrors and a
> proper metalink file created (or a download page, I suppose there could be
> a number of preferences here).
>
> So I was just wondering what everyone thought.  Any criticisms or
> expansions to this idea?  I'm aware that the isn't a full fledged proposal
> yet, but that is what I'm trying to get to eventually.  Thanks for any
> input.

Hi Kyle,

(Hope I'm not butting in here but) thanks for the interest in
metalink. I think your idea would be interesting! We use Drupal on
http://metalink.packages.ro/ , but the metalink generation is done by
a perl script and something automatic and integrated with Drupal would
be great and a lot easier for most people!

A few things, I don't know if they're possible or how well they would
fit in with Drupal: It would be nice to be able to scan a directory
for new files and have .metalinks created, even if they were not
uploaded within Drupal. A lot of the projects that use metalinks use
them for large files like ISOs, so those are probably uploaded with
ftp or synced from mirrors.

Also, there are no tools for uploading or repairing integrity with
metalinks. For instance, if you want to upload a 4 gig ISO you could
create a .metalink before hand listing the full file checksums and
chunk checksums. If you upload the file, and there's an error in
transfer, the chunk checksums could be used to scan the file and you
could just upload the corrected chunk (you'd need to create this, or
work with another tool I guess). Likewise, if there's corruption or an
error in the file later, it could be automatically repaired from a
small chunk from the other mirrors.

> I don't know metalinker.org (and I don't have internet access right
> now) but is metalinker.org's goal to distribute the bandwidth and to
> help guarantee availability (i.e. distributed file cache).  It sounds
> like a fun project, but I'm not sure it has enough 'meat' to keep you
> busy for 2 full months.  Plus, it might be too vendor-specific.

Hi Dries, yes that's basically the goal! Guaranteeing availability and
integrity. It's done with an XML format supported by 11 download
programs (mostly open source but some proprietary) and used by
projects like OpenOffice.org, cURL, openSUSE, and other distributions.
They mainly list mirrors, p2p source, checksums, and other file
information. It's not specific to any vendor. I wouldn't know how much
meat the proposal has, but I'd be willing to work with anyone
interested.

more info:

http://en.wikipedia.org/wiki/Metalink
http://www.geospatialsemanticweb.com/2007/02/25/metalink-unifies-internet-downloads
http://slashdot.org/article.pl?sid=07/02/25/144209
http://www.freesoftwaremagazine.com/articles/using_metalinks
http://www.osresources.com/3_18_en.html

-- 
(( Anthony Bryan
  )) Metalink [ http://www.metalinker.org ]