[development] Summer of Code Idea

Tue Mar 6 22:15:00 UTC 2007

Hi All,
Thanks for all the replies, ideas and support.

@Dries
I'd like to do anything I can to make sure that this project has enough
'meat'.  Even if this means just improving file handling capabilities of
Drupal, adding AJAX functionality for uploads, whatever it takes.  I do
think that there are quite a few variations on automatic mirroring, such
as mirroring to servers on a local network over FTP, and then presenting
the downloads over HTTP.  I think that there are some pretty complicated
use cases in here, and I would like to cover all of them.

@Anthony
I was planning on putting directory scanning functionality in.  It would
be nice if files could just be uploaded through an HTML form but
unfortunately that's somewhat limited.  I think it would be possible using
the cron functionality that Drupal already has.  From that, it may be
possible for PHP to possible mirror a file to other FTP locations so that
a user only has to upload the file once.

I really like the suggestion of having kind of 'reverse metalinks' if you
want to call it that.  That being said, it sounds pretty hard and sounds
like it would mean implementing a metalink client with PHP.  Maybe it
would be possible to interface with aria2 and have it do all the hard
lifting.

To everyone,
If anyone has any ideas I'd be happy to here them.  I'd enjoy doing
something that ultimately benefits Drupal the most, or adds some cool new
functionality to Drupal.  I don't have all the answers so anyone's input
is appreciated.

Thanks!
Kyle Cunningham

>> I've become very interested in participating in the upcoming summer of
>> code for Drupal and I have an idea about what I'd like to do with my
>> proposal.  I was just wondering what you all thought.
>>
>> I was thinking of making a system that automatically mirrored files,
>> scanned folders and otherwise made downloads available through the use
>> of
>> metalinks (http://www.metalinker.org/).  So for instance, upload a file
>> with Drupal and then that file is automatically sent to mirrors and a
>> proper metalink file created (or a download page, I suppose there could
>> be
>> a number of preferences here).
>>
>> So I was just wondering what everyone thought.  Any criticisms or
>> expansions to this idea?  I'm aware that the isn't a full fledged
>> proposal
>> yet, but that is what I'm trying to get to eventually.  Thanks for any
>> input.
>
> Hi Kyle,
>
> (Hope I'm not butting in here but) thanks for the interest in
> metalink. I think your idea would be interesting! We use Drupal on
> http://metalink.packages.ro/ , but the metalink generation is done by
> a perl script and something automatic and integrated with Drupal would
> be great and a lot easier for most people!
>
> A few things, I don't know if they're possible or how well they would
> fit in with Drupal: It would be nice to be able to scan a directory
> for new files and have .metalinks created, even if they were not
> uploaded within Drupal. A lot of the projects that use metalinks use
> them for large files like ISOs, so those are probably uploaded with
> ftp or synced from mirrors.
>
> Also, there are no tools for uploading or repairing integrity with
> metalinks. For instance, if you want to upload a 4 gig ISO you could
> create a .metalink before hand listing the full file checksums and
> chunk checksums. If you upload the file, and there's an error in
> transfer, the chunk checksums could be used to scan the file and you
> could just upload the corrected chunk (you'd need to create this, or
> work with another tool I guess). Likewise, if there's corruption or an
> error in the file later, it could be automatically repaired from a
> small chunk from the other mirrors.
>
>> I don't know metalinker.org (and I don't have internet access right
>> now) but is metalinker.org's goal to distribute the bandwidth and to
>> help guarantee availability (i.e. distributed file cache).  It sounds
>> like a fun project, but I'm not sure it has enough 'meat' to keep you
>> busy for 2 full months.  Plus, it might be too vendor-specific.
>
> Hi Dries, yes that's basically the goal! Guaranteeing availability and
> integrity. It's done with an XML format supported by 11 download
> programs (mostly open source but some proprietary) and used by
> projects like OpenOffice.org, cURL, openSUSE, and other distributions.
> They mainly list mirrors, p2p source, checksums, and other file
> information. It's not specific to any vendor. I wouldn't know how much
> meat the proposal has, but I'd be willing to work with anyone
> interested.
>
> more info:
>
> http://en.wikipedia.org/wiki/Metalink
> http://www.geospatialsemanticweb.com/2007/02/25/metalink-unifies-internet-downloads
> http://slashdot.org/article.pl?sid=07/02/25/144209
> http://www.freesoftwaremagazine.com/articles/using_metalinks
> http://www.osresources.com/3_18_en.html
>
> --
> (( Anthony Bryan
>   )) Metalink [ http://www.metalinker.org ]
>