Best way to handle very large downloads and HTTP downloads on the server


I've an upcoming project where I will need to handle very large uploads from browsers (either the classic input type="file" or a Java Applet), and looking for the best tool to do the job on the server.

Theses are the things I need :

  • low memory consumption on the server
  • ability to save the file in its final destination on the server (no copying the file around)
  • no blocking of other important tasks done by the webserver
  • good handling of files up to 2 gb
  • authorization of files (permissions would be given in the app)

I still have some latitude on what technology to use so I would like to have some advice in order to be able to choose the best technology on the server to handle this task :

  • ASP.NET ?
  • Java ?
  • Amazon S3 ?
  • Other choices ?

I'm more used to the Microsoft Stack, but willing to change if necessary : as told above, I'm just looking for the best tool for the job.

Thanks !

Update : The server side is the part I'm really interested in for this question, not the client side.

It looks like it may be trivial, but when you start to digg a bit you see 4 Mb limitations with .NET, downloads that use a lot of memory, that CAN block other threads (when you have a limit on the number of threads, and if a thread can execute for the duration of 2 Gb file upload/download over the internet : well this ain't gonna scale very well, will it ?), etc.

You'll need:

  • Client-side code (Java Applet, Silverlight, etc) to break files in small chunks
  • Server-side code (ASP.NET, Java, doesn't matter) to build those files back

I just finished an application exactly like that; I'd use Silverlight (WebRequest async), ASP.NET (IHttpHandler/IHttpAsyncHandler) and SQL Server 2005 (UPDATETEXT/READTEXT) for file storage.

UPDATE: About ASP.NET server-side code:

ASP.NET default config will allow 100 threads per processor; IHttpAsyncHandler won't block your process and there you can write your file content directly to context.Response.OutputStream.

For upload, you'll also send several data chunks, but in multiple HTTP connections; while this can bring some overheat due HTTP headers, works very well in my tests.