I've an upcoming project where I will need to handle very large uploads from browsers (either the classic input type="file" or a Java Applet), and looking for the best tool to do the job on the server.
Theses are the things I need :
- low memory consumption on the server
- ability to save the file in its final destination on the server (no copying the file around)
- no blocking of other important tasks done by the webserver
- good handling of files up to 2 gb
- authorization of files (permissions would be given in the app)
I still have some latitude on what technology to use so I would like to have some advice in order to be able to choose the best technology on the server to handle this task :
- ASP.NET ?
- Java ?
- Amazon S3 ?
- Other choices ?
I'm more used to the Microsoft Stack, but willing to change if necessary : as told above, I'm just looking for the best tool for the job.
Update : The server side is the part I'm really interested in for this question, not the client side.
It looks like it may be trivial, but when you start to digg a bit you see 4 Mb limitations with .NET, downloads that use a lot of memory, that CAN block other threads (when you have a limit on the number of threads, and if a thread can execute for the duration of 2 Gb file upload/download over the internet : well this ain't gonna scale very well, will it ?), etc.
- Client-side code (Java Applet, Silverlight, etc) to break files in small chunks
- Server-side code (ASP.NET, Java, doesn't matter) to build those files back
I just finished an application exactly like that; I'd use Silverlight (
WebRequest async), ASP.NET (
IHttpAsyncHandler) and SQL Server 2005 (
READTEXT) for file storage.
UPDATE: About ASP.NET server-side code:
ASP.NET default config will allow 100 threads per processor;
IHttpAsyncHandler won't block your process and there you can write your file content directly to
For upload, you'll also send several data chunks, but in multiple HTTP connections; while this can bring some overheat due HTTP headers, works very well in my tests.