4

I am working on a file upload system which will store individual parts of large files on more than one server. So the distribution of a 1GB file will look something like this:

Server 1: 0-128MB
Server 2: 128MB-256MB
Server 2: 256MB-384MB
... etc

The intention of this is to allow for redundancy (each part will exist on more than one server), security (no one server has access to the entire file), and cost (bandwidth expenses are distributed).

I am curious if anyone has an opinion on how I might be able to "trick" web browsers into downloading the various parts all in one link.

What I had in mind was something like:

  1. Browser is linked to Server 1, which provides a content-size of the full file
  2. Once 128MB is served, Server 1 will intentionally close the connection
  3. Hopefully, the browser will try to restart the download, requesting Server 1
  4. Server 1 provides a 3XX redirect to Server 2
  5. Browser continues downloading from Server 2

I don't know for certain that my example works, as I haven't tested it yet. I was curious if there were other solutions someone might have?

I'd like to make the whole process as easy as possible (ideally requiring no work beyond a simple download). I don't want the users to have to use another program (ie: cat'ing the files together). I'd also like to not use a proxy server, since it would incur extra bandwidth costs.

As far as I'm aware, there is no javascript solution for writing a file, if there was one, that would be great.

GoldenNewby
  • 4,382
  • 8
  • 33
  • 44

4 Answers4

2

AFAIK this is not possible by using the HTTP protocol. You can probably use a custom browser extension but it would depend on the browser. Another alternative is to create a Java applet that would download the file from different servers. The applet can accept the URLs to the different servers as parameters.

Coder
  • 376
  • 2
  • 4
2

To save the generated file: https://stackoverflow.com/a/4551467/329062

That solution stores the file in memory though, so it won't work with very large files.

You can download the partial files into a JS variable using JSONP. That will also let you get around the same-origin policy.

Community
  • 1
  • 1
Greg
  • 12,119
  • 5
  • 32
  • 34
  • That's a really cool tricky, I really like it; however, it won't work for the size of files I'm dealing with (100MB+). Awesome find though, thanks. – GoldenNewby Feb 26 '12 at 22:38
1

It's possible to do in modern browsers over standard HTTP.

You can use XHR2 with CORS to download file chunks as ArrayBuffers and then merge them using Blob constructor and use createObjectURL to send merged file to the user.

However, I suspect that browsers will store these objects in RAM, so it's probably a bad idea to use it for large files.

Kornel
  • 97,764
  • 37
  • 219
  • 309
1

Javascripts security model will only allow you to access data from the same origin where the Javascript came from - i.e. not multiple servers.

If you are going to have the file bits on multiple servers, you will need the user to load the web page, fetch the bit and then finally stick the bits together in the correct order. If you can manage to get all your users to do this (correctly), you are a better man than I.

Ed Heal
  • 59,252
  • 17
  • 87
  • 127