4

I am creating a web service of scheduled posts to some social network.
Need help dealing with file uploads under high traffic.

Process overview:

  • User uploads files to SomeServer (not mine).
  • SomeServer then responds with a JSON string.
  • My web app should store that JSON response.

Option 1: Save, cURL POST, delete tmp
The stupid way I made it work:

  1. User uploads files to MyWebApp;
  2. MyWebApp cURL's the file further to SomeServer, getting the response.

Option 2: JS magic
The smart way it could be perfect:

  1. User uploads the file directly to SomeServer, from within an iFrame;
  2. MyWebApp gets the response through JavaScript.

But this is(?) impossible due to the 'Same Origin Policy', isn't it?

Option 3: nginx proxying?
The better way for a production server:

  1. User uploads files to MyWebApp;
  2. nginx intercepts the file uploads and sends them directly to the SomeServer;
  3. JSON response is also intercepted by nginx and processed by MyWebApp.

Does this make any sense, and what would be the nginx config for, say, /fileupload Location to proxy it to SomeServer?

Serge
  • 1,531
  • 2
  • 21
  • 44
  • You can overcome the Same Origin Policy, but for that to happen, SomeServer should add [Cross-Origin Resource Sharing (CORS)](http://en.wikipedia.org/wiki/Cross-Origin_Resource_Sharing) headers to its responses. Check if it implements CORS, or whether you can persuade SomeServer's developers to implement it. – Dmytro Shevchenko Apr 13 '12 at 22:14
  • SomeServer does implement CORS headers, but unfortunately, only for their own root domain. The "Access-Control-Allow-Origin" HTTP header offers a single value. I doubt their developers would tweak that for my service. So, back to nginx / post forwarding. – Serge Apr 17 '12 at 16:05
  • You can also use `postMessage ` to communicate between frames regardless of source domain – tkone Apr 18 '12 at 01:13
  • @tkone This is true, but you cannot upload files with `postMessage`. Well, maybe you could somehow serialize/deserialize a file. But in any case, using `postMessage` requires development on the SomeServer's side, which @SergikS doesn't seem to be able to do. By the way, `postMessage` is used as a part of communication way in EasyXDM which I mentioned in my asnwer below. – Dmytro Shevchenko Apr 18 '12 at 01:58
  • @tkone: postMessage requires server side programming.But how could you upload files with postMessage !! – Abdur Rahman Apr 19 '12 at 11:29

2 Answers2

1

I can see only two major approaches to this problem: server-side proxying and javascript/client-side cross-site uploading. Your approaches 1 and 3 are the same thing. It shouldn't really matter whether you POST files with means of cURL or nginx - not performance-wise anyway. So if you already implemented approach 1 from your question, I don't see any reason to switch to 3.

In regards to javascript and Same Origin Policy, it seems there are many ways to achieve your goal, but in all of these ways, either your scenario must be supported by SomeServer's developers, or you have to have some sort of access to SomeServer. Here's an approximate list of possibilities:

  • CORS—your domain must be allowed to access SomeServer's domain;
  • Changing document.domain—this requires that your page and target page are hosted on subdomains of the same domain;
  • Using a flash uploader (e.g. SWFUpload)—it is still required that your domain is allowed via the cross-domain policy, in case of Flash, via a crossdomain.xml in the root of SomeServer's domain;
  • xdcomm (e.g. EasyXDM)—requires that you can upload at least an html page to the target domain. This page can then be used as a javascript proxy for your manipulations with SomeServer's iframe.

The last one could, actually, be a real possibility for you, since you can upload files to SomeServer. But of course, it depends on how it's implemented—for example, in case there is another domain the files are served from, or if there are some security measures which won't allow you to host html files, it may not work out.

Community
  • 1
  • 1
Dmytro Shevchenko
  • 33,431
  • 6
  • 51
  • 67
  • SomeServer is vk, if that makes more sense. For file uploads they first provide the available CDN server url, then you upload files to that url. It only accepts images/audio/video, no html/js or whatsoever. So I would roll back to the initial question of configuring nginx. Bypassing php/tmp storage/curl would save a good hand of server resources, at least with curl memory / php threads. So I am still looking at this approach. – Serge Apr 18 '12 at 05:18
  • I had an impression that your web app was the main one, and you loaded SomeServer in an iframe. But does VK allow that? Can you add more details on what your app is (standalone site? VK iframe application?), and how exactly it uses VK? – Dmytro Shevchenko Apr 18 '12 at 08:48
  • That's far beyond the initial question's scope. @Shedal, You can reach me in Skype or drop me an email to discuss the app itself. The question regarding nginx proxying file uploads is still open. – Serge Apr 18 '12 at 11:12
  • @Serg I just thought that the "JS magic" may be not completely out of question in some cases, depending on how your app is set up. In regards to nginx proxying - have you tried using [HttpProxyModule](http://wiki.nginx.org/HttpProxyModule)? E.g. as described in [this answer](http://serverfault.com/a/312114/117436). – Dmytro Shevchenko Apr 18 '12 at 11:23
  • thanks for the other answer link. Seems like forwarding POST data is against HTTP spec. Perhaps I should stick with the initial man-in-the-middle strategy: save the user upload on MyWebApp, then curl it further to SomeServer. – Serge Apr 18 '12 at 11:35
  • @Serg Actually, HttpProxyModule forwards requests server-side, not via 301/302 HTTP response header redirect. So POSTs **will** be redirected this way. Matthew's quote from specification was to explain why the original problem arised - JihoKang tried to redirect with nginx rewrite which uses HTTP response header. What I'm not completely sure of is whether the client's browser will get JSON response (when using HttpProxyModule). On the other hand, I don't really see why not. Anyway, you'll have to test it. – Dmytro Shevchenko Apr 18 '12 at 11:41
  • That's the next question then: how to let nginx process the response to the request it has forwarded to another server, instead of just passing it back to the requesting browser? `Nginx in the middle` : ) – Serge Apr 19 '12 at 03:46
  • What's the problem in passing the response back to the browser? – Dmytro Shevchenko Apr 19 '12 at 06:34
  • not really a problem: with this setup JS would be able to pass JSON response back to myWebApp — since it just received that from it: same origin. Just an imperfection now: why pass the response to browser, than JS/ajax it back, when we just had all the necessary data inside the proxy module? =) – Serge Apr 19 '12 at 09:15
  • Yes, but I think you want too much now ;) Proxy is just what it is - a proxy. It's transparent for the client, and for your application as well. If you want some additional logic, you can stay at option #1 where you implement the proxying part manually. – Dmytro Shevchenko Apr 19 '12 at 09:31
1

I don't have a server to use to stand in for SomeServer for me to test out my suggestions, but I'll give it a shot anyway. If I'm wrong, then I guess you'll just have to use Flash (sample code from VK).

How about using an iFrame to upload the file to SomeServer, receive the JSON response, and then use postMessage to pass the JSON response from the iFrame to your main window from your site. As I understand it, that is pretty much the motivation for creating postMessage in the first place.

Overall, I'm thinking of something like this or YUI's io() module but with postMessage added to get around the same origin policy.

Or in VK's case, using their explicit iFrame support. It looks to me like you can add a method to the global VK object and then call that method from the VK origin domain using VK.callMethod(). You can use that workaround to create a function that can read the response from the hidden iFrame.

So you use VK.api('photos.getUploadServer', ...) to get the POST URL.

Then you use JS to insert that URL as the action for your FORM that you use to upload the file. Follow the example under "Uploading Files in an HTML Form" in the io() docs and in the complete function, use postMessage to post the JSON back to your parent window. See example and docs here. (If it doesn't work with io(), you can certainly make it work using the roll-your-own example code if I'm right about VK.callMethod().)

Then in response to the postMessage you can use regular AJAX to upload the JSON response back to your server.

Old Pro
  • 24,624
  • 7
  • 58
  • 106
  • SomeServer's url is new each time — picked from a large CDN. File upload is made of 3 steps: **1)** api call to `api.SomeServer` which returns an url like `cdn1234.SomeServer` plus some parameters; **2)** HTTP POST to that url with field `file1` containing picture file data; Server returns nothing but a JSON string with a hash field I would need further. **3)** another api call to 'enable' the uploaded picture by its hash. – Serge Apr 19 '12 at 03:40
  • One really bad solution I've seen was to inject JS into the returned answer unsing a custom browser plugin (seen one for Chrome), that every user of my web app would have to install. That plugin would then append my JS to every page from SomeServer domains, enabling the cross origin communication. But that's a really weird solution. – Serge Apr 19 '12 at 03:41
  • @Serg, you say SomeServer is actually VK. Have you read [VK's iFrame documentation](http://vk.com/developers.php?oid=-17680044&p=IFrame_Applications)? Particularly at the bottom, **Using VK API in an IFrame Application** – Old Pro Apr 19 '12 at 05:12
  • @OldPro That's for the case when the application is hosted in an iFrame inside VK. I gather that Serg has his setup the other way round. Concerning `postMessage`, I believe it's only possible if you have an html/js proxy hosted at SomeServer. – Dmytro Shevchenko Apr 19 '12 at 08:23
  • @OldPro sure I read that. Calling api methods is not the problem here. Its the file upload via a POST request, and the JSON response to it — which myWebApp, hosted on a separate domain, wants to receive. Its a three party puzzle: **U** uploads to **A**, **B** wants to know the **A** response. No way to tweak **A**'s internals. Same Origin Policy needs to be overridden without browser hacks and no access to one of the servers. – Serge Apr 19 '12 at 09:09
  • @Serg, I added more detail to my answer to show step by step how to do it. If that still doesn't work, you can do it with Flash, using [VK's example upload solution](http://vk.com/app1732516) – Old Pro Apr 19 '12 at 20:56
  • @OldPro The only (unsolvable) problem here is getting around Same Origin Policy. What is your suggestion about that? Using `postMessage`? But then you would have to have an html/js proxy at VK server, isn't that true? – Dmytro Shevchenko Apr 20 '12 at 09:33
  • @Shedal, the hidden info behind the info is that *SameServer* just wouldn't be realistically set up to not provide a solution to the OP's problem. Here, *SameServer* is VK, and VK provides a solution. Now their solution uses Flash, but as it turns out, their iFrame solution should work, too. You don't need an html/js proxy at VK, you just need an iFrame at VK that will talk to your window at your site (using `postMessage` or some other mechanism). That's what VK's iFrame support provides. – Old Pro Apr 20 '12 at 11:26
  • @OldPro Can you understand Russian? The [iFrame support](http://vk.com/developers.php?oid=-17680044&p=IFrame_Applications) is for applications which run in an iframe, where the main window is VK itself. I believe Serg has it the other way round, i.e. he has a page which opens VK in an iFrame. Serg, can you confirm? – Dmytro Shevchenko Apr 20 '12 at 11:30
  • @Shedal, no, I do not understand Russian. According to the English language documentation and examples, the iFrame support is for embedding a VK iFrame in your page on your server so your page can access VK. – Old Pro May 05 '12 at 21:48