Get contents of another site's element using only JavaScript

Question

I'm making a userscript that requires finding a word related to another word. I found a site where I can go to http://semantic-link.com/#/stack, and there will be a list of words related to stack in the format of:

<!-- various stuffs here, e.g. <head>, a <div>, another <div>, start of <body> -->

    <div id="word0" class="word" onclick="updateTitle("stacks");" style="opacity: 1;">
        stacks
    </div>
    <div id="word1" class="word" onclick="updateTitle("flue");" style="opacity: 1;">
        flue
    </div>
    <div id="word2" class="word" onclick="updateTitle("popped");" style="opacity: 1;">
        popped
    </div>
    <div id="word3" class="word" onclick="updateTitle("overflow");" style="opacity: 1;">
        overflow
    </div>

In this case, I would want to just get the string stacks from that page. All of the solutions I have found don't work because they:

need a custom page,
use PHP, or
use $.ajax(), $.get(), or $('something').load() which are limited to the domain they are executed in.

Is there any way I can get the contents of the #word0 element as a string, using only JavaScript? Alternatively, is there another way to find a word related to another word?

What about a scriptlet/bookmarklet? What do you plan on using the results for? Pretty sure doing this is a violation of cross-site scripting which is why you would have to use those tools. (And why ajax won't work) — zero298, Apr 11 '14 at 21:06
I don't think that this would be possible with JavaScript, but you could use PHP's (if that is your server-side language) http://us2.php.net/file_get_contents — Jay Blanchard, Apr 11 '14 at 21:06
@zero298 I was going to send the word as a message in an SE chat room. — The Guy with The Hat, Apr 11 '14 at 21:10
@ProllyGeek No, you can't [Get DOM content of cross-domain iframe](http://stackoverflow.com/q/6170925/691711) — zero298, Apr 11 '14 at 21:10
If you do not own the page or have the option of AJAX then you cannot get the content of a remote page using JavaScript alone. — Jay Blanchard, Apr 11 '14 at 21:11
See if there is a restful API or a way to do some JSONP magic or maybe even a CORS if you are lucky. — zero298, Apr 11 '14 at 21:13
@ProllyGeek the same-origin policy does. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Same_origin_policy_for_JavaScript — Kevin B, Apr 11 '14 at 21:16
you could certainly ask [the owner](http://mpacula.com/) of semantic-link for a RESTful API that you could call — pennstatephil, Apr 11 '14 at 21:20
You **CAN** do this using the techniques from ["How to get an AJAX get-request to wait for the page to be rendered before returning a response?"](http://stackoverflow.com/a/11489451/331508). Greasemonkey and Tampermonkey do cross-domain mashups all the time. — Brock Adams, Apr 11 '14 at 21:36

score 6 · Answer 1 · edited Apr 12 '14 at 01:42

6

You state in your question that:

$.ajax(), $.get(), or $('something').load() ... are limited to the domain they are executed in.

This appears to point the finger at jQuery. Actually, the limitation is with Javascript in the browser itself, and is down to a particular security aspect called Same Origin Policy:

The policy permits scripts running on pages originating from the same site – a combination of scheme, hostname, and port number – to access each other's DOM with no specific restrictions, but prevents access to DOM on different sites

In essence, what you are trying to do is not possible, unless the remote server has CORS enabled to allow queries from your server to call across to it. As you are not in control of the remote server, I doubt this is set up.

edited Apr 12 '14 at 01:42

hichris123

10,145
15
56
70

answered Apr 11 '14 at 21:11

Adrian Wragg

7,311
3
26
50

Actually, userscripts have APIs, like `GM_xmlhttpRequest` to do cross-domain AJAX. That is not the main problem here anyway, as the target page is AJAX driven -- which requires more of a workaround. – Brock Adams Apr 11 '14 at 21:40
@BrockAdams Just to clarify, am I right in saying that any user script would require the user to install a third-party plugin or script first (which would probably trigger security warnings)? – Adrian Wragg Apr 11 '14 at 22:00
No. Chrome and Opera support this natively. But use the Greasemonkey, Tampermonkey and/or Scriptish extensions; they are better. As for security and privacy: their code's openly viewable and/or open-source and has a solid track record. Frankly, those extensions are less risk than installing Chrome or Firefox in the first place. – Brock Adams Apr 11 '14 at 22:26
@BrockAdams Sorry, but you seem to be saying that *by default* Chrome and Opera allow cross-domain scripting without the user installing any plugins or scripts into the browser itself. It's obviously an area that you know better than I do, so I think our wires have been crossed somewhere here. – Adrian Wragg Apr 11 '14 at 22:33
No, I'm saying that by *default* Those browsers allow cross-domain scripting, *from a userscript*, without the user installing any plugins or extensions. The OP stated and implied that he was "making a userscript", so installing a userscript is a given. – Brock Adams Apr 11 '14 at 22:38
@BrockAdams You're right; I was distracted by the OP coming across cross-domain issues, which (if s/he *was* actually writing a userscript) should not have been an issue. Thanks for clarifying. – Adrian Wragg Apr 11 '14 at 22:45
Cross domain is still an issue for *jQuery* AJAX, even from a userscript. (jQuery does not use the privileged `GM_xmlhttpRequest` function) So, that part's still right. – Brock Adams Apr 11 '14 at 22:57

score 4 · Answer 2 · answered Apr 11 '14 at 21:08

4

This isn't possible with just javascript due to the same-origin policy. There has to be some server-side intervention to retrieve the html, whether that be through a 3rd party service (such as YQL) or your own php(or otherwise) script.

answered Apr 11 '14 at 21:08

Kevin B

94,570
16
163
180

Rakesh Bollampally · Accepted Answer · 2014-04-11T22:43:21.483

2

Why not just get the JSON he is getting? Correct me if I'm wrong but you are only after related words but not what a user is clicking?

$.get("http://semantic-link.com/related.php?word="+word);

EDIT: try this:

$.getJSON("https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20json%20where%20url%3D'http%3A%2F%2Fsemantic-link.com%2Frelated.php%3Fword%3Dword'&format=json&diagnostics=true&callback=",function(data){
alert(data.query.results.json.json[0].v);
});

edited Apr 11 '14 at 22:43

answered Apr 11 '14 at 21:34

Rakesh Bollampally

181
7

Because that call doesn't work across different domains in JavaScript. – Adrian Wragg Apr 11 '14 at 22:22
1

@AdrianWragg, that call works fine, and to that domain, from a userscript, if `$.get` is replaced by `GM_xmlhttpRequest`. – Brock Adams Apr 11 '14 at 22:42
You are right. It does not have necessary headers. Added YQL version – Rakesh Bollampally Apr 11 '14 at 22:42

score 1 · Answer 4 · answered Apr 11 '14 at 21:35

1

I think what you want is quite feasible and quite simple to accomplish.

First you must notice that when you query for any word in that website it'll make an "ajax" call behind the scenes, like for example: http://semantic-link.com/related.php?word=shrewd. Being aware of that URL is the trick.

And please notice that it brings you a JSON format as response. So all you need to do with now is to parse it. Just replace the word in the URL in the example above by the one you want to use and you're done. ;)

answered Apr 11 '14 at 21:35

leandroico

1,207
13
17

1

That works on the site itself because they are on the same domain. The functionality would be different when a call is made from a different domain, due to the cross-domain issues mentioned. – Adrian Wragg Apr 11 '14 at 22:03
Yeah. It still must be done from back-end. But it's better to know there is a JSON friendly URL for it in advance. – leandroico Apr 11 '14 at 22:05
1

That is a good point, and worth knowing if the original questioner does decide to go down that route instead of implementing a pure JS solution. – Adrian Wragg Apr 11 '14 at 22:07
2

Calling that URL works perfectly fine from a userscript using `GM_xmlhttpRequest`. It's pure JS (standard userscript API) and no back-end required. – Brock Adams Apr 11 '14 at 22:35

Get contents of another site's element using only JavaScript

4 Answers4