How to get content of a remote website (I don't have access to modify that website)

Question

Let's say for example I want to get the <head> tag of youtube or another remote website. Currently, I get an error that states:

Access to XMLHttpRequest at ‘WebSite I request to get the head section’ from origin ‘My Website’ has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

I have tried doing an XMLHttpRequest but I'm blocked by the origin. In this case, I'm not trying to inject or modify something but instead get the head section which you can easily get by copy-pasting from the source code (so I don't see a security concern there).

This is the function I'm using to make that happen:

function httpGet(theUrl) {
    if (window.XMLHttpRequest) {// code for IE7+, Firefox, Chrome, Opera, Safari
        xmlhttp = new XMLHttpRequest();
    }
    else {// code for IE6, IE5
        xmlhttp = new ActiveXObject("Microsoft.XMLHTTP");
    }
    xmlhttp.onreadystatechange = function () {
        if (xmlhttp.readyState == 4 && xmlhttp.status == 200) {
            console.log(xmlhttp.responseText);
            return xmlhttp.responseText;
        }
    }
    xmlhttp.open("GET", theUrl, false);
    xmlhttp.send();
}

Is there a way to possibly send a request to a remote site and get the head content (read-only)? Currently, there are similar questions here but none of them answer as different things worked for different developers and I have a hard time understanding if that is a possibility.

Update: here is the manifest.json file however, i want to mention again that the user doesn't have that domain open in a tab or anywhere else. It's just a random domain.

{
  "manifest_version": 3,
  "version": "0.0.5",
  "action": {
    "default_popup": "popup/popup.html"
  },
  "permissions": ["activeTab", "scripting"],
  "host_permissions": ["<all_urls>"]
}

If you just get the tag, you should use `executescript`. This sample gets `document.title`, but the principle is the same.[sample](https://stackoverflow.com/questions/75318186/chrome-extension-fetching-the-value-of-a-dom-element/75318947#75318947) — Norio Yamamoto, Mar 05 '23 at 00:21
@NorioYamamoto execute script would work if you have the URL open in a tab, in my case I don't. I just want to hit that URL that is not open, not in an iframe or anything, and still, be able to get the head section. — Mobination, Mar 05 '23 at 00:39
I think it will be easier to get the answer if you post `manifest.json`. — Norio Yamamoto, Mar 05 '23 at 00:58
Posting in comments is very hard to read. You can edit your question. — Norio Yamamoto, Mar 05 '23 at 01:14
@NorioYamamoto yes realized that thank you, updated my question. — Mobination, Mar 05 '23 at 01:16
Your code is correct. The error is unrelated to the current code, so I guess you look at the old errors in chrome://extensions page. Use devtools instead. Note that the popup is a separate window so it has its own separate devtools: right-click inside the popup and select "inspect" in the menu. Also make sure you reload the extension after editing manifest.json. — wOxxOm, Mar 05 '23 at 03:07

Norio Yamamoto · Accepted Answer · 2023-03-05T03:05:03.467

I tested your extension and it worked fine. The manifest.json is slightly modified but basically the same.

manifest.json

{
  "name": "hoge",
  "manifest_version": 3,
  "version": "0.0.5",
  "action": {
    "default_popup": "popup.html"
  },
  "host_permissions": [
    "<all_urls>"
  ]
}

popup.html

<html>
<body>
  <script src="popup.js"></script>
</body>
</html>

popup.js

function httpGet(theUrl) {
  if (window.XMLHttpRequest) {// code for IE7+, Firefox, Chrome, Opera, Safari
      xmlhttp = new XMLHttpRequest();
  }
  else {// code for IE6, IE5
      xmlhttp = new ActiveXObject("Microsoft.XMLHTTP");
  }
  xmlhttp.onreadystatechange = function () {
      if (xmlhttp.readyState == 4 && xmlhttp.status == 200) {
          console.log(xmlhttp.responseText);
          return xmlhttp.responseText;
      }
  }
  xmlhttp.open("GET", theUrl, false);
  xmlhttp.send();
}

const url = "https://stackoverflow.com/";
httpGet(url);

Thank you. Even though the content is being printed, if you look at the console, the result of const ret = httpGet(url); console.log(ret); is returning undefined. Can you please tell me how to capture that data so console.log would show it instead of logging the response in the function itself? — Mobination, Mar 05 '23 at 01:55
I'm sorry. It's my bug that I am printing the return value of `httpGet`. I will fix that. — Norio Yamamoto, Mar 05 '23 at 03:04

How to get content of a remote website (I don't have access to modify that website)

1 Answers1