Access to image and video data from browser extension (vs CORS)

Question

I'm trying to write a browser extension that does some image processing but I need access to the image data. My approach was to create a hidden canvas element, draw images and video to it via drawImage, and then read the pixel data with getImageData. This works just fine but on many pages half the content is denied by CORS errors.

I'm still confused as to why CORS exists (something along the lined of not stealing data, but then if the data is on the client's computer isn't it "stolen" already? :S). All it seems to lead to is retarded hacks like JS script injection. So 1. it doesn't work because it's too complicated for every browser to police correctly and 2. devs are punished and have to write browser-specific workarounds. So I'm thinking I must have the wrong idea because this seems pretty stupid.

Taking a step back, I think the idea of an extension that can do some image processing is perfectly normal and not malicious so please do not reply with "no, you shouldn't be doing this for security reasons".

I suspect that the browser is treating the extension as something foreign that could be trying to do malicious things. How can I reassure the browser that the client wants these features and have it grant me access to the image and video content? I already have full access to the DOM, how is a little bit extra going to make any difference??

Is there another way to get image/video data from an extension?

Answer 1

After adding the right permissions to the manifest file, you can deal with cross-origin data as without being hindered by the same origin policy. In a background page or any other page within the extension's process, you can get a working demo of your described logic as follows:

// background.js
var canvas = document.createElement('canvas');
var context = canvas.getContext('2d');
var image = document.createElement('img');
image.onload = function() {
    context.drawImage(image, 0, 0);
    // Demo: Show image
    window.open(canvas.toDataURL('image/png'));
};
image.src = 'https://stackoverflow.com/favicon.ico';

Here is a minimal manifest file for this specific demo:

{
    "name": "Get image data",
    "version": "1",
    "manifest_version": 2,
    "background": {
        "scripts": ["background.js"]
    },
    "permissions": [
        "https://stackoverflow.com/favicon.ico",
        "http://cdn.sstatic.net/stackoverflow/img/favicon.ico"
    ]
}

The second permission is necessary because https://stackoverflow.com/favicon.ico redirects to http://cdn.sstatic.net/stackoverflow/img/favicon.ico.

Note that the code does not work in content scripts, because the DOM is shared and Chrome cannot offer unrestricted origin access to content scripts only.

Depending on what you really want, you could also just try to get the image data in raw form using XMLHttpRequest, and process it in any way you desire. This method also works in content scripts, and their advantage is that it is more efficient, and also preserves the bytes of the image (with the canvas method, the image is parsed and processed by the browser before it is converted to a data URL).

var x = new XMLHttpRequest();
x.responseType = 'blob';
x.open('get', 'http://stackoverflow.com');
x.onload = function() {
    var fileReader = new FileReader();
    fileReader.onloadend = function() {
        // fileReader.result is a data-URL (string)
        window.open(fileReader.result);
    };
    // x.response is a Blob object
    fileReader.readAsDataURL(x.response);
};
x.send();

Answer 2

A workaround to the problem is to use Chrome's desktopCapture feature. Note this means the user has to interact with a popup asking for the permission every time you want to record (which could be left on for a long time, but I guess it's not very resource efficient).

I'm putting the code here if it helps.

background.js

var pending_request_id = null;

var video = document.createElement('video');
var canvas = document.createElement('canvas');
var ctx = canvas.getContext("2d");
var theStream = null;

var CAPTURING = false;

/*
 * Setting up recording when the user clicks the extension icon
 */
chrome.browserAction.onClicked.addListener(function() {
  if (CAPTURING) {
    console.log('stop capturing');
    theStream.getVideoTracks()[0].stop();
    chrome.browserAction.setIcon({path: "icons/icon128.png"});
    CAPTURING = false;
  } else {
    pending_request_id = chrome.desktopCapture.chooseDesktopMedia(
      ["screen","tab"],
      onAccessApproved
    );
  }
});

function onAccessApproved(id) {
  console.log('onAccessApproved: ' +id)
  if (!id) {
    console.log("Access rejected.");
    return;
  }
  navigator.webkitGetUserMedia({
    audio:false,
    video: {
      mandatory: {
        chromeMediaSource: "desktop",
        chromeMediaSourceId: id,
        maxWidth: 4000,
        maxHeight: 4000
      }
    }
  }, gotStream, getUserMediaError);
}

function getUserMediaError(error) {
  console.log("getUserMedia() failed:", error);
}

function gotStream(stream) {
  console.log("gotStream", stream);
  theStream = stream;
  video.src = URL.createObjectURL(theStream);
  video.play();
}

video.addEventListener('loadedmetadata',function(){
  console.log('loadedmetadata')
  chrome.browserAction.setIcon({path: "icons/icon128_recording.png"});
  CAPTURING = true;
}, false);


/*
 * You can trigger this when you want; I am doing it on every mouse move over the video (triggered by the content_script.js)
 */
chrome.runtime.onMessage.addListener(
  function(request, sender, sendResponse) {
    if (!CAPTURING) {
      console.log('User needs to activate the recording of the screen')
      return false; // no response
    }

    ctx.drawImage(video, 0, 0);
    var url = canvas.toDataURL('image/png');
    window.open(url);
  }
);