here is a fast way of cheating the nested iteration that _.difference will invoke:
var arr1 = [123, 456, 789],
arr2 = [123, 456, 789, 098],
has = {};
arr1.forEach(function(a){ this[a]=1;}, has);
alert( arr2.filter(function(a){return !this[a]; }, has) );
by using this in the iteration, we hand a pure function to JS that can be executed at the maximum possible speed.
note that this won't work for arrays of objects, or mixed-type arrays like [1, "1"], but it should work for the problem described and demonstrated by the question.
edit: it you want bi-directional compares (like arr1 having, arr2 missing or vice-versa), reverse and repeat the code above. you'll still only be at 40million computations compared to 100trillion that an indexOf()-using method will cost...