The problem is that jQuery UI lives in the browser window and will stay "alive" as long as the browser window stays open. The tabs however are in a content tab, they will disappear as soon as you close that tab. jQuery UI must be keeping a local reference to the tabs somewhere and that reference doesn't let the tabs to get garbage collected even after the tab is closed (zombie compartment). One solution would be finding that reference and getting rid of it. But even if you solved it this way (I cannot since I don't know enough about jQuery) - the issue will likely come back later because jQuery UI is designed to run in a web page, this kind of memory leaks isn't considered there.
A more robust solution would be running jQuery UI in the same context as the UI it is responsible for. This way when you close the tab both jQuery UI and its widgets will be disposed - no more zombie compartments. This could be done using the message manager for example:
// Load content script into the current tab
var contentScriptURL = "chrome://.../content/contentScript.js";
gBrowser.selectedBrowser.messageManager.loadFrameScript(contentScriptURL, false);
And contentScript.js
would then load jQuery and jQuery UI into its context using mozIJSSubscriptLoader:
var scriptLoader = Components.classes["@mozilla.org/moz/jssubscript-loader;1"]
.getService(Components.interfaces.mozIJSSubScriptLoader);
scriptLoader.loadSubScript("chrome://.../content/jquery.js");
var jQuery = jQuery.noConflict(true);
scriptLoader.loadSubScript("chrome://.../content/jquery-ui.js", jQuery);
...
$j(mydiv).find('#targetid').tabs({selected: 2});
This content script has the same chrome privileges as the scripts running in your overlay but it will be unloaded once the tab is closed - and along with it any references it might be keeping to contents of that tab.