Why do callbacks passed to chrome extension background script silently fail or return false?

Question

I'm refactoring a chrome extension that's worked for years. I'm moving chrome.runtime.* calls from content scripts to extension background so scripts can run externally (on my domain instead of content script in the extension). Manifest v2, externally connectable is setup properly, all good.

Here's what happens. My web page script sends a message to extension with a callback for the results. Listener in extension background script gets the message with all correct data. Callback looks ok, it's a function object. Calling it on client side with test data works as intended. Otherwise, the callback never does anything.

Code

Here's sample code to illustrate what happens. Presenting python (brython) for explanation, same thing happens with javascript. Using console.log as the callback for testing purposes.

# ===== client script

chrome.runtime.sendMessage (EXTN_ID, {'action': 'get_tabs', 'args': {}}, console.log)

# ===== server side, extension background script

def get_tabs (args, sender, callback) :

    args ['windowId'] = sender.tab.windowId  # tabs for caller's window
    callback ('foo')  # <--- ok prints foo in client console
    chrome.tabs.query (args, callback)  # <-- usually does nothing: never prints to client console, no error messages in extension background console.  but sometimes prints "false" in client console

    # --- or try this way
    def check_result (res) :
        console.log (res)  # <-- ok prints tabs to extension background console
        callback (res)     # <-- does nothing: never prints to client console, no errors in extension background console

    chrome.tabs.query (args, check_result)  # <-- calls back check_result above, prints tabs to extension bg console. but nothing happens on client side and no error messages in either console

# ----------------
# message listener, this part works fine

def receive (msg, sender, callback) :
    if (msg.action == 'get_tabs') :
        get_tabs (msg.args, sender, callback) ;
    return true  # <-- for async responses, per wOxxOm comment

chrome.runtime.onMessageExternal.addListener (receive)

Observed Behavior

Other than test data, the callback never produces any results:

• passing it to chrome.tabs.query (args, callback) does nothing. silently fails
• invoking the callback manually in the extension before calling chrome.tabs.query works just fine. returns data correctly (but not useful, since I don't have real values to return yet).
• wrapping it in another callback (i.e. extension-side wrapper function around client-side callback) shows that chrome.tabs.query returns the correct results. but passing them back to first callback silently fails.
• once the extension calls chrome.tabs.query, the client side usually never sees the callback invoked at all. but sometimes, the callback mysteriously receives a single value: false. Even when the wrapper gets the correct results, the client callback is called with "false" instead of the real results.

Attempted Fixes

I tried all sorts of crazy things to fix this:

1. Tried wrapping python callback function in a javascript function, doesn't help (later confirmed same issue occurs when using pure javascript on both ends). No change.
2. Tried removing *args and **kwargs from all function signatures in the call chain, thinking brython might be mangling the callback parameter somehow. Didn't make a difference.
3. Tried saving the callback on client side and sending a callback identifier instead in message to extension, in case there was a problem invoking the function across a protection boundary (client script -> extension background -> chrome internal). Like this:

• client stores callback and sends callback id to extension instead
• extension creates local callback to get results from chrome.tabs.query
• local callback sends a new message back to client with the results and callback id
• client gets callback id from message and invokes stored callback with the results

The last way probably would've worked, but I couldn't find a good way to send reply messages from the extension to the client without using a client-provided callback. No API for client side to listen for messages from extension - chrome docs say only clients can initiate messages. No way that I could find for extension to use window.postMessage to contact a client-side listener. Could do it by opening a long lived connection from client with chrome.runtime.connect instead of sending a single message. But that seems like overkill for one round-trip message and reply. And managing open connections across long-lived web pages seems like more hassle than it's worth.

None of this helped. I later confirmed the same issue happens when the entire chain is javascript. Whether it's js or python, the callback works "correctly" when it's called early before chrome.tabs and not at all after chrome.tabs. Python doesn't seem to be the problem.

Answer 1

After 3 days and 3 nights bashing my head against the wall, I finally (finally!) figured out what's going on. There are two different problems at play:

1. The tabs array returned by chrome.tabs.query does not serialize. Calling the callback with the tabs array creates a serialization error. AFAICT this error is not reported anywhere in any consoles (client or extension). The callback just gets silently swallowed. Way to go, chrome. :(

2. A callback function is apparently a magic function that can only be called once. Call it the first time and it works. Call it a second time and absolutely nothing happens. Again no error messages in any consoles. Just... nothing. In your code it looks like everything went swimmingly. But the second call never invokes the function at all. It expires. It's pining for the fjords. Another completely silent error. Thanks chrome! >:(

Fix

The fix is this. One, don't call your callback (for testing or otherwise) until you're ready to return data. Removing callback ('foo') and the first tabs.query fixes that issue. Now you'll just get a silent fail on the last line of check_result when the arg can't be serialized.

So you have to serialize the return data yourself. But calling JSON.stringify doesn't work, it gives an error about circular references. I made a new stringify function to avoid circular refs as described here - but better to use Set than [] for seen. Then serialize the answer, send it back, and bob's your uncle! Client console prints tabs. Finally.

Revised code

# ===== server side, extension background script

def get_tabs (args, sender, callback) :

    args ['windowId'] = sender.tab.windowId  # tabs for caller's window
    def check_result (res) :
        res = stringify (res)  # <-- custom func to avoid circular refs
        callback (res)         # <-- it works!!  glory be hallelujah!

    chrome.tabs.query (args, check_result)