TextMate grammars depend on a particular regex implementation / library called Oniguruma, which is implemented in C. Monaco however is designed to run in the browser, and the JavaScript regex engine available there is not compatible with Oniguruma. All of this is explained in detail in the "Why doesn't the editor support TextMate grammars?" section of Monaco's FAQ. It also mentions the possibiliy of perhaps eventually compiling Oniguruma to WebAssembly to work around this.
VSCode itself uses vscode-textmate for its TMLanguage handling, which has the Oniguruma library as a native dependency. VSCode can have native dependencies because it doesn't run in a browser environment.
According to monaco-textmate's readme, it is actually heavily based on vscode-textmate:
99% of the code in this repository is extracted straight from vscode-textmate
And it does use the WASM approach mentioned earlier:
monaco-textmate
relies on onigasm
package to provide oniguruma
regex engine in browsers. onigasm itself relies on WebAssembly
.
As to why syntax highlighting doesn't always work as expected with monaco-textmate... I have no idea, I expect this is simply a bug in the implementation. Perhaps wait for a response from the maintainer, the issue you linked is fairly new.
At least conceptually there shouldn't be a reason why it couldn't achieve the same syntax highlighting VSCode does, since it uses the same regex flavour.