5

VSCode is built on top of MonacoEditor which doesn't support Textmate grammars and themes. But somehow VSCode made it possible. I am curious how VSCode is able to do this.

I am asking because I am making a code editor (based on Monaco) with TextMate grammar and theme support. But I am unable to understand how I can achieve it.

Though there are packages like monaco-textmate to make TextMate grammars work with Monaco, syntax highlighting is still not working properly.

Gama11
  • 31,714
  • 9
  • 78
  • 100
REDDY PRASAD
  • 1,309
  • 2
  • 14
  • 29

1 Answers1

8

TextMate grammars depend on a particular regex implementation / library called Oniguruma, which is implemented in C. Monaco however is designed to run in the browser, and the JavaScript regex engine available there is not compatible with Oniguruma. All of this is explained in detail in the "Why doesn't the editor support TextMate grammars?" section of Monaco's FAQ. It also mentions the possibiliy of perhaps eventually compiling Oniguruma to WebAssembly to work around this.

VSCode itself uses vscode-textmate for its TMLanguage handling, which has the Oniguruma library as a native dependency. VSCode can have native dependencies because it doesn't run in a browser environment.

According to monaco-textmate's readme, it is actually heavily based on vscode-textmate:

99% of the code in this repository is extracted straight from vscode-textmate

And it does use the WASM approach mentioned earlier:

monaco-textmate relies on onigasm package to provide oniguruma regex engine in browsers. onigasm itself relies on WebAssembly.

As to why syntax highlighting doesn't always work as expected with monaco-textmate... I have no idea, I expect this is simply a bug in the implementation. Perhaps wait for a response from the maintainer, the issue you linked is fairly new.

At least conceptually there shouldn't be a reason why it couldn't achieve the same syntax highlighting VSCode does, since it uses the same regex flavour.

Gama11
  • 31,714
  • 9
  • 78
  • 100
  • Thanks for anwser. But i understood why tmgrammers don't work with Monaco. As for i know tmgrammer generate multiple scopes for a token but monaco doesn't support multiple scopes. So, package author ended up picking last scope. I am thinking that might be the reason why highlighting doesn't work properly. But somehow vscode able to solve it. I want to how vscode did it. – REDDY PRASAD May 22 '19 at 10:54
  • 1
    I mean, at the end of the day, VSCode also only uses one of the scopes for highlighting, a token can only have one color after all. It simply uses the most specific scope, I think. – Gama11 May 24 '19 at 09:07
  • @Gama11 Reducing scopes to the top-most scope is weaker though. Suppose a token has scopes `storage.type` and `keyword.declaration`. Monaco would only consider the top most, i.e., `keyword.declaration`. If a theme file only defines a colour for `storage.type`, this token would not be coloured in Monaco, while VSCode would fall back to the previous scope. – Safron Jan 12 '23 at 14:45
  • Having multiple scopes also allows VSCode to define theming rules such as `documentation.block string.quoted`, which will only fire for tokens that have both the scopes `string.quoted` _and_ `documentation.block`. (i.e., for strings enclosed in documentation) I've seen some use cases that require that - or at least would require more effort to work around the fact that multiple scopes wouldn't be supported. – Safron Jan 12 '23 at 14:47