How can I find Unicode characters that are not UTF-8 in Visual Studio Code?

Question

I have Unicode characters that I can't see that are not UTF-8, and I need to spot them.

I used the extension Highlight Bad Chars (Kevin Wenger), but it's not sufficient. In particular, I don't know which these characters are and I don't want to have to define them in advance.

How can I do this with Visual Studio Code?

Note: there are instances where you do not want to spot some Unicode non-utf8 characters: https://github.com/microsoft/vscode/issues/140960 — VonC, Feb 17 '22 at 01:06
Related (the same underlying cause for the problem): *[Compilation error: stray ‘\302’ in program, etc](https://stackoverflow.com/questions/19198332/)* — Peter Mortensen, Apr 25 '23 at 17:31

score 55 · Answer 1 · edited Apr 25 '23 at 17:17

55

Find [^\x00-\x7f] and check Use Regular Expression.

It is taken from Finding Those Pesky Unicode Characters in Visual Studio.

edited Apr 25 '23 at 17:17

Peter Mortensen

30,738
21
105
131

answered Jul 01 '21 at 07:22

pvilas

812
6
7

User · Accepted Answer · 2023-02-23T22:03:45.027

35

You can try the Gremlins extension which I found better than Highlight Bad Chars (Kevin Wenger) (at least, Gremlins worked out of the box; I couldn't get Highlight Bad Chars to highlight anything).

edited Feb 23 '23 at 22:03

answered Dec 22 '20 at 07:59

User

62,498
72
186
247

2

It's open source: https://github.com/nhoizey/vscode-gremlins – JinSnow Jan 21 '22 at 07:59
2

Highlight Bad Chars was updated on 11/2/2021, so part of the answer no longer holds true. – gargoylebident Mar 26 '22 at 23:40
2

IMO, Gremlins is a very targeted and well-done plugin. – jpt Jun 17 '22 at 11:30
very cool plugin, thanks for the headsup – Jonathan Laliberte Dec 06 '22 at 18:54

score 8 · Answer 3 · edited Apr 25 '23 at 17:28

In Stable Build v1.63 there is a new method for highlighting various Unicode characters that might otherwise be difficult to spot in your code. These are the new settings:

You can use these colorCustomizations to change the default orange borders:

{
  "workbench.colorCustomizations": {

    "editorUnicodeHighlight.border": "#00ff37",
    "editorUnicodeHighlight.background": "#f00",        // will be in vscode v1.66

    // "minimap.unicodeHighlight": "#ff0000",           // removed in v1.64
    // "editorOverviewRuler.unicodeForeground": "#ff0000"    // removed in v1.64
}

Apparently, indicators for these Unicode warnings will not be shown in the minimap or overview ruler as of v1.64, see Consider removing the Unicode highlight scroll bar decoration

Here is how a Unicode zero-width space (U+200B) appears with these settings:

The zero-width character is an invisible Unicode character controlled by the Unicode Highlight: Invisible Characters setting above.

From v1.63 release notes: Unicode highlighting

Read this article or this article for how invisible or confusable Unicode characters can be used in Unicode spoofing attacks.

Note that this feature does not detect all possible Unicode spoofing attacks, as this depends on your font and locale settings. Also, the detection of ambiguous characters is done heuristically. To be on the safe side, the restricted mode of the workspace trust should be used to review source code, as all non-ASCII characters are highlighted in untrusted workspaces.

The settings editor.unicodeHighlight.invisibleCharacters, editor.unicodeHighlight.ambiguousCharacters or editor.unicodeHighlight.nonBasicASCII can be set to false to disable the corresponding feature.

Individual characters can be excluded from being highlighted and characters in comments or text and markdown documents are not highlighted by default.

How can I find Unicode characters that are not UTF-8 in Visual Studio Code?

3 Answers3

Linked