warning C4819: How to find the character that has to be saved in unicode?

Question

I have seen the following warning recently through my VS2010.

Warning 21 warning C4819: The file contains a character that cannot be represented in the current code page (936). Save the file in Unicode format to prevent data loss

c:\boost\vs2010_boost1.49\include\boost\format\alt_sstream_impl.hpp 1

Based on MSDN, the file contains a character that has to be saved in unicode format.

Question: I didn't touch those files myself. Is there a way that I can find those characters and correct it manually. In other words, I don't want to save the source file in unicode format.

Thank you

Copy and paste into notepad++ then change the encoding (Encoding menu) to the wanted codepage. Paste back. — Oded, May 08 '12 at 15:28
You can save a copy as ANSI and compare with the original with your favourite file compare tool. — Álvaro González, May 08 '12 at 15:29
Do I have to recompile boost library? That will be tons of work since those files have to check-in. — q0987, May 08 '12 at 15:37

schellack · Accepted Answer · 2018-05-03T03:28:24.220

54

You can use Notepad++ to find all Unicode characters in a file using a regular expression:

Open your file in Notepad++.
Ensure that you select UTF-8 from the Encoding menu.
Open the search box (use CTRL-F or go to the Search menu and select Find...).
Under Search Mode, select the radio button for Regular expression.
Enter [^\x00-\x7F] in the Find what box and hit the Find Next button to see what you get.

After you find the Unicode character(s), you can remove/change them, change the encoding back to ANSI, and save the file.

You don't have to use Notepad++, of course. The RegEx will work in other text editors, e.g., Sublime Text.

edited May 03 '18 at 03:28

answered May 08 '12 at 15:41

schellack

10,144
1
29
33

This was a big help tracking down a unicode issue in an asp file that I had to save. Thank you! – James Drinkard Jun 07 '13 at 20:24
Using the regex above with TextCrawler saved me having to do this with 1000 files. – Jesse Good Sep 15 '13 at 08:59
2

Also, while using the Find dialog box in Notepad++, use the "Mark All" feature in the Mark tab. – WEFX Oct 01 '14 at 13:54
if possible i would like to give double up-vote for this – V-SHY Mar 23 '15 at 06:13
1

You can also use this in VS – bob0the0mighty Nov 05 '15 at 15:09
You just saved us several hours of digging. Thanks! – Scott Hulme Nov 26 '15 at 14:45
In my case it was because of word `isn’t` in a comment where `’` is unicode char. – Shital Shah Dec 06 '17 at 00:09
@bob0the0mighty How? – John Jun 16 '21 at 01:31
Sorry @John, It's been nearly 6 years and I have no idea anymore. – bob0the0mighty Jun 16 '21 at 15:59
@bob0the0mighty Thank you for your reply. I found the solution. The short-key is `ctrl+F` – John Jun 17 '21 at 01:05
Damn, I misunderstood your question. Yeah, ctrl-f will bring up the find input. shift+ctrl+f will bring up a more advanced find and replace menu that is easier to use if you're working over an entire solution. – bob0the0mighty Jun 17 '21 at 13:49

score 6 · Answer 2 · edited May 23 '17 at 11:47

6

I met this problem in my project and tried to modify all non-unicode characters. But I had to give up and found a another way, as there were too many files with such problem (even though all of them are in comments).

Then I found a quick way to fix this by setting 'system locale'.

Control Panel -> Clock,Language,and Region -> Region and Language -> 
Administrative -> Language for non-Unicode programs -> Change system locale -> English

I think this could fix your problem if your 'system locale' is not English.

https://stackoverflow.com/a/37871883/3148107

edited May 23 '17 at 11:47

Community

1
1

answered Jun 17 '16 at 01:54

claymore

69
1
5

why would it be ok by setting locale to English? This file still contains a non-unicode char – mopodafordeya Aug 22 '16 at 16:16
I guess if locale is not English, non-unicode chars are decoded to some strange characters, which could not be recognized by the compiler. – claymore Aug 23 '16 at 06:57

score 3 · Answer 3 · answered Aug 24 '17 at 22:10

3

You can use the regex shown above in Visual Studio to locate those characters. Here is the Regex: [^\x00-\x7F]

answered Aug 24 '17 at 22:10

user3097514

319
2
6

score 2 · Answer 4 · edited Mar 07 '13 at 03:54

2

at line 176:

BOOST_ASSERT(0); // §27.4.3.2 allows undefined-behaviour here

You see, there is a character before 27 in the above line.

edited Mar 07 '13 at 03:54

dualed

10,262
1
26
29

answered Mar 07 '13 at 03:24

何灵恩

21
1

score 1 · Answer 5 · answered Nov 16 '19 at 13:06

Basically, you might have compiled a file which is encoded in another charset that is not the default charset of your operating system. One way is to delete those characters that can not decoded by your default encoding, but I prefer the other way which is just using Notepad++ to trans-coding your file. To achieve this goal, click Encoding->Convert To xxx, where xxx is your operating system default encoding.

warning C4819: How to find the character that has to be saved in unicode?

5 Answers5

Linked