22

In my C# code, I am extracting text from a PDF document. When I do that, I get a string that's in UTF-8 or Unicode encoding (I'm not sure which). When I use Encoding.UTF8.GetBytes(src); to convert it into a byte array, I notice that the whitespace is actually two characters with byte values of 194 and 160.

For example the string "CLE action" looks like

[67, 76, 69, 194 ,160, 65 ,99, 116, 105, 111, 110]

in a byte array, where the whitespace is 194 and 160... And because of this src.IndexOf("CLE action"); is returning -1 when I need it to return 1.

How can I fix the encoding of the string?

Wai Ha Lee
  • 8,598
  • 83
  • 57
  • 92
omega
  • 40,311
  • 81
  • 251
  • 474

3 Answers3

32

194 160 is the UTF-8 encoding of a NO-BREAK SPACE codepoint (the same codepoint that HTML calls  ).

So it's really not a space, even though it looks like one. (You'll see it won't word-wrap, for instance.) A regular expression match for \s would match it, but a plain comparison with a space won't.

To simply replace NO-BREAK spaces you can do the following:

src = src.Replace('\u00A0', ' ');
iCollect.it Ltd
  • 92,391
  • 25
  • 181
  • 202
RichieHindle
  • 272,464
  • 47
  • 358
  • 399
5

Interpreting \xC2\xA0 (=194, 160) as UTF8 actually yields \xA0 which is unicode non-breaking space. This is a different character than ordinary space and thus, doesn't match ordinary spaces. You have to match against the non-breaking space or use fuzzy-matching against any whitespace.

Jonas Schäfer
  • 20,140
  • 5
  • 55
  • 69
4

In UTF8 character value c2 a0 (194 160) is defined as NO-BREAK SPACE. According to ISO/IEC 8859 this is a space that does not allow a line break to be inserted. Normally text processing software assumes that a line break can be inserted at any white space character (this is how word wrap is normally implemented). You should be able to simply do a replace in your string of the characters with a normal space to fix the problem.

Kevin
  • 704
  • 3
  • 4