Why are some unintended symbols added to my string?

Question

I wrote a console application which fetches strings from some fields in a Sharepoint list. Then I simply write the strings to console. This works fine for the most fields. There is one MultiLineTextField with RichText enabled where i had to remove all the html-tags, that causes this issue.

Even after all the tags are removed the strings seem to contain question marks which were never added to the string. The most weird thing about this is when I set a breakpoint and look into the string's value there are no question marks, but they suddenly appear on the console output.

The only thing I could think of was to Trim the string. Because sometimes they appear in front of the actual string sometimes they are at the and of it, but never in between.

So this is what I tried:

myString = myString.Trim();
myString = myString.Replace("?",string.Empty);

But this does not solve the issue. Besides this would not be a smart solution in case one of the strings would be supposed to contain question marks. For detailed code please see the link above.

Also Convert.ToBase64String(Encoding.UTF8.GetBytes(myString)) gives me the following output:

4oCLTWVobCwgRWllciwgV2Fzc2VyLCBIYWNrZmxlaXNjaCA=

Smells as an encoding issue. Have a look at the [System.Text.Encoding](http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx) class and check which encoding sharepoint used for that field... — rene, Apr 23 '14 at 07:24

Marc Gravell · Accepted Answer · 2014-04-23T13:06:39.840

5

There are probably some non-printing unicode (or possibly low ASCII) characters in the end of the string. The console has a different encoding, and will often render such as ?. Basically: use the indexer (yourString[n]) or yourString.ToCharArray() to investigate what is actually in the string aroung the location of the ?.

With the edit, we can see that the string has a zero-width space (decimal 8203) at the start:

enter image description here

edited Apr 23 '14 at 13:06

answered Apr 23 '14 at 07:25

Marc Gravell

1,026,079
266
2,566
2,900

Oh bugger, you beat me to it by second! – Steve Pettifer Apr 23 '14 at 07:26
There seems to be nothing around the ?. – チーズパン Apr 23 '14 at 10:00
@bodycountPP then I suggest you post the actual string contents, perhaps in base-64-encoded UTF-8 so that people can reliably reproduce exactly what you are seeing – Marc Gravell Apr 23 '14 at 10:13
@MarcGravell I do not know what you are asking me for. Do you want me to simply show you the console output? – チーズパン Apr 23 '14 at 10:57
@bodycountPP the console output won't help me; I mean the output of `Convert.ToBase64String(Encoding.UTF8.GetBytes(yourString))` – Marc Gravell Apr 23 '14 at 12:41
@MarcGravell It is: "4oCLTWVobCwgRWllciwgV2Fzc2VyLCBIYWNrZmxlaXNjaCA=" also did an edit on the initial post. – チーズパン Apr 23 '14 at 12:59
1

@bodycountPP you have a zero-width space at the start; - the first char is 8203; one of these: http://www.fileformat.info/info/unicode/char/200b/index.htm – Marc Gravell Apr 23 '14 at 13:04
Would iterating the char-array for this 'Zero Width Space' be a suitable solution or is there a more elegant way? – チーズパン Apr 23 '14 at 13:14
@bodycountPP well, to answer that you'd need to know *where it came from*, so that you can think about any other likely characters that you need to strip. But removing it should be pretty simple via replace. – Marc Gravell Apr 23 '14 at 13:18

score 1 · Answer 2 · edited May 23 '17 at 12:02

1

Sounds like you're maybe having a problem with unicode characters. Chances are you're outputting the string as ASCII instead of Unicode. Take a look at this question as it sounds like you may be experiencing the same problem.

edited May 23 '17 at 12:02

Community

1
1

answered Apr 23 '14 at 07:25

Steve Pettifer

1,975
1
19
34

Why are some unintended symbols added to my string?

2 Answers2