Removing non-ASCII characters from string

Question

I am trying to strip non-ASCII character from strings I am reading from a text file and can't get it to do so. I checked some of the suggestions from posts in SO and other sites, all to no avail.

This is what I have and what I have tried:

String in text file:

2021-03-26 10:00:16:648|2021-03-26 10:00:14:682|MPE->IDC|[10.20.30.40:41148]|203, ?  ?'F?~?^?W?|?8wL?i??{?=kb ?   Y  R?

String read from the file:

"2021-03-26 10:00:16:648|2021-03-26 10:00:14:682|[10.20.30.40:41148]|203,\u0016\u0003\u0001\0?\u0001\0\0?\u0003\u0001'F?\u001e~\u0018?^?W\u0013?|?8wL\v?i??{?=kb\t?\tY\u0005\0\0R?"

Methods to get rid of non-ASCII characters:

Regex reAsciiPattern = new Regex(@"[^\u0000-\u007F]+");  // Non-ASCII characters
sLine = reAsciiPattern.Replace(sLine, "");   // remove non-ASCII chars

Regex reAsciiPattern2 = new Regex(@"[^\x00-\x7F]+");  // Non-ASCII characters
sLine = reAsciiPattern2.Replace(sLine, "");   // remove non-ASCII chars

string asAscii = Encoding.ASCII.GetString(
    Encoding.Convert(
        Encoding.UTF8,
        Encoding.GetEncoding(
            Encoding.ASCII.EncodingName,
            new EncoderReplacementFallback(string.Empty),
            new DecoderExceptionFallback()
            ),
        Encoding.UTF8.GetBytes(sLine)
    )
);

What am I missing?

Thanks.

I think you're confusing ASCII with 'printable'. The characters you have are ascii, just not printable. — Poul Bak, Mar 29 '21 at 01:20

score 4 · Answer 1 · answered Mar 29 '21 at 08:49

This can be done without a Regex using a loop and a StringBuilder:

var sb = new StringBuilder();
foreach(var ch in line) {
   //printable Ascii range
   if (ch >= 32 && ch < 127) { 
       sb.Append(ch);
   } 
}

line = sb.ToString();

Or you can use some LINQ:

line = string.Concat(
  line.Where(ch => ch >= 32 && ch < 127)
);

If you must do this with Regex then the following should suffice (again this keeps printable ASCII only)

line = Regex.Replace(line, @"[^\u0020-\u007e]", "");

Try It Online

If you want all ASCII (including non-printable) characters, then modify the tests to

ch <= 127 // for the loops
@"[^\u0000-\u007f]" // for the regex

score 2 · Answer 2 · answered Mar 29 '21 at 01:13

2

You can use the following regular expression to get rid of all non-printable characters.

Regex.Replace(sLine, @"[^\u0020-\u007E]+", string.Empty);

answered Mar 29 '21 at 01:13

msmolcic

6,407
8
32
56

1

Wouldn't these be covered by the first regular expression I had listed? – NoBullMan Mar 29 '21 at 02:47
Thank you all for responding. I believe as Poul Bak had commented, I had confused non-ascii with non-printable. In the end I found the solution at anothe SO post (https://stackoverflow.com/questions/40564692/c-sharp-regex-to-remove-non-printable-characters-and-control-characters-in-a/40568888). I posted the solution. – NoBullMan Mar 29 '21 at 12:44

score 0 · Answer 3 · answered Mar 29 '21 at 12:43

This is what worked for me based on a post here

using System.Text.RegularExpressions;
...

Regex reAsciiNonPrintable = new Regex(@"\p{C}+");  // Non-printable characters
string sLine;

using (StreamReader sr = File.OpenText(Path.Combine(Folder, FileName)))
{
    while (!sr.EndOfStream)
    {
        sLine = sr.ReadLine().Trim();

        if (!string.IsNullOrEmpty(sLine))
        {
            Match match = reAsciiNonPrintable.Match(sLine);

            if (match.Success)
                continue;    // skip the line
            ...
        }
        ...
    }
    ....
}

Jeppe Stig Nielsen · Answer 4 · 2021-03-29T13:02:54.277

Since a string is an IEnumerable<char> where each char represents one UTF-16 code unit (possibly a surrogate), you can also do:

var ascii = new string(sLine.Where(x => x <= sbyte.MaxValue).ToArray());

Or if you want only printable ASCII:

var asciiPrintable = new string(sLine.Where(x => ' ' <= x && x <= '~').ToArray());

I realize now that this is mostly a duplicate of pinkfloydx33's answer, so go and upvote that.

If the string contains accented letters, the result can depend on the normalization, so compare:

var sLine1 = "olé";
var sLine2 = sLine1.Normalize(NormalizationForm.FormD);

Removing non-ASCII characters from string

4 Answers4