C# Imap Sort command with special characters

Question

I'm working on a problem with imap sort extention:

My command is the following:

  var query = "icône";
            //byte[] bytes = Encoding.Default.GetBytes(query);
            //query = Encoding.UTF8.GetString(bytes);

            var command = "SORT (REVERSE ARRIVAL) UTF-8 " + "{" + query.Length + "}";
            var imapAnswerString = client.Command(command);
            imapAnswerString = client.Command(query);

I get the following error: BAD Error in IMAP command SORT: 8bit data in atom

I found this: C# Imap search command with special characters like á,é

But I don't see how to prepare my code to send this request sucessfully.

Answer 1

If you want to stick with MailSystem.NET, the answer that arnt gave is correct.

However, as I point out here (and below for convenience), MailSystem.NET has a lot of architectural design problems that make it unusable.

If you use an alternative open source library, like MailKit, you'd accomplish this search query far more easily:

var query = SearchQuery.BodyContains ("icône");
var orderBy = new OrderBy[] { OrderBy.ReverseArrival };
var results = folder.Search (query, orderBy);

Hope that helps.

Architectural problems in MailSystem.NET include:

MailSystem.NET does not properly handle literal tokens - either sending them (for anything other than APPEND) or for receiving them (for anything other than the actual message data in a FETCH request). What none of the authors seem to have noticed is that a server may choose to use literals for any string response.

What does this mean?

It means that the server may choose to respond to a LIST command using a literal for the mailbox name.

It means that any field in a BODYSTRUCTURE may be a literal and does not have to be a quoted-string like they all assume.

(and more...)

MailSystem.NET, for example, also does not properly encode or quote mailbox names:

Example from MailSystem.NET:

public string RenameMailbox(string oldMailboxName, string newMailboxName)
{
    string response = this.Command("rename \"" + oldMailboxName + "\" \"" + newMailboxName + "\"");
    return response;
}

This deserves a Jean-Luc Picard and Will Riker face-palm. This code just blindly puts double-quotes around the mailbox name. This is wrong for at least 2 reasons:

1. What if the mailbox name has any double quotes or backslashes? It needs to escape them with \'s.
2. What if the mailboxName has non-ASCII characters or an &? It needs to encode the name using a modified version of the UTF-7 character encoding.

Most (all?) of the .NET IMAP clients I could find read the entire response from the server into 1 big string and then try and parse the response with some combination of regex, IndexOf(), and Substring(). What makes things worse is that most of them were also written by developers that don't know the difference between unicode character counts (i.e. string.Length) and octets (i.e. byte counts), so when they try to parse a response to a FETCH request for a message, they do this after parsing the "{}" value in the first line of the response:

int startIndex = response.IndexOf ("}") + 3;
int endIndex = startIndex + octets;

string msg = response.Substring (startIndex, endIndex - startIndex);

The MailSystem.NET developers obviously got bug reports about this not working for international mails, so their "fix" was to do this:

public string Body(int messageOrdinal)
{
    this.ParentMailbox.SourceClient.SelectMailbox(this.ParentMailbox.Name);

    string response = this.ParentMailbox.SourceClient.Command("fetch "+messageOrdinal.ToString()+" body", getFetchOptions());
    return response.Substring(response.IndexOf("}")+3,response.LastIndexOf(" UID")-response.IndexOf("}")-7);
}

Essentially, they assume that the UID key/value pair will come after the message and use that as a hack-around for their incompetence. Unfortunately, adding more incompetence to existing incompetence only multiplies the incompetence, it doesn't actually fix it.

The IMAP specification specifically states that the order of the results can vary and that they may not even be in the same untagged response.

Not only that, but their FETCH request doesn't even request the UID value from the server, so it's up to the server whether to return it or not!

TL;DR

How to Evaluate an IMAP Client Library

1. The first thing you should do when evaluating an IMAP client library implementation is to see how they parse responses. If they don't use an actual tokenizer, you can tell right off the bat that the library was written by people who have no clue what they are doing. That is the most sure-fire warning sign to STAY AWAY.

2. Does the library handle untagged ("*") responses in a central place (such as their command pipeline)? Or does it do something retarded like try and parse it in every single method that sends a command (e.g. ImapClient.SelectFolder(), ImapClient.FetchMessage(), etc)? If the library doesn't handle it in a central location that can properly deal with these untagged responses and update state (and notify you of important things like EXPUNGE's), STAY AWAY.

3. If the library reads the entire response (or even just the "message") into a System.String, STAY AWAY.

Answer 2

You're almost there. Your final command should be something like

x sort (reverse arrival) utf-8 subject {6+}
icône

ie. you're just missing a search term to describe where the IMAP server should search for icône and sort the results. There are many other search keys, not just subject. See RFC3501 page 49 and following pages.

Edit: The + is needed after the 6 in order to send that as a single command (but requires that the server support the LITERAL+ extension). If the server doesn't support LITERAL+, then you will need to break up your command into multiple segments, like so:

C: a001 SORT (REVERSE ARRIVAL) UTF-8 SUBJECT {6}
S: + ok, whenever you are ready...
C: icône
S: ... <response goes here>

Answer 3

Thanks all for your answer.

Basically the way MailSystem.net sends requests (Command method) is the crux of this problem, and some others actually.

The command method should be corrected as follows:

First, when sending the request to imap, the following code works better than the original one:

 //Convert stuff to have to right encoding in a char array
            var myCommand = stamp + ((stamp.Length > 0) ? " " : "") + command + "\r\n";
            var bytesUtf8 = Encoding.UTF8.GetBytes(myCommand);
            var commandCharArray = Encoding.UTF8.GetString(bytesUtf8).ToCharArray();

#if !PocketPC
            if (this._sslStream != null)
            {


                this._sslStream.Write(System.Text.Encoding.UTF8.GetBytes(commandCharArray));
            }
            else
            {
                base.GetStream().Write(System.Text.Encoding.UTF8.GetBytes(commandCharArray), 0, commandCharArray.Length);
            }
#endif

#if PocketPC
                        base.GetStream().Write(System.Text.Encoding.UTF8.GetBytes(commandCharArray), 0, commandCharArray.Length);
#endif

Then, in the same method, to avoid some deadlock or wrong exceptions, improve the validity tests as follows:

if (temp.StartsWith(stamp) || temp.ToLower().StartsWith("* " + command.Split(' ')[0].ToLower()) || (temp.StartsWith("+ ") && options.IsPlusCmdAllowed) || temp.Contains("BAD Error in IMAP command"))
                    {
                        lastline = temp;
                        break;
                    }

Finally, update the return if as follows:

if (lastline.StartsWith(stamp + " OK") || temp.ToLower().StartsWith("* " + command.Split(' ')[0].ToLower()) && !options.IsSecondCallCommand || temp.ToLower().StartsWith("* ") && options.IsSecondCallCommand && !temp.Contains("BAD") || temp.StartsWith("+ "))
                return bufferString;

With this change, all commands work fine, also double call commands. There are less side effects than with the original code.

This resolved most of my problems.