What you want to do is writing an algorithm that compresses strings. However, this isn't really something you can do by just snapping your fingers.
You should first consider the type of raw input you're receiving: is it humanly comprehensible text? Then my advice is to use a pattern-searching algorithm, which looks for repeating sequences of characters and marks their position, using instead their coordinates from that point on. That's what most file-compression algorithms, like WinRar's, mainly do. To decode you then iterate the compressed string, look for coordinate headers and use them to re-ensemble the whole thing.
EDIT: Also, your solution is wrong because the string parameter for byte[].ToString(string) isn't the variable you want to assign, but rather a format. You should do something on the lines of:
private void encodeToolStripMenuItem_Click(object sender, EventArgs e)
{
UTF8Encoding utf8 = new UTF8Encoding();
string textstring;
string encodedString;
textstring = richTextBox1.Text;
byte[] encodedBytes = utf8.GetBytes(textstring);
richTextBox1.Clear();
encodedString = encodedBytes.ToString();
richTextBox1.Text = encodedString;
}
But I doubt that will make the string significantly shorter.