Finding the most frequently repeated words in the paragraph in c#

Question

I want to create a words map in the paragraph. For example; my string is "go bread John yesterday going is music musics...". the words map in that string: music=2, go=2, bread=1, John=1, yesterday=1. Word suffixes will be considered as a common root.(example musics=music). How about this topic I do coding in C#?

you'll need a list of valid words for this task. then you can use regex to loop through your word array and count the matches. — Yami, Apr 19 '13 at 14:35
(1) "music" and "musics" have a common prefix, not a common suffix. (2) You should try something and see if you can do it yourself, then post some code to see if someone would help you fix a problem in your code. — Sergey Kalinichenko, Apr 19 '13 at 14:36
What part of this are you having trouble with? Reading the paragraph? Parsing the words? Keeping track of the words? Stemming? Do you have any idea how you're going to approach the problem? — Jim Mischel, Apr 19 '13 at 14:39

score 0 · Accepted Answer · answered Apr 19 '13 at 14:43

0

In regards to the suffix, this just looks for an s, you can modify to look for other suffixes.

string words = "go bread John yesterday going is music musics";
List<string> wordroots = words.Split(new [] {" "}, StringSplitOptions.RemoveEmptyEntries).ToList();
var rootcount = wordroots
    .Select(wr =>
    {
        if (wr.EndsWith("s"))
            wr = wr.Substring(0, wr.Length - 1);
        return wr;
    })
    .GroupBy(g => g);

foreach (var group in rootcount)
    Console.WriteLine(string.Format("Found word: {0} {1} times.", group.Key, group.Count()));

answered Apr 19 '13 at 14:43

Joey Gennari

2,361
17
26

Thank you Joey. Partially solved the problem in your code, such as. But because it must be go of the time 2 because going derive from go. How it can work? – emre_ceylan Apr 19 '13 at 17:43
Joey, I figured out the problem. Your code is right. Other shots just by adding code to the common root of the words may be interpreted as attached. For example, if (wr.EndsWith("ing")) wr = wr.Substring(0, wr.Length - 3); – emre_ceylan Apr 19 '13 at 19:13

score 0 · Answer 2 · answered Apr 19 '13 at 14:45

You can first transform all plural forms to single (or plural) forms, you decide, but be consistent, such that music=musics. This is not hard as there is C# code to get the plural form, see for example this post.

You can then create a dictionary:

Dictionary<string, int> data=new Dictionary<string, int>();
foreach(string item in YourInputs)
{
    if(data.ContainsKey(item)
         data[item]++;
    else
         data.Add(item, 1);
}

score 0 · Answer 3 · edited May 23 '17 at 10:25

You'll first need a word stem library. Snowball suggested in this SO topic seems like a good place to start.

Even with a stemmer, you'll undoubtedly get a pretty massive list of words from a small article, so your best bet to keep track of all these will probably be an SQL Database. However if you only temporarily need to keep track of these values, a simple string table will probably do the trick.

Finding the most frequently repeated words in the paragraph in c#

3 Answers3