Split sentence into words but having trouble with the punctuations in C#

Question

I have seen a few similar questions but I am trying to achieve this.

Given a string, str="The moon is our natural satellite, i.e. it rotates around the Earth!" I want to extract the words and store them in an array. The expected array elements would be this.

the 
moon 
is 
our 
natural 
satellite 
i.e. 
it  
rotates 
around 
the 
earth

I tried using String.split( ','\t','\r') but this does not work correctly. I also tried removing the ., and other punctuation marks but I would want a string like "i.e." to be parsed out too. What is the best way to achieve this? I also tried using regex.split to no avail.

string[] words = Regex.Split(line, @"\W+");

Would surely appreciate some nudges in the right direction.

The second `.` in `i.e.` and a punctuation `.` are indistinguishable unless you add a list of words that are known to end in `.`. — CodesInChaos, Sep 05 '11 at 18:58
I note that the moon *rotates* around its own axis. It *revolves* around the earth. — Eric Lippert, Sep 06 '11 at 00:04

TheCodeKing · Accepted Answer · 2011-09-06T22:49:33.870

39

A regex solution.

(\b[^\s]+\b)

And if you really want to fix that last . on i.e. you could use this.

((\b[^\s]+\b)((?<=\.\w).)?)

Here's the code I'm using.

  var input = "The moon is our natural satellite, i.e. it rotates around the Earth!";
  var matches = Regex.Matches(input, @"((\b[^\s]+\b)((?<=\.\w).)?)");

  foreach(var match in matches)
  {
     Console.WriteLine(match);
  }

Results:

The
moon
is
our
natural
satellite
i.e.
it
rotates
around
the
Earth

edited Sep 06 '11 at 22:49

answered Sep 05 '11 at 18:55

TheCodeKing

19,064
3
47
70

But isn't that going to include punctuation as part of the word? So in the example above the last word would be "Earth!" ... – Jim Mischel Sep 05 '11 at 19:06
1

No it won't match the punctuation in earth. `\b` matches on word boundaries. – TheCodeKing Sep 05 '11 at 19:16
@Thecodeking, What about matching "i.e."? or something like "u.n.i.c.e.f"? – Richard N Sep 05 '11 at 20:47
Comes out as `u.n.i.c.e.f` or `i.e` :) – TheCodeKing Sep 05 '11 at 21:00
How about "can't"? Won't that come out as "can" and "t"? – Jim Mischel Sep 06 '11 at 18:21
And are you using this with Regex.Split? – Richard N Sep 06 '11 at 22:40
Thanks for the update. Ill mark this as the answer. I will use my existing string split method but will keep this as an option. Clearly, I need to learn more about reg exps. – Richard N Sep 07 '11 at 21:49
Six years later it's still beautiful :) – Lincoln Bergeson Jun 09 '17 at 21:39

score 9 · Answer 2 · answered Sep 05 '11 at 18:57

9

I suspect the solution you're looking for is much more complex than you think. You're looking for some form of actual language analysis, or at a minimum a dictionary, so that you can determine whether a period is part of a word or ends a sentence. Have you considered the fact that it may do both?

Consider adding a dictionary of allowed "words that contain punctuation." This may be the simplest way to solve your problem.

answered Sep 05 '11 at 18:57

Greg D

43,259
14
84
117

Regex does this with `\b` so you don't have to, admittedly there are some grey areas. For instance `i.e.` with match as `i.e`. – TheCodeKing Sep 05 '11 at 19:23

score 1 · Answer 3 · edited Sep 05 '11 at 18:54

1

This works for me.

var str="The moon is our natural satellite, i.e. it rotates around the Earth!";
var a = str.Split(new char[] {' ', '\t'});
for (int i=0; i < a.Length; i++)
{
    Console.WriteLine(" -{0}", a[i]);
}

Results:

 -The
 -moon
 -is
 -our
 -natural
 -satellite,
 -i.e.
 -it
 -rotates
 -around
 -the
 -Earth!

you could do some post-processing of the results, removing commas and semicolons, etc.

edited Sep 05 '11 at 18:54

Uwe Keim

39,551
56
175
291

answered Sep 05 '11 at 18:53

Cheeso

189,189
101
473
713

Would this be the best solution. Would post processing be considered inefficient for cases like these? – Richard N Sep 07 '11 at 21:48

score 1 · Answer 4 · answered Sep 05 '11 at 19:06

1

Regex.Matches(input, @"\b\w+\b").OfType<Match>().Select(m => m.Value)

answered Sep 05 '11 at 19:06

Kirill Polishchuk

54,804
11
122
125

Split sentence into words but having trouble with the punctuations in C#

4 Answers4

Linked