0

I am still playing with Stanford's CoreNLP and I am encountering strange results on a very trivial test of Coreference resolution.

Given the two sentences :

The hotel had a big bathroom. It was very clean.

I would expect "It" in sentence 2 to be coreferenced by "bathroom" or at least "a big bathroom" of sentence 1.

Unfortunately it point to "The hotel" which in my opinion is wrong.

Is there a way to solve this problem ? Do I need to train anything or is it supposed to work out of the box ?

    Annotation a = getPipeline().getAnnotation("The hotel had a big bathroom. It was very clean.");

    System.out.println(a.get(CorefChainAnnotation.class));

output :

{1=CHAIN1-["The hotel" in sentence 1, "It" in sentence 2], 2=CHAIN2-["a big bathroom" in sentence 1]}

Many thanks for your help.

azpublic
  • 1,404
  • 4
  • 20
  • 42

1 Answers1

5

Like many components in AI, the Stanford coreference system is only correct to a certain accuracy. In the case of coreference this accuracy is actually relatively low (~60 on standard benchmarks in a 0-100 range). To illustrate the difficulty of the problem, consider the following apparently similar sentence with a different coreference judgment:

The hotel had a big bathtub. It was very expensive.

Gabor Angeli
  • 5,729
  • 1
  • 18
  • 29
  • Thanks again Gabor for your help. It seems there's still a lot of work to do in the domain of coreference resolution. Are there any other libraries / algorithms that can do it in a better way the Stanford NLP ? – azpublic May 29 '15 at 06:29
  • 1
    The best coreference system seems to hop back and forth between Stanford and Berkeley (http://nlp.cs.berkeley.edu/projects/coref.shtml) -- cue angry researcher interjecting with a system I've overlooked. Really, this is just a hard problem. So much so that there's a decent amount of interest behind using it as a more accurate type of Turing Test: http://en.wikipedia.org/wiki/Winograd_Schemas_Challenge – Gabor Angeli May 29 '15 at 07:37
  • Okay ...Very Interesting. Cant wait for a real AI to be developed so I'll give berkeley a shot. I'm also looking into nltk to see if its any good. Thanks again for your time ! – azpublic May 29 '15 at 08:05