I have written a simple piece of code to see exactly how the span_tokenize function works. Documentation for this can be found here: http://www.nltk.org/api/nltk.tokenize.html
Here is my piece of code
import nltk
from nltk.tokenize.api import StringTokenizer
sample = StringTokenizer()
sample.span_tokenize("Hulk is Boss")
The output is as follows:
<generator object StringTokenizer.span_tokenize at 0x113a7b258>
I can't understand why I am getting a generator object as an output when the expected output is tuple(int,int) as can be seen from nltk documentation
A nudge in the right direction is appreciated !!
