How can I get the terms of a Lucene document field tokens after they are analyzed?

Question

I'm using Lucene 5.1.0. After Analyzing and indexing a document, I would like to get a list of all the terms indexed that belong to this specific document.

{        
        File[] files = FILES_TO_INDEX_DIRECTORY.listFiles();
        for (File file : files) {
            Document document = new Document();
            Reader reader = new FileReader(file);
            document.add(new TextField("fieldname",reader));            
            iwriter.addDocument(document);
        }  

        iwriter.close();
        IndexReader indexReader = DirectoryReader.open(directory);
        int maxDoc=indexReader.maxDoc();
        for (int i=0; i < maxDoc; i++) {
            Document doc=indexReader.document(i);
            String[] terms = doc.getValues("fieldname");
        }
}

the terms return null. Is there a way to get the saved terms per document?

you can find your answer in this post, [http://stackoverflow.com/questions/5427845/how-can-i-read-a-lucene-document-field-tokens-after-they-are-analyzed?rq=1](http://stackoverflow.com/questions/5427845/how-can-i-read-a-lucene-document-field-tokens-after-they-are-analyzed?rq=1) — Allen Chou, Aug 03 '15 at 14:12
That wasn't exactly what I asked for, But I found the answer by using a TokenStream and charTermAttribute — Yaara Gazit, Aug 04 '15 at 16:30

score 1 · Accepted Answer · answered Aug 04 '15 at 16:34

Here is a sample code for the answer, using a TokenStream

 TokenStream ts= analyzer.tokenStream("myfield", reader);
            // The Analyzer class will construct the Tokenizer, TokenFilter(s), and CharFilter(s),
            //   and pass the resulting Reader to the Tokenizer.
            OffsetAttribute offsetAtt = ts.addAttribute(OffsetAttribute.class);
            CharTermAttribute charTermAttribute = ts.addAttribute(CharTermAttribute.class);

            try {
                ts.reset(); // Resets this stream to the beginning. (Required)
                while (ts.incrementToken()) {
                    // Use AttributeSource.reflectAsString(boolean)
                    // for token stream debugging.
                    System.out.println("token: " + ts.reflectAsString(true));
                    String term = charTermAttribute.toString();
                    System.out.println(term);

                }
                ts.end();   // Perform end-of-stream operations, e.g. set the final offset.
            } finally {
                ts.close(); // Release resources associated with this stream.
            }

How can I get the terms of a Lucene document field tokens after they are analyzed?

1 Answers1