DelimitedPayloadFilter in PyLucene?

Question

I am trying to implement a python version of the java from http://searchhub.org/2010/04/18/refresh-getting-started-with-payloads/ using pylucene. My analyzer is producing an lucene.InvalidArgsError on the init call to the DelimitedTokenFilter

The class is below, and any help is greatly appreciated. The java version compiled with the JAR files from the pylucene 3.6 build works fine.

import lucene
class PayloadAnalyzer(lucene.PythonAnalyzer):
    encoder = None
    def __init__(self, encoder): 
        lucene.PythonAnalyzer.__init__(self) 
        self.encoder = encoder

    def tokenStream(self, fieldName, reader):
        result = lucene.WhitespaceTokenizer( lucene.Version.LUCENE_CURRENT, reader )
        result = lucene.LowerCaseFilter( lucene.Version.LUCENE_CURRENT, result )
        result = lucene.DelimitedPayloadTokenFilter( result, '|', self.encoder )
        return result

My initial problem with the arguments is solved if I use the default delimiter in place of '|': lucene.DelimitedTokenFilter.DEFAULT_DELIMITER. This doesn't seem to solve my problem though. everything compiles but the payload always comes back as 1.0 . — M C, Nov 17 '12 at 00:53

score 0 · Answer 1 · answered Dec 09 '14 at 05:27

The doc of jcc says:

When JCC sees these special extension java classes it generates the C++ code implementing the native methods they declare. These native methods call the corresponding Python method implementations passing in parameters and returning the result to the Java VM caller.

So you should edit the file java/org/apache/pylucene/search/similarities/PythonDefaultSimilarity.java in pylucene.

Add some code like this:

import org.apache.lucene.util.BytesRef;
public native float scorePayload(int docId, int start, int end, BytesRef payload);

After this, your code can override the method scorePayload.

class PayloadSimilarity(PythonDefaultSimilarity):

    def scorePayload(self, docId, start, end, payload):
        return PayloadHelper.decodeFloat(payload.bytes, end)


class PayloadAnalyzer(PythonAnalyzer):
    encoder = None

    def __init__(self, encoder):
        super(PayloadAnalyzer, self).__init__()
        self.encoder = encoder

    def createComponents(self, fieldName, reader):
        source = WhitespaceTokenizer(Version.LUCENE_44, reader)
        result = LowerCaseFilter(Version.LUCENE_44, source)
        result = DelimitedPayloadTokenFilter(result, u'|', self.encoder)
        return self.TokenStreamComponents(source, result)

I test the code above under pylucene4.8. It works fine.

DelimitedPayloadFilter in PyLucene?

1 Answers1

Linked