Declare mrjob mapper without ignoring key

Question

I want to declare a mapper function with mrjob. Because my mapper function needs to refer to some constants to do some calculations so I decide to put these constants into the Key in the mapper (Is there any other way?). I read mrjob tutorial on this site but all examples ignore the key. For example:

class MRWordFrequencyCount(MRJob):

def mapper(self, _, line):
    yield "chars", len(line)
    yield "words", len(line.split())
    yield "lines", 1

def reducer(self, key, values):
    yield key, sum(values)

Basically, I'd like something like:

def mapper(self, (constant1,constant2,constant3,constant4,constant5), line):
    My calculation goes here

Please suggest me how to do it. Thank you

vishnu viswanath · Accepted Answer · 2015-11-17T03:05:34.937

You can set your constants in your __init__

from mrjob.job import MRJob

class MRWordFrequencyCount(MRJob):

    def mapper(self, key, line):
        yield "chars", len(line)
        yield "words", len(line.split())
        yield "lines", 1
        yield "Constant",self.constant

    def reducer(self, key, values):
        yield key, sum(values)

    def __init__(self,*args,**kwargs):
        super(MRWordFrequencyCount, self).__init__(*args, **kwargs)
        self.constant = 10


if __name__ == '__main__':
    MRWordFrequencyCount.run()

Output:

"Constant"  10
"chars" 12
"lines" 1
"words" 2

Or, you can use RawProtocol

from mrjob.job import MRJob
import mrjob


class MRWordFrequencyCount(MRJob):
    INPUT_PROTOCOL = mrjob.protocol.RawProtocol

    def mapper(self, key, line):
        yield "constant", key
        yield "chars", len(line)
        yield "words", len(line.split())
        yield "lines", 1

    def reducer(self, key, values):
        if str(key) != "constant":
            yield key, sum(values)
        else:
            yield "constant",list(values)


if __name__ == '__main__':
    MRWordFrequencyCount.run()

if input is :

constant1,constant2,constant3   The quick brown fox jumps over the lazy dog

output:

"chars" 43
"constant"  ["constant1,constant2,constant3"]
"lines" 1
"words" 9

Declare mrjob mapper without ignoring key

1 Answers1