Can Hadoop ReflectionUtils initialize enums?

Question

I need to pass a list of Enum Bucket types between Mapper and Reducer, I have implemented custom BucketArrayWritable according to implementation-of-an-arraywritable-for-a-custom-hadoop-type, and the Bucket Enum has a no-argument constructor, but I always get the error

java.lang.RuntimeException: java.lang.NoSuchMethodException: Bucket.<init>()
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
    at org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:58)
    at org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:64)
    at org.apache.hadoop.io.ArrayWritable.readFields(ArrayWritable.java:95)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
    at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:139)
    at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKey(ReduceContextImpl.java:114)
    at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.nextKey(WrappedReducer.java:296)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:163)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:610)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:449)
Caused by: java.lang.NoSuchMethodException: com.turn.platform.profile.mapreduce.counting.Bucket.<init>()
    at java.lang.Class.getConstructor0(Class.java:3082)
    at java.lang.Class.getDeclaredConstructor(Class.java:2178)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)

Reducer might want to use reflection to init object, but Enum constructors are private by default,

8.9.2 Enum Body Declarations
In an enum declaration, a constructor declaration with no access modifiers is private.

I don't know wether it is because Hadoop ReflectionUtils cannot find private constructors

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.util.function.Predicate;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Writable;


public enum Bucket implements Writable{

    // right bound exclusive
    MAX((count) -> {
        if (count > getValue()) {
            return true;
        }
        return false;
    }, (count, _userId, incre) -> {
        value = count;
        userId = _userId;
    }),
    TOTAL((count) -> {
        return true;
    }),
    BUCKET_0_10((count) -> {
        if (count < 10) {
            return true;
        }
        return false;
    }),
    BUCKET_10_100((count) -> {
        if (count >= 10 && count < 100) {
            return true;
        }
        return false;
    })
    ;

    private static long value = 0;
    private static LongWritable userId = new LongWritable(0);

    private TriConsumer<Long, LongWritable, Long> consumer;
    private Predicate<Long> predicator;

    Bucket() {

    }

    Bucket(Predicate<Long> predicator) {
        this.predicator = predicator;
    }

    Bucket(Predicate<Long> predicator, TriConsumer<Long, LongWritable, Long> consumer) {
        this.consumer = consumer;
        this.predicator = predicator;
    }

    public static long getValue() {
        return value;
    }

    public static void setValue(long newVal) {
        value = newVal;
    }

    public TriConsumer<Long, LongWritable, Long> getConsumer() {
        if (consumer == null) {
            consumer = (count, _userId, incre) -> {
                setValue(getValue() + incre);
                userId = _userId;
            };
        }
        return consumer;
    }

    public Predicate<Long> getPredicator() {
        return predicator;
    }

    @Override
    public void write(DataOutput out) throws IOException {
        out.writeLong(value);
        userId.write(out);
    }

    @Override
    public void readFields(DataInput in) throws IOException {
        value = in.readLong();
        userId.readFields(in);
    }
}

score 0 · Accepted Answer · edited Jun 20 '20 at 09:12

0

The ReflectionUtils class tries to initialize the enum by reflection, but enums cannot be instantiated this way, this is stated in the Java Language Specification, and like this answer says, is mainly to allow the use of == to compare enums. With reflection you can only obtain the existing references.

Regarding to the use of enums, you could 'wrap' the enum in a class, this way you can override the equals and hashcode method that is normally used in the reducers. Also depending on what you are doing WritableUtils has the methods writeEnum and readEnum. You can check more about how implement the wrapper class in this answer.

edited Jun 20 '20 at 09:12

Community

1
1

answered Mar 29 '18 at 21:51

Jose Da Silva Gomes

3,814
3
24
34

Hi, thanks for your answer. I actually have read all of those answers. I tried to use wrapper class to hold enums, but enums are static from [answer](https://stackoverflow.com/questions/663834/in-java-are-enum-types-inside-a-class-static), and all instances of the wrapper class share the same enums, which is not I want, do you have any ideas? Thanks – Han Yu Mar 30 '18 at 17:40
From [answer](https://stackoverflow.com/questions/2798485/java-objects-shared-variables), it seems like all instance of the wrapper class will share the same enums, so I think it is possible to pass enum between mapper and reducers – Han Yu Mar 30 '18 at 17:48
The first answer refers to nested enums, yes they are static. You should only use the enum to `difference` the class, your `Writable` is stateful, so you have to implement it in the class and then use the variables that you want for each instance, e.g. value shouldn't be static – Jose Da Silva Gomes Mar 30 '18 at 17:49
Sorry, last comment should be impossible to pass enum as a class between mapper and reducer – Han Yu Mar 30 '18 at 18:05

Can Hadoop ReflectionUtils initialize enums?

1 Answers1