Hadoop: How can i have an array of doubles as a value in a key-value pair?

Question

I have a problem were i need to aggregate some vectors in order to find some statistics. For example i have vectors of doubles and i need to sum them. My vectors look like this:

      1,0,3,4,5
      2,3,4,5,6
      3,4,5,5,6

My key-value pairs so far are (String,String). But every time i need to add these vectors, i first have to convert them to double arrays, add them up and finally convert the aggregate vector into string. I think it would be a lot faster if i just could have key-value pairs in the form (String,double array). There would be no need to convert them back and forth. My problem is that i cant find a way to have double arrays as value. Is there any easy way rather than create a new custom type?

i have only worked with Text and other primitive types extending Writable. I haven't worked with arrays. — jojoba, Feb 18 '12 at 15:01

kornero · Accepted Answer · 2012-02-18T17:46:34.283

You mean something like this?

Map<String, List<Double>> arrays = new HashMap<String, List<Double>>();

double[] array;
arrays.put("ArrayKey", Arrays.asList(array));

then you could call your map method:

map(String key, String arrayKey) {
    List<Double> value = arrays.get(arrayKey);
}

Also you can serialize your double array, and then deserialize it back:

package test;

import org.apache.commons.codec.binary.Base64InputStream;
import org.apache.commons.codec.binary.Base64OutputStream;

import java.io.*;
import java.util.Arrays;

public class Test {

    public static void main(String[] args) throws IOException, ClassNotFoundException {
        double[] array = {0.0, 1.1, 2.2, 3.3};
        String stringValue = serialize(array);
        map("Key", stringValue);
    }

    public static void map(String key, String value) throws ClassNotFoundException, IOException {
        double[] array = deserialize(value);
        System.out.println("Key=" + key + "; Value=" + Arrays.toString(array));
    }

    public static String serialize(double[] array) throws IOException {
        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        Base64OutputStream base64OutputStream = new Base64OutputStream(byteArrayOutputStream);
        ObjectOutputStream oos = new ObjectOutputStream(base64OutputStream);
        oos.writeObject(array);
        oos.flush();
        oos.close();
        return byteArrayOutputStream.toString();
    }

    public static double[] deserialize(String stringArray) throws IOException, ClassNotFoundException {
        ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(stringArray.getBytes());
        Base64InputStream base64InputStream = new Base64InputStream(byteArrayInputStream);
        ObjectInputStream iis = new ObjectInputStream(base64InputStream);
        return (double[]) iis.readObject();
    }
}

OUTPUT:

Key=Key; Value=[0.0, 1.1, 2.2, 3.3]

Mapping is faster, but serialization will be more usefull if you use nodes and clusters for that (if you need to pass your arrays into another JVM):

 private static class SpeedTest {
        private static final Map<String, List> arrays = new HashMap<String, List>();

        public static void test(final double[] array) throws IOException, ClassNotFoundException {
            final String str = serialize(array);
            final int amount = 10 * 1000;

            long timeStamp = System.currentTimeMillis();
            for (int i = 0; i < amount; i++) {
                serialize(array);
            }
            System.out.println("Serialize: " + (System.currentTimeMillis() - timeStamp) + " ms");

            timeStamp = System.currentTimeMillis();
            for (int i = 0; i < amount; i++) {
                deserialize(str);
            }
            System.out.println("Deserialize: " + (System.currentTimeMillis() - timeStamp) + " ms");

            arrays.clear();
            timeStamp = System.currentTimeMillis();
            // Prepaire map, that contains reference for all arrays.
            for (int i = 0; i < amount; i++) {
                arrays.put("key_" + i, Arrays.asList(array));
            }
            // Getting array by its key in map.
            for (int i = 0; i < amount; i++) {
                arrays.get("key_" + i).toArray();
            }
            System.out.println("Mapping: " + (System.currentTimeMillis() - timeStamp) + " ms");
        }
    }

OUTPUT:

Serialize: 298 ms
Deserialize: 254 ms
Mapping: 27 ms

No. I mean key-value pairs in hadoop, not for a structure. Key-values in hadoop is what a map task passes to a reduce task. — jojoba, Feb 18 '12 at 15:04
hmm..this looks very interesting...i am trying for the last couple of hours to solve my problem..is seems like that what i had done was efficient enough...your answer solves another problem that i may have..those serialize and deserialize functions, are an efficient way of converting strings to double arrays back and forth? (sorry for my previous question, perhaps i didn't write enough hints of my problem) — jojoba, Feb 18 '12 at 17:12
Thanks for your interest. I will take a look of this tomorrow :) — jojoba, Feb 18 '12 at 19:08

Hadoop: How can i have an array of doubles as a value in a key-value pair?

1 Answers1

Linked