How do I sort data into a Zebra table in Pig?

Question

I am trying to store unsorted data from a CSV into a Zebra table in Pig using TableStorer. Do I need to do an ORDER BY before the store to make sure it's sorted and/or do I need to pass some information to the TableStorer to indicate the sort field?

Answer 1

As per the documentation at : Zebra and Pig in Sorting Data section :

Pig allows you to sort data by ascending or descending order (for more information, see the Pig reference manual). Currently, Zebra supports tables that are sorted in ascending order. Zebra does not support tables that are sorted in descending order; if Zebra encounters a table to be stored that is sorted in descending order, Zebra will issue a warning and store the table as an unsorted table.

So in case you want to save data sorted in descending order , it would be a good idea to sort the data tuples in pig script and then store them to Zebra table. At any time, the data in Pig is a collection of values. The data can always be sorted before saving/storing to destination by doing a simple ORDER BY.

Example:

A = LOAD 'data' AS (a1:int,a2:int,a3:int);

DUMP A;
(1,2,3)
(4,2,1)
(8,3,4)
(4,3,3)
(7,2,5)
(8,4,3)

In this example relation A is sorted by the third field, f3 in descending order. Note that the order of the three tuples ending in 3 can vary.

X = ORDER A BY a3 DESC;

DUMP X;
(7,2,5)
(8,3,4)
(1,2,3)
(4,3,3)
(8,4,3)
(4,2,1)

STORE X INTO 'output' USING org.apache.hadoop.zebra.pig.TableStorer('');