Pig script to get top 3 data in a single record

Question

I have the sample data as

user_id, date, accessed url, session time
the data refers to the top 3 interests of the user depending on the session time.

Got the data using the code:

top3 =  FOREACH DataSet{
    sorted = ORDER DataSet BY sessiontime DESC;
    lim    = LIMIT sorted 3;
    GENERATE flatten(group), flatten(lim);
};

Output:

    (1,20,url1,2484)
    (1,20,url2,1863)
    (1,20,url3,1242)
    (2,22,url4,484)
    (2,22,url5,63)
    (2,22,url6,42)
    (3,25,url7,500)
    (3,25,url8,350)
    (3,25,url9,242)

But I want my output to be like this:

(1,20,url1,url2,url3)
(2,22,url4,url5,url6)
(3,25,url7,url8,url9)

Please help.

Please provide a sample of the input, as well as the code that you used to group the sets. Without that it is difficult to tell what is going on. — Davis Broda, Oct 17 '13 at 13:41

score 0 · Accepted Answer · answered Oct 17 '13 at 17:02

0

You are close. The problem is that you FLATTEN the bag of URLs when you really want to keep them all in one record. So do this instead:

top3 =  FOREACH DataSet{
    sorted = ORDER DataSet BY sessiontime DESC;
    lim    = LIMIT sorted 3;
    GENERATE flatten(group), lim.url;
};

Based on the output you got, you will now get

(1,20,{(url1),(url2),(url3)})
(2,22,{(url4),(url5),(url6)})
(3,25,{(url7),(url8),(url9)})

Note that the URLs are contained inside a bag. If you want to have them as three top-level fields, you will need to use a UDF to convert a bag into a tuple, and then FLATTEN that.

answered Oct 17 '13 at 17:02

reo katoa

5,751
1
18
30

Can you be more specific regarding the UDF? – sravani malla Oct 18 '13 at 10:26
See, for example -- http://stackoverflow.com/questions/18544602/how-to-flatten-a-group-into-a-single-tuple-in-pig/18548696#18548696 – reo katoa Oct 18 '13 at 13:24
is there any UDF in java for this? – sravani malla Oct 22 '13 at 05:30
See the answer to this question - http://stackoverflow.com/questions/19057223/using-pig-how-do-i-parse-a-mixed-format-line-into-tuples-and-a-bag-of-tuples/19058408#19058408 – reo katoa Oct 22 '13 at 11:14

Pig script to get top 3 data in a single record

1 Answers1