0

I am trying to read in a log file using Apache Pig. After reading in the file I want to use my own User Defined Functions in Python. What I'm trying to do is somthing like the following code, but it results in ERROR 1066:Unable to open iterator for alias B, which I have been unable to find a solution for via google.

register 'userdef.py' using jython as parser;
A = LOAD 'test_data' using PigStorage() as (row);
B = FOREACH A GENERATE parser.split(A.row);
DUMP B;

However, if I replace A.row with an empty string '' the function call is completed and no error ocurrs (but the data is not passed nor processed either).

What is the proper way to pass the row of data to the UDF in string format?

kira_codes
  • 1,457
  • 13
  • 38
  • For people who found this post when looking for [ERROR 1066: Unable to open iterator for alias](http://stackoverflow.com/questions/34495085/error-1066-unable-to-open-iterator-for-alias-in-pig-generic-solution) here is a [generic solution](http://stackoverflow.com/a/34495086/983722). – Dennis Jaheruddin Dec 28 '15 at 15:18

1 Answers1

1

You do not need to specify A.row, row alone or $0 should work. $0 is the first column, $1 the second one.

Be careful, PigStorage will automatically split your data if it finds any delimiter, so row may be only first element of each row.

Antony.

AntonyBrd
  • 403
  • 2
  • 10
  • Thanks, I was able to get it working as: `register 'userdef.py' using jython as parser; A = LOAD 'test_data' using PigStorage() as (row:chararray); B = FOREACH A GENERATE parser.split(row); DUMP B;` However, I cannot manipulate the passed data in any way or it breaks. If I just return what is passed in it runs fine. Any ideas? – kira_codes Jun 30 '15 at 14:06
  • I don't know if I really understand what you mean, you should show the output. Maybe, your UDF returns a tuple and you jave to flatten it. – AntonyBrd Jun 30 '15 at 14:10
  • actually it turns out that that I need to cast the passed in value to a string or convert it to an array before I can manipulate it. Not really sure why, but it seems to work. – kira_codes Jun 30 '15 at 14:38