4

I am using pentaho data integration for ETL. I am running the job in ubuntu server as a shell script. It is running for some time after that it is getting killed without throwing any error. Please help me what is the problem and tell me if I am missing any.

LOG:

INFO  14-03 11:46:52,369 - set_var - Dispatching started for transformation         [set_variable]
INFO  14-03 11:46:52,370 - Get rows from result - Finished processing (I=0, O=0, R=1, W=1, U=0, E=
INFO  14-03 11:46:52,370 - Set var - Setting environment variables...
INFO  14-03 11:46:52,371 - Set var - Set variable BOOK_PATH to value [...........]
INFO  14-03 11:46:52,371 - Set var - Set variable FOLDER_NAME to value [...........]
INFO  14-03 11:46:52,375 - Set var - Finished after 1 rows.
INFO  14-03 11:46:52,375 - Set var - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
INFO  14-03 11:46:52,377 - validate - Starting entry [file]
INFO  14-03 11:46:52,378 - file - Loading transformation from XML file 
INFO  14-03 11:46:52,386 - file - Dispatching started for transformation [file][file:///c:/check/file.txt]
INFO  14-03 11:46:52,390 - path - Optimization level set to 9.
INFO  14-03 11:46:52,391 - filename - Finished processing (I=0, O=0, R=0, W=13, U=0, E=0)
INFO  14-03 11:46:52,403 - path - Finished processing (I=0, O=0, R=13, W=13, U=0, E=0)
INFO  14-03 11:46:52,407 - filenames - Finished processing (I=0, O=14, R=13, W=13, U=0, E=0)
INFO  14-03 11:46:52,409 - validate - Starting entry [Check_database]
INFO  14-03 11:46:52,410 - Check_database - Loading transformation from XML file[file:///c:/check/missing.ktr]
INFO  14-03 11:46:52,418 - count - Dispatching started for transformation [count]
INFO  14-03 11:46:52,432 - count - Finished reading query, closing connection.
INFO  14-03 11:46:52,433 - Set var - Setting environment variables...
INFO  14-03 11:46:52,433 - count - Finished processing (I=1, O=0, R=0, W=1, U=0, E=0)
INFO  14-03 11:46:52,433 - Set var - Set variable Count to value [0]
INFO  14-03 11:46:52,436 - Set var - Finished after 1 rows.
INFO  14-03 11:46:52,436 - Set var - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
Killed `
Cynosure
  • 161
  • 2
  • 3
  • 9
  • What's in the logs when it dies? Often Kitchen and Pan processes will return 0 status even when the job/transformation errors out. – Gordon Seidoh Worley Mar 05 '13 at 15:09
  • Its showing the progress stucking up with out error getting killed – Cynosure Mar 06 '13 at 10:30
  • Please post the log: it's very difficult to assess the problem with such limited information. – Gordon Seidoh Worley Mar 06 '13 at 13:44
  • I have attached the log file. can you please tell me wheather size of the file will cause this problem – Cynosure Mar 14 '13 at 08:08
  • We were running a JIRA instance in a Win2003 box that started randomly "blinking" (as in, gone). It turned out that when the JVM hits the ceiling of it's it allocated HEAP space, it just evaporates with a note in a system log (it's the JVM's log), not the app log. So after sweating this for a couple of days, we found out the Win2003 box was incapable of running JIRA with the number of concurrent users we were experiencing, so midnight switch to a Win2008RC2 with 8x the memory, we were good. So, look down the chain into the JVM and it's cohorts if you haven't already. – Jared Farrish Mar 14 '13 at 08:20
  • [Maybe this could help.](http://stackoverflow.com/questions/8146152/something-keeps-killing-my-java-process-on-ubuntu-anyone-know-why) – Jared Farrish Mar 14 '13 at 08:21

2 Answers2

3

Most likely your machine unable to provide the memory demand from your transformation and hence silently killing it. You can modify the PDI memory allocation by tweaking the PENTAHO_DI_JAVA_OPTIONS option in your spoon.sh file. I had the same issue, below was what I did:

  • created an environment variable PENTAHO_DI_JAVA_OPTIONS in my system. This variable if not set, kitchen will use its default. Creating this system variable gave me the control to decrease or increase memory allocation as per the transformation complexity (at least for testing on my local machine).
  • My machine had 8G ram, and all the processes including kitchen already used it up. So reduced PDI memory demand by export PENTAHO_DI_JAVA_OPTIONS=-Xms1g -Xmx3g . Min. 1G and Max 3G.

I might be wrong but it worked for me even though the transformation threw GC Memory outage error. At least it was not killed silently.

Did not have to do the above setting in a dedicated standalone server, since no other process except PDI was working in it.

I was using PDI 8.3, iOS BigSur, when the process got killed silently.

Hope this helps someone :).

sarit kumar
  • 113
  • 8
0

most likely you are running out of memory. check your machine's resources while running the ETL.

Dora
  • 1