3

I was trying to implement logging in Talend. So I made a job using normal components, I have recorded the error, info and debug through Twarn and Tdie. Using Logcatcher , I am segregating the log into two files of debug and error. This part is working fine.

Now I have made two jobs,

First:- Using trowgenerator generating lines, then sending to tmap and from tmap, I am sending to two twarn components based on some condition.

Second:- A job which has tlogcatcher, t_filterrow and segregating to two files based on some condition.

So I am running the second job in parallel with the first job to capture the logs, but it is not working.

I have tried several options and stuck at this point.

Main idea is to have a common logging component which can be plugged to any job for the logs.

Is there any ways with which it can be achieved ?

Deepan Ram
  • 842
  • 1
  • 10
  • 25
  • can you try to keep tlogcatcher and the logic to separate log into files in one job, and then you include this job (using tRunJob) in your second job and only run second job.. – garpitmzn Jan 26 '17 at 17:23
  • Have tried this option, but not working. Went through the documentation of tRunjob, it says one sub job triggered after another sub job is completed. Trying different options. – Deepan Ram Jan 27 '17 at 10:27

3 Answers3

1

Here are the 3 ways that popped in:

log4j: Enterprise version has log4j built in, which means you can set a debug level in tWarn/tDie and configure log4j to put these into different files. (When using it I found it best to have the warnings as my own info level log, since talends info level is too deep.) I'm sure log4j can be an "hacked" into open studio as well. That would solve your problem. https://exchange.talend.com has many results when searching for log4j, you should take a look.

joblet: If you prefer your current way I see 2 options, for you. First is to try and do create a joblet, however using the same file from multiple jobs is not a good idea and probably results in errors. This is not a nice solution but could work if we only talk about 1 job that needs to produce 2 files.

Project level logging: You can try to enable project level logging but that logs everything into files. It basically means redirect the logcatcher output to a file. Thats 1 file not 2, but at least it works. https://help.talend.com/display/TalendOpenStudioforBigDataUserGuide520EN/2.6+Customizing+project+settings (Stats & Logs)

There really is no nice solution to what you need.

Balazs Gunics
  • 2,017
  • 2
  • 17
  • 24
  • I am looking into the option of Talend Open Studio of log4j, but i would like to clarify one point. I have multiple jobs in my project, when I run anyone, I want to generate 2 files based on info or error. – Deepan Ram Jan 27 '17 at 10:48
  • with log4j you can redirect the logs by their levels into different appenders (1 for info 1 for error 1 for all, etc.) – Balazs Gunics Jan 28 '17 at 11:32
1

Talend has built-in logging available and its very easy to setup. Step 1: create AMC database. Step 2. Set project level properties for logging. Step 3. check AMC database for logs. All errors and tWarns will be logged. Optional Step 4: create reports based on logging data in the AMC database.

You can log to file instead of db, but logging to db allows more robust query and reporting. Also if you run jobs out of the TAC you can join AMC to the TAC db for some really useful reports on your jobs.

You can log:

  • Errors and Warnings
  • Stats (job duration)
  • FlowMeter (data flow metrics)

Screen 1 shows project setup in Talend Studio

enter image description here

Screen 2 shows a query from the AMC db, log table. notice there are two other tables, one for stats, and another for flow. also notice in the row selected there is message from a tWarn.

enter image description here

E LaRoche
  • 1,106
  • 1
  • 7
  • 8
  • Thanks for the explanation. I have used this section, but I would like to build a Log component for Talend which can be used in any job, which creates different level of log files for individual jobs and not for the entire project. – Deepan Ram Jan 27 '17 at 11:18
  • why not log everything, per my example above, then extract what info you need in a report, which can be broken out by Job? This may be simpler than what you are proposing, and you might find the extra info useful. Are you also using the TAC as well? if so you can build great reports by joining between the 2 databases. – E LaRoche Jan 27 '17 at 16:35
  • Because if you want to deploy this onto 100 machine behind firewalls then logDB is not an option. I don't think he has TAC because then log4j would be there, out of the box. – Balazs Gunics Jan 28 '17 at 11:34
0

Thanks all for the extended support. I was able to achieve what i was trying to achieve.

I have created a standard job, which is giving twarn and tdie logs. Then I have a joblet which catches the logs and distributes the logs based on error or info/debug.

Deepan Ram
  • 842
  • 1
  • 10
  • 25
  • I am new to Talend and was looking for same job error framework for it . Looking for joblet which can be dropped to any job whenever we want to catch detailed lob logs and which can be easily read by prod support teams etc.Something which is there in data stage .I tried AMC configuration but there we have different dB tables for same.Refering that you were able to achieve it ,would like to know how can it be achieved and job design reference will be really appreciated. – Constant Learner Mar 22 '20 at 14:50