Jitterbit4 vs Jitterbit 5+ vs Talend vs Other ETL tools

Question

Whilst I find Jitterbit 4 a fairly powerful tool, I guess that my company and I have kinda maxed out the capabilities of v4 of the thing, or so it seems.

I am trying to keep some now business critical processes alive, and finding that I'm swimming against the tide.

Any experience of improvements to be gained to moving to a later version of Jitterbit that make this route worthwhile, or time to move to a more able platform? I've used in the past Business Objects DM, but I don't think our budget would stretch to that.

I've done some limited research, but I need more information than some generalized blog quotes to form a case for either upgrading, or moving platform.

I'd like to assign multiple automated triggers - for example M-F every 15 minutes, S&S every hour. It would be nice to be able to open more than one project at a time in the IDE.

I have to look after a number of processes which take data from CSV files, or MySQL/MSSQL tables, and upload to Netsuite CRM, or extract data from Netsuite CRM and move to MySQL/MSSQL. (interaction with Netsuite is via SOAP requests using XML) Up until November these processes were generally run perhaps 3 or 4 times a day, but a number of processes now are running at 15 or 5 minute intervals. I've done some optimisation work, but the server is running pretty much at max speed - the limit being that we can update up to 2000 records per hour to Netsuite. And the company wants to do more in 2015.

The limit to Netsuite is absolute - however the problems I am wanting to sort out include better control of logging - I can't seem to turn off logging on bits I don't want or need to be logged. I'd like to be able to open two projects in one IDE, so I can compare code. And I'd like to be able to open the development IDE on one server, but open the admin panel to view the other server - the IDE I use allows only one login.

If Talend or something else can offer these sorts of advantages then perhaps it's the way to go - especially as Jitterbit isn't a skill found in a lot of DevOps here in the UK, but Talend and other things are.

Answer 1

I'm going to start this by saying I really don't have any knowledge about Jitterbit at all so have no real comparison. The other thing to add is that some of the things you want are available in the enterprise licences for Talend but not in the free Talend Open Studio (TOS) edition. If you have absolutely zero budget you could probably get by with TOS and using external scripts to build your jobs and projects and to run them using Cron or some other way of launching the built JARs.

I'll start by talking about what you can do with the enterprise editions of Talend (such as Talend Enterprise Data Integration).

Talend's enterprise editions come with a Talend Administration Centre (TAC) that can be used to schedule jobs on multiple triggers and deploy on chosen Job Execution Servers to run the jobs. It's pretty trivial to set up Cron style triggers to run every 15 minutes M-F and then another one to run every hour Saturday and Sunday. The TAC also provides a centralised reference to all of the Talend cluster's configuration and settings as well as creating users and assigning privileges. You can also see some logging when Talend is configured to use the Activity Monitoring Console (AMC) and then any job level logging can be configured in the job itself and then viewed in the execution history of the task.

I'm not sure what you mean about being able to open two projects at a time to be able to compare code and what you'd use it for but you can open multiple jobs at the same time to look at them. Multiple projects at the same time is a no go. I guess you could install the Studio twice in separate locations with separate work spaces (Talend Studio is based on Eclipse) and then open a project in each and compare them visually. I'm not really sure why you would do that though.

If you're finding that you have lots of processes running that are maxing out your job execution server you can easily add more job execution servers and deploy some of the tasks to the extra job execution server. Unfortunately you can't just add a bunch of job execution servers and have the TAC load balance the work across them. With just TOS you could always just have more commodity machines that you manually deploy prebuilt binaries to and execute (it's just running a binary JAR so they only need a JRE on them). It might be a bit of a pain to organise though.

Talend's enterprise editions also come with some centralised source control in the form of SVN (although quite bastardised) which is useful if you ever intend to add more team members as putting TOS into source control can be a pain.

As for non enterprise specific things Talend generates reasonably performant Java code (has easily matched any of my requirements with essentially no effort in optimisation up to now). For instance I tend to hit around 3 requests per second when dealing with internal network web services. Obviously if Netsuite simply takes a long time to respond to each request then that might not help.

Talend has out of the box connectors to easily connect to all of your mentioned data sources minus Netsuite directly (although there is an unofficial NetSuite connector on the TalendForge) but as with Jitterbit you should be able to easily do XML over SOAP to talk to it.

If I were you I'd download TOS and see if it does what you need as is. If you think you might want some of the enterprise capabilities then they offer a free 30 day trial.

You might want to try and be critical and think about the things you might potentially lose from moving away from Jitterbit as well.