3

I hear the term "ETL tool" used a lot lately and started digging in to learn more about them. I found a great list of open source ETL tools for Java here and started reading up on them.

But that made me really confused.

Most of these tools (CloverETL, Pentaho, etc.) are GUI tools. Some of them, such as Smooks, are pure Java frameworks. I guess this makes sense: some ETL users may be non-technical and/or would like to use a GUI tool to set up transformations. Other users will be developers who want to tap into the raw ETL power of these tools.

My question(s): are there any benefits (additional features, etc.) that these GUI tools offer over the pure Java frameworks, or vice versa? Do the "major player" GUI tools such as CloverETL and Pentaho - which bill themselves as GUI tools - also have Java APIs that I can accomplish the same things with (programmatically)? Or are they pure GUI tools? I can't find Java Docs anywhere (for either one).

IAmYourFaja
  • 55,468
  • 181
  • 466
  • 756
  • 1
    Pentaho Kettle can be used both as GUI tool and Java framework. Check out the link for Java API examples: http://wiki.pentaho.com/display/EAI/Pentaho+Data+Integration+-+Java+API+Examples – Raihan Feb 01 '12 at 21:40
  • Just to comment on terminology - most of the big ETL tools (Informatica, DataStage, Ab Initio) are not Java-based and have a bespoke Client/Server architecture...the phrase ETL Tool generally incorporates both elements. I know you're specifically talking about Java-based ETL solutions, but just wanted to clarify that this is a small subset of ETL Tools (Check out http://www.gartner.com/technology/reprints.do?id=1-17QG4XL&ct=111020&st=sb). – bob Feb 02 '12 at 13:11
  • Agree with Raihan. Additionally, it can be used as a CLI tool (pan.sh). We use Pentaho Data Integration Designer (Spoon) to design transformations and run them in batches using Pan CLI tool. – Hendy Irawan Feb 28 '12 at 14:11

1 Answers1

3

I would say that no, there's really no real advantage in using a non-GUI tool for ETL.

In most typical situations, a GUI approach is much more efficient for ETL jobs because the tools should offer you a way to do data task rapidly and almost without writing custom code. That's because an ETL platform is by philosophy no more than a code generation platform: the task drawn on the canvas is translated (ideally, in the most suitable way) by the tool engine in machine code that's directly executed under the hood, without intermediate agents. The bigger ones have a complex client-server architecture, but the basic idea stays the same.

How deeply this generated code is hidden, it depends by the platform. Some, like Pentaho or Datastage, really make it inaccessible to the user; others, like Talend (which produces java code in a class easily embeddable in application or executed directly) or SAS Data Integration Studio (which produces a .sas file) gives the developer the possibility to dig into the generated code. But it's always an option left to the hardcore developer, while the regular user will almost never go inside the code to do her everyday job.

Gabriele B
  • 2,665
  • 1
  • 25
  • 40