4

Appologies if I am asking a inappropraite question but I have been hearing this phrase "Metadata driven" for years but could not ever understand.

Metadata as per my understanding is Data (iformation) about data! I understand this more or less!!

But when I hear "MetaData driven" (specaily in ETL world) could not figure it out exactly what it means.

I have good experience with one ETL tool SSIS, so example in it's context will be easy to unsersatnd.

Pritesh
  • 1,938
  • 7
  • 32
  • 46

1 Answers1

10

Assume you are moving 5 rows from table A to table B and you would like to make sure that only the rows matching a particular criteria are affected. In this case your process depends on data and is, therefore, an example of a data-driven design.

Now, let's imagine you have a few "similar" source and/or target table schemas which are similar in the way you would like to process them but are different in their exact implementation (table name, column names, column data types, or even a DB type: Oracle, MS SQL, Sybase, even a flat file or an XML) so what you would like is to "plug-in" sources and targets, DB connections, etc for a particular ETL during the actual run of the ETL.

What you need is a clear separation of the "logical" ETL process from a "physical" implementation. In other words you would like to have an ETL being described in a generic logical units/terms which are substituted by actual physical ones during its run.

What you get then is a descrption of an ETL process that is generic enough for any situation and gets a proper customization to be run for specific source/target systems based on metadata of those sources and targets - a metada-driven design, which allows you to have a generic "logical" representation of your ETL process that becomes a "physical instantiation" at a run-time.

YePhIcK
  • 5,816
  • 2
  • 27
  • 52
  • @YePhlck +1 for answer. So kind of pluggable ETL logic is metadata driven? Does this phrase has some more meaning (context dependent) in this link http://empoweredholdings.com/2010/05/informatica-abinitio-ssis-owb-pentaho-etl-talend/ we see below mentioned multiple uses of it.  Grew up as a metadata driven, GUI development environment.  especially being metadata driven, the focus was not performance nor parallelism  It was not quite a code-generator, but not quite a metadata driven tool either – Pritesh Aug 02 '12 at 11:23
  • You might use something like this when you are just copying a table from a source system into an ODS. – N West Aug 02 '12 at 17:41
  • Not as much "pluggable" as... Probably the best way to think of metadata is to compare it to math functions: it's a first *derivative* of data :) It's a "data about data" - a description of what your data look like. Once you wrap your head around that you would see that "metadata-driven design" means "design that is based on data about data" – YePhIcK Sep 29 '12 at 19:41