Stack Overflow
Stack Overflow

Questions tagged [matillion]

Matillion is a product which transforms businesses' data, across its various locations and forms, into cloud data warehouses to enable informed decision making.

Matillion is a product which transforms businesses' data, across its various locations and forms, into cloud data warehouses to enable informed decision making. https://www.matillion.com/

79 questions
4
votes
2 answers

Does anyone has "Best Practice" to share for Unit/Integration/Regression testing with Snowflake?

We are embarking in a project using both Matillion and Snowflake and want to put in place some Unit/Integration/Regression testing. Automated would be brilliant but manual would be good too. We could invent something (simple) ourselves... but it…
Eric Mamet
  • 2,681
  • 2
  • 13
  • 43
2
votes
1 answer

Snowflake - Column default values not effective with COPY command?

I'm using Matillion to load data into Snowflake, both on Azure. When I create tables I specify default values for the columns, I don't like having NULLs in the warehouse. From what I've read, the Database Query orchestration component in Matillion…
JFS
  • 33
  • 3
2
votes
1 answer

How can I convert a Matillion Job variable of type DateTime to a python datetime.datetime?

I have a Job variable last_updated of type DateTime. I'm trying to use it from a Python Script component, I thought it will show up as datetime.datetime object in python but it isn't : print(latest_updated) # Sat Jan 01 00:00:00 UTC…
RubenLaguna
  • 21,435
  • 13
  • 113
  • 151
1
vote
0 answers

Moving GA4 tables from BigQuery to Snowflake using Matillion ETL

I am trying to move the GA4 data of two websites from BigQuery to a Snowflake table using Matillion ETL. BigQuery tables are named events_YYYYMMDD format. The query I am using in the Matillion BigQuery orchestration job is below: select * from…
Sreeraj
  • 316
  • 1
  • 8
1
vote
1 answer

Looping Through Tables in a DB in Informatica

I am looking for a way in Informatica to pull data from a table in a database, load it in Snowflake, and then move on to the next table in that same DB and repeating that for the remaining tables in the database. We currently have this set up…
Perdue
  • 479
  • 9
  • 19
1
vote
0 answers

is there a away to auto manage database changes in ETL pipeline

The schema of one of the databases that is connected to our business Intelligent data warehouse is constantly evolving (CICD), consequently Each time a change occurred, we need to manually update the ETL pipeline to map the new changes. An example…
ryan
  • 101
  • 9
1
vote
1 answer

(#2635) You are calling a deprecated version of the Ads API. Please update to the latest version

so there have been a bunch of similar questions, though I haven't managed to find the answer in any of them, so here we are. I'm trying to pull some data from the Facebook API using Python, and at here's the error I'm…
H8oddo
  • 123
  • 9
1
vote
3 answers

Matillion: How to identify performance bottleneck

We're running Matillion (v1.54) on an AWS EC2 instance (CentOS), based on Tomcat 8.5. We have developped a few ETL jobs by now, and their execution takes quite a lot of time (that is, up to hours). We'd like to speed up the execution of our jobs,…
dovregubben
  • 364
  • 2
  • 16
1
vote
1 answer

Incremental load from Azure to Snowflake using Matillion

I want to perform incremental load from Azure Blob storage to Snowflake table using Matillion ETL. There is a JDBC incremental load component but it works for a select few databases and Snowflake isn't one of them. I tried searching the…
Coder1990
  • 89
  • 8
1
vote
1 answer

Unable to connect Azure to Matillion

I want to get files from Azure Blob storage and use Matillion ETL tool to write them to Snowflake table. In Matillion, the default is coming as AWS. Very little documentation is available for Azure. My Matillion account is partner connect account…
Coder1990
  • 89
  • 8
1
vote
2 answers

Snowflake check if Schema exists

Snowflake doesn't accept T-sql but i'm trying to Create schema If it doesn't exists. IF NOT EXISTS (SELECT * FROM SYS.SCHEMA WHERE NAME = 'CBS') CREATE SCHEMA 'CBS'; How to do this in Snowflake?
user1810575
  • 823
  • 3
  • 24
  • 45
1
vote
0 answers

How to fix historical table in Redshift db?

I had an issue with updating my historical table in db. I have 3 steps in ETL job: Detect changes - Matillion ETL finds differencies between two tables and writes it into tmp table Close current - this step updates all rows with current_flag = 'Y'…
Anchee
  • 43
  • 3
1
vote
4 answers

Data Ingestion: Load Dynamic Files from S3 to Snowflake

Situation: A csv lands into AWS S3 every month. The vendor adds/removes/modifies columns from the file as they please. So the schema is not known ahead of time. The requirement is to create a table on-the-fly in Snowflake and load the data into said…
ak2021
  • 21
  • 3
1
vote
1 answer

merge two json statements into one using matillion ETL for snowflake

I want to combine these two JSON into one using matillion ETL for snowflake JSON statement 1: [{ "StepId": 1, "ParameterFileGroup": 1, "ParameterGroup": 1, "Parameter": 4, "Filter": "", …
Gowtham_7
  • 217
  • 1
  • 8
1
vote
1 answer

I need to drop a column from a table in matillion-snowflake

I need to drop the column named Conditions before loading it intoenter image description here the target table using matillion ETL for snowflake. however I want to do this in a transformation job. I've attached the image for reference.
user14865552
1
2 3 4 5 6