Stack Overflow
Stack Overflow

Questions tagged [meltano]

Meltano integrates best-in-class open source components and enables teams to collaborate on data projects and pipelines more efficiently and with higher confidence. The focus is bringing these qualities to the first step in any data journey: integration and transformation, aka EL(T): Extract, Load, Transform.

22 questions
7
votes
1 answer

Using dbt and Meltano, how can I prevent multiple dbt job runs from conflicting with each other?

When running dbt jobs in Meltano, dbt run jobs may collide with each other if run out of a triggered context - for instance, when an on-demand job collides with a scheduled job or a CI-based job. If dbt run operates on the same tables at the same…
aaronsteers
  • 2,277
  • 2
  • 21
  • 38
5
votes
2 answers

How to override a stream's primary key properties or incremental replication key in Meltano?

There are times when we want to override a stream's key properties (primary key) or it's incremental replication key. What's the best way to do this in Meltano?
aaronsteers
  • 2,277
  • 2
  • 21
  • 38
3
votes
1 answer

How to kill a running EL pipeline in Meltano

When running an Extract/Load pipeline with Meltano, what's the best way (or ways) to kill a running job? Generally these would be run via Airflow but would be nice to have a process that worked also with bare meltano elt and/or meltano run…
aaronsteers
  • 2,277
  • 2
  • 21
  • 38
3
votes
1 answer

How to manually edit STATE tracking in Meltano?

There are several scenarios where we it would be useful to manually edit job state in Meltano. For example, if a table is accidentally deleted in the target SQL warehouse, the user may want to remove the incremental bookmark for just that table…
aaronsteers
  • 2,277
  • 2
  • 21
  • 38
2
votes
1 answer

How to debug a Singer Tap using VS Code

In creating new Singer Taps using the Meltano SDK, it's not always clear how to setup testing in VS Code. What's the best way to get the VS Code test features working?
aaronsteers
  • 2,277
  • 2
  • 21
  • 38
2
votes
1 answer

Unable to connect to AWS redshift via Meltano

hope you all are doing great. Can any one support me with an issue regarding Meltano? I want to perform EL (Extract Load) where I extract data from shopify and load it to AWS Redshift. To extract data from Shopify, I am using tap-shopify extractor…
Zaki Khurshid
  • 33
  • 5
2
votes
1 answer

apache superset query error on datetime with timezone column

I have a table with a column of type TIMESTAMP WITH TIME ZONE and I try to simply SELECT * FROM table, but I get this message: postgresql error: '>=' not supported between instances of 'datetime.timedelta' and 'int'` Am I entering the data…
Erik
  • 2,782
  • 3
  • 34
  • 64
1
vote
1 answer

Meltano with MWAA

I have an AWS MWAA environment to run an ETL Pipeline, and I was planning to consider using Meltano in it. I seem to find no example(s) or documentation that is specific to MWAA. So, I was wondering if that is possible? also, I know that its…
b-onmo
  • 13
  • 3
1
vote
1 answer

Meltano: tag-postgres to target-postgres: data type such as uuid is converted to varchar

I'm actually working on a Meltano project where I have to extract data from one postgres database and load to the final warehouse (also a postgres database) using "Key-based Incremental Replication". After the loading process of meltano, all table…
Dev024
  • 13
  • 3
1
vote
1 answer

Running Meltano commands raises an error: __init__() got an unexpected keyword argument 'request_timeout'

Running Meltano commands raises an error: __init__() got an unexpected keyword argument 'request_timeout' This is happening on a simple: $ meltano state clear # OR $ meltano elt
ricardossr
  • 11
  • 1
1
vote
1 answer

With Singer taps and targets, why are columns created in the target even if they are deselected?

When using singer taps and targets, we sometimes see the target creating columns for columns that are explicitly filtered out (deselected) in the tap. Why does this this happen and how can we resolve the issue?
aaronsteers
  • 2,277
  • 2
  • 21
  • 38
1
vote
1 answer

How to provide a relative start date for Meltano CI/CD pipelines on Singer

When we are running a Meltano build/test cycle (such as in a CI/CD pipeline), we want our Singer pipelines to run as follows: Don't use pre-captured state bookmarks that might permit a stream to entirely not have a meaningful run. (For instance, if…
aaronsteers
  • 2,277
  • 2
  • 21
  • 38
1
vote
1 answer

Singer-IO JIRA tap - Not getting records

I am getting only Projects and Versions Stream RECORDS only; Catalog(properties.json) contain 11 streams. How to get other Streams RECORDS also? I run this command: tap-jira -c config.json -p properties.json > finaloutput.json My properties.json is…
Sun
  • 3,444
  • 7
  • 53
  • 83
0
votes
0 answers

Debugging and troubleshooting singer-tap development for a SQL source using the Meltano SDK

Does anyone have experience developing SQL singer-taps using the Meltano SDK? Problem My biggest problem is troubleshooting the exceptions I am experiencing when testing my tap. Context I am developing in an Ubuntu environment running on a windows…
Justin Cohen
  • 1
0
votes
0 answers

Poetry configuration error when adding Meltano tap-csv extractor

I'm attempting to add the tap-csv extractor to my Meltano project in a virtual environment but run into the following error when running meltano add extractor tap-csv: RuntimeError The Poetry configuration is invalid: - Additional properties…
Danial B
  • 1
1
2