Stack Overflow
Stack Overflow

Questions tagged [singer-io]

For questions regarding singer.io tap and target creation, orchestration and best practices.

Singer describes how data extraction scripts—called “taps” —and data loading scripts—called “targets”— should communicate, allowing them to be used in any to move data from any source to any destination. Send data between databases, web APIs, queues, and just about anything else you can think of.

15 questions
5
votes
2 answers

How to override a stream's primary key properties or incremental replication key in Meltano?

There are times when we want to override a stream's key properties (primary key) or it's incremental replication key. What's the best way to do this in Meltano?
aaronsteers
  • 2,277
  • 2
  • 21
  • 38
3
votes
1 answer

tap-mysql returns no data

I have been using tap-mysql from singer.io Below is my config file: { "host": "localhost", "port": "3306", "user": "root", "password": "password"} This successfully returns schema on --discover and my properties file is: { "streams": [ { …
Parithi
  • 51
  • 3
2
votes
1 answer

How to debug a Singer Tap using VS Code

In creating new Singer Taps using the Meltano SDK, it's not always clear how to setup testing in VS Code. What's the best way to get the VS Code test features working?
aaronsteers
  • 2,277
  • 2
  • 21
  • 38
1
vote
1 answer

With Singer taps and targets, why are columns created in the target even if they are deselected?

When using singer taps and targets, we sometimes see the target creating columns for columns that are explicitly filtered out (deselected) in the tap. Why does this this happen and how can we resolve the issue?
aaronsteers
  • 2,277
  • 2
  • 21
  • 38
1
vote
1 answer

How to provide a relative start date for Meltano CI/CD pipelines on Singer

When we are running a Meltano build/test cycle (such as in a CI/CD pipeline), we want our Singer pipelines to run as follows: Don't use pre-captured state bookmarks that might permit a stream to entirely not have a meaningful run. (For instance, if…
aaronsteers
  • 2,277
  • 2
  • 21
  • 38
1
vote
1 answer

Singer-IO JIRA tap - Not getting records

I am getting only Projects and Versions Stream RECORDS only; Catalog(properties.json) contain 11 streams. How to get other Streams RECORDS also? I run this command: tap-jira -c config.json -p properties.json > finaloutput.json My properties.json is…
Sun
  • 3,444
  • 7
  • 53
  • 83
1
vote
1 answer

singer.io incremental load is not working

Working with singer.io trying to get data from mysql using xampp on Ubuntu and able to fetch historical data but then i tried to get the incremental load the code is not working. Not able to get the replication_key_value if anyone can help i will be…
farhan pirzada
  • 187
  • 8
1
vote
1 answer

Singer.io -> tap-postgres to target-redshift. getting error to move the data to redshift

I am trying tap-postgres to target-redshift. But am getting this error when writing data to redshift. I am reading from tap-postgres. tap-Postgres to target-csv is working fine. tap-postgres -> conf.json { "host": "localhost", "port": 5432, …
Phani Kiran Maddukuri
  • 11
  • 2
1
vote
0 answers

KeyError: '_id' CRITICAL ('Exception writing records', KeyError('_id')) Singer.io

I got this error while doing ELT from mongo-tap to redshift. When I simply extract data from MongoDB to CSV it works fine. but when piping out output to redshift I am getting this _id error. not sure what is the reason. Can anybody guide me More…
Waleed Arshad
  • 1,081
  • 3
  • 12
  • 24
0
votes
0 answers

Debugging and troubleshooting singer-tap development for a SQL source using the Meltano SDK

Does anyone have experience developing SQL singer-taps using the Meltano SDK? Problem My biggest problem is troubleshooting the exceptions I am experiencing when testing my tap. Context I am developing in an Ubuntu environment running on a windows…
Justin Cohen
  • 1
0
votes
1 answer

Where files from tap are kept in Meltano

I have the following combination of tap/target in Meltano: tap-marketo and target-s3-parquet. I want to extract data from tap-marketo from data A to date B in the past. I saw that we can only define start_date and max_export_days. I have tried to…
marz
  • 83
  • 5
0
votes
0 answers

How to activate a new version of postgres-singer-io tap? `key_properties` change detected

After updating a singer.io target-postgres tap to use a new key, I see the following error: target_postgres.exceptions.PostgresError: ('Exception writing records', PostgresError("`key_properties` change detected. Existing values are: ['uri'].…
Michael K
  • 439
  • 3
  • 13
0
votes
1 answer

Which values can singer.write_records() take for its records parameter and how are they processed?

When using the singer.write_records() method we pass a stream id as a string and a record as dict, for example: singer.write_records( stream_name="example_stream", records={"product1": "val1", "product2": "val2", "shop":…
Moritz Wolff
  • 436
  • 1
  • 7
  • 16
0
votes
0 answers

Why is my Pipelinewise ETL not inserting rows into Snowflake?

I'm using the Pipelinewise ETL utility which implements the Singer.io spec. Data is making it from my postgres instance -> s3 -> Snowflake stage but the data is not being inserted into Snowflake tables. Anyone have any ideas? I've…
Ryan Fisher
  • 1,485
  • 1
  • 19
  • 32
-1
votes
1 answer

Failure to run shell script via subprocess.Popen

I am trying to a script run.sh using python subprocess The script creates a csv file and a state.json file This is the script(works fine when i run it from terminal) : #!/bin/bash export…
Shabari nath k
  • 920
  • 1
  • 10
  • 23