Stack Overflow
Stack Overflow

Questions tagged [airbyte]

25 questions
2
votes
0 answers

docker compose v2 not found

"docker compose v2 not found! please install docker compose!" this massage appear as I trying torun airbyte although I have Docker Compose version 2.18.1 installed I can check it in Terminal also I am working in m1 chip macbook how can I solve this…
Mohammed Salah
  • 21
  • 1
2
votes
1 answer

Deploying Airbyte with just docker-compose.yaml and varibles

I am working on a project with Airbyte. Is pretty easy the deployment with docker compose following the documentation. I just had to clone the repository and use docker compose up. When I clone the repository a lot of files are downloaded. When I…
Marcus Vicentini
  • 43
  • 4
1
vote
0 answers

Is there an efficient way to sync MongoDB documents containing schema and data to SQL-Servers?

Context: I'm currently working on a data pipeline project where the aim is to sync MongoDB documents into a SQL server. However, the schema and data are on separate documents in the form shown below: Schema Document: { schema_id: someHex, …
NeroM1
  • 11
  • 3
1
vote
1 answer

Octavia apply Airbyte gives

I'm trying to create a new BigQuery destination on Airbyte with Octavia cli. When launching: octavia apply I receive: Error: {"message":"The provided configuration does not fulfill the specification. Errors: json schema validation failed when…
tdebroc
  • 1,436
  • 13
  • 28
0
votes
1 answer

How to use dbt and airbyte to cross data from different source?

I set up a small data stack for my company and it includes Airbyte to extract and load data from external services to a BigQuery datawarehouse, and it works well, we have now a massive amount of raw data and started to explore it with our dataviz…
Trent
  • 5,785
  • 6
  • 32
  • 43
0
votes
0 answers

Integrating Airbyte as a multi-container app in the main docker-compose.yml

I am building a data pipeline with Airbyte, PostgreSQL and dbt. PostgreSQL and DBT I can easily set up via my main docker-compose.yml but with Airbyte I am not sure. Airbyte itself is a multi-container app so it has it's own docker-compose.yml. To…
user22316802
  • 1
  • 2
0
votes
1 answer

Unable to run Airbyte on Amazon EC2

I am trying to run Airbyte on EC2, but when I follow the official guideline (https://docs.airbyte.com/deploying-airbyte/on-aws-ec2/) the part with sudo yum install -y docker-compose-plugin docker compose version gives that: docker: 'compose' is not…
BoIde
  • 306
  • 1
  • 3
  • 16
0
votes
0 answers

(Airbyte) MySQL to Snowflake error: connection was unexpectedly lost

I'm using airbyte to transfer data from MySQL database to Snowflake Database. Airbyte is deploy on EC2 t2.medium (CPU usage is between 60 to 70%) with 16GB storage capacity. When the airbyte connection is use to sync only a few table it's working…
user15915737
  • 165
  • 2
  • 15
0
votes
1 answer

open source data stack - Airbyte, Airflow, ?,?

I am building a open source data stack for a large-scale batch pipeline. The data is later to be used in a ML model that is updated quarterly. I want to use Airbyte for ingestion and Airflow for generel orchestration. In general, I want to use…
user22316802
  • 1
  • 2
0
votes
1 answer

Disable Airbyte SSL Verification for custom api connector

I'm setting up a custom connector in an opensource installation of airbyte, to get data from an api. The problem is that the api's ssl certificate is self signed and airbyte is complaining about it. Is there a way to disable ssl verification? ps:…
Felipe Souza
  • 23
  • 4
0
votes
0 answers

Using a KubernetesPodOperator to run a Docker command that uses Airbyte's Salesforce connector to ingest a report

I am using the Salesforce source connector for Airbyte to ingest a report from Salesforce. The ingestion itself can be done with a single Docker command (tested locally and it works): docker run --rm -v…
eulerwizard
  • 1
  • 1
0
votes
1 answer

Missing synced tables in Clickhouse database

I am using Airbyte to sync some tables from a Postgres database >> Clickhouse db. According to Airbyte, the synchronization was successful and my tables were created in the target database in Clickhouse. However, when I try to run queries over my…
brenda
  • 656
  • 8
  • 24
0
votes
0 answers

Airbyte pg to pg failing exactly after 8 gb of data transfer

I am using latest Airbyte source and destination postgres connectors at the time of writing this. When 8 gb of data is transferred, soon after that the sync fails with no error in logs. I tried this 3-4 times, same behaviour each time. I set…
Yusuf
  • 321
  • 1
  • 7
  • 19
0
votes
0 answers

Airbyte is failing replication after 5 or 6 tables

I am working on implementing the Airbyte in my organization but we have a problem that I can not manage to solve. We have a replication from Postgres 11 to Postgres 11 databases using CDC and it works fine for the first 5 or 6 tables but after I…
mbsouksu
  • 78
  • 1
  • 7
0
votes
1 answer

Airbyte - Spark SQL query keeps failing (Updates are in conflict for these columns: data_user_id)

I have coded an airbyte custom destination using python. I have implemented an incremental sync dedup operation using this query, MERGE INTO {self.schema_name}.{table_name} AS target USING {self.schema_name}.{table_name}_temp AS source ON…
Kazim Raza
  • 145
  • 12
1
2