BigQuery removes fields with Postgres Array during datastream ingestion

Question

I have this table named student_classes:

| id  | name     | class_ids  |
| ----| ---------| -----------|
| 1   | Rebecca  | {1,2,3}    |
| 2   | Roy      | {1,3,4}    |
| 3   | Ted      | {2,4,5}    |

name is type text / string
class_ids is type integer[]

I created a datastream from PostgreSQL to BigQuery (following these instructions), but when I looked at the table's schema in BigQuery the class_ids field was gone and I am not sure why.

I was expecting class_ids would get ingested into BigQuery instead of getting dropped.

Please provide enough code so others can better understand or reproduce the problem. — Community, Nov 12 '22 at 10:19

score 1 · Answer 1 · answered Mar 01 '23 at 12:27

1

Official docs list this as a known limitation:

"Datastream doesn't support columns of the Array data types." https://cloud.google.com/datastream/docs/sources-postgresql#postgresqlknownlimitations

answered Mar 01 '23 at 12:27

petrusgomes

317
3
12

BigQuery removes fields with Postgres Array during datastream ingestion

1 Answers1