0

Using

bq --project_id=ny_proj load --source_format=NEWLINE_DELIMITED_JSON my_data.table1 \
sample.json schema.conf

I can load the data into table. However, i have been facing some limitations. In the json file, I have to ensure all the values are given properly in a single line. For instance, details of student_1 should be given in line 1 and student_2 should be in line 2 and so on.

In my case, there is one column called sql_script(which will hold the huge query inside the record). Below content for reference

{"id":"101","frequency":"daily","sql_script":"select * from table where 
description = 'some random is given here for the sake of representation'"}

because of the query length, the single record is jumping into another line. While doing bq load, this is failing.

The below one works, if everything is fit in single line.

{"id":"101","frequency":"daily","sql_script":"select * from table"}
Saravana Kumar
  • 37
  • 6
  • 2
    It's normal, the JSON LINE, is a single line per JSON. You must flatten every single line before performing the load. Else, you have to write your own BigQUery loader – guillaume blaquiere Aug 29 '23 at 13:30

1 Answers1

2

@guillaume is correct, It is stated that the json files to be loaded to BQ is new line delimited.

JSON data must be newline delimited. Each JSON object must be on a separate line in the file.

You can visit this link for more details and list of limitations about loading Json in BigQuery

Nestor Ceniza Jr
  • 976
  • 3
  • 11