BigQuery howto load data from local file as content

Question

I have a requirement where in I will receive file content which I need to load to BigQuery tables. Standard API shows how to load data from local file but I don't see any variant of the load method which accepts file content as string rather than a file path. Any idea how I can achieve this ?

Answer 1

As we can see in the source code and official documentation load function loads data only from a local file or Storage File. Allowed options are:

• AVRO,
• CSV,
• JSON,
• ORC,
• PARQUET

The load job is created and it will run your data load asynchronously. If you would like instantaneous access to your data, insert it using Table insert function, where you need to provide the rows to insert into the table:

// Insert a single row
table.insert({
  INSTNM: 'Motion Picture Institute of Michigan',
  CITY: 'Troy',
  STABBR: 'MI'
}, insertHandler);

If you want to load i.e. CSV file, firstly you need to save data to a CSV in Node.js manually. Then, load it as a single column CSV using load() method. That will load the whole string as a single column.

Additionally, what I can recommend you is to use Dataflow templates, i.e. Cloud Storage Text to BigQuery, that read text files stored in Cloud Storage, transform them using a JavaScript User Defined Function (UDF), and output the result to BigQuery. But your data to load needs to be stored in Cloud Storage.