1

I am trying to use the bq CLI to create a BigQuery Data Transfer Service following the instructions here, but for some reason, when I run this in the PowerShell I keep getting this error:

The filename, directory name, or volume label syntax is incorrect.

Here is my command

bq mk --transfer_config --project_id=ingestion-test --data_source=google_cloud_storage --display_name='DTA' --target_dataset=Data --params='{"data_path_template":"gs://mydata-766451231152/a_*","destination_table_name_template":"alpha","file_format":"CSV","max_bad_records":"0","ignore_unknown_values":"true","field_delimiter":"|","skip_leading_rows":"0","allow_quoted_newlines":"true","allow_jagged_rows":"false","delete_source_files":"false"}' --data_source=google_cloud_storage

I have tried all the normal things like escaping double quotes with \, "", etc. It seems the error is in the --params part because without that, the command actually goes through, but obviously fails because --params is required. All examples I have seen online show the same syntax.

What is wrong?

Maxim
  • 4,075
  • 1
  • 14
  • 23
AIK DO
  • 288
  • 1
  • 4
  • 13
  • The command you provided works correctly on my end (changing the bucket, dataset and project ID of course). What's the Cloud SDK version you're using (check that with `gcloud info`)? Also you're specifying `--data_source` twice there - I don't think it's related but I doubt it's needed... I reckon you can remove the last one. – Maxim Jan 18 '20 at 06:00
  • Also, since you mention it fails when using the PowerShell, does it work when you run it through the Cloud Shell for example? It could be something more related to Windows. – Maxim Jan 18 '20 at 06:12
  • Is bq a .cmd file? Can --params be taken from a file? It looks like json. – js2010 Jan 18 '20 at 06:48
  • It's PowerShell, unfortunately: sadly, you must manually `\ `-escape the `"` inside your `'...'` string - see the linked post and in particular [this answer](https://stackoverflow.com/a/55604316/45375). – mklement0 Jan 18 '20 at 12:57
  • Aah this was frustrating. What finally worked was escaping the | delimiter with ^. – AIK DO Jan 18 '20 at 18:48
  • ^ is a cmd quote symbol. I suspect that bq is a .bat script. They should make a .ps1 version. – js2010 Jan 18 '20 at 19:31

0 Answers0