0

Google Play Console reports are using UTF-16 encoding, BigQuery - UTF-8.

How I can automatic converting the CSV files from UTF-16 to UTF-8?

My code in PowerShell:

$date = (Get-Date).AddDays(-2).Date.ToString('yyyy-MM') 
$date2 = $date.Replace('-', '')

$typefile = 'app_version'

$table = $typefile + '$' + $date2 + '01'
$csv_file = 'gs://pubsite_prod_rev_******_'+ $date2 + '_' + $typefile + '.csv'
$csv_file2 = $date2 + '_' + $typefile + '.csv'

& gsutil cp $csv_file C:\***\Scripts\gc\$csv_file2
& bq load --replace report.$table C:\***\Scripts\gc\$csv_file2

Error:

BigQuery error in load operation: Error processing job
'majestic-cairn-****:bqjob_r171ebea2_*****_1': Error while reading
data, error message: CSV table encountered too many errors, giving up. Rows: 1;
errors: 1. Please look into the error stream for more details.
Failure details:
- file-00000000: Error while reading data, error message: Too many
values in row starting at position: 0.
Aleksandr Osiyuk
  • 1
  • 2

1 Answers1

0

As TheIncorrigible1 mentioned, can use Powershell to do the encoding conversion

(Get-Content -Path $Path) | Out-File -FilePath $Path -Encoding UTF8

this command will convert CSV files from UTF-16 to UTF-8 in. You may have to specify the encoding in the first command as this 

(Get-Content -Path $Path -Encoding UTF16) | Out-File -FilePath $Path -Encoding UTF8`

There is a detailed answer from ajk in this post

enle lin
  • 1,664
  • 8
  • 14