0

Faing issue loading below row into BQ. I have already tried suggestion given on stackoverflow but no luck. Below is my data:

Id;description;curator_id;Q_code;ITG_NUM ABC;”weather is good for skiing \”B\””;ROLLON;HFCKQ:5051 CDE;"NO wonder it is good"B"";;MOVEON;CKHLP;5058

I have added below parameters as suggested:

job_config = bigquery.LoadJobConfig(
        autodetect=True,
        source_format=bigquery.SourceFormat.CSV,
        quote_character = '"',
        encoding='UTF-8',
        field_delimiter=';',
        write_disposition="WRITE_APPEND",
        skip_leading_rows=1,
    )

But still getting error: Error detected while parsing row starting at position: 155482. Error: Data between close double quote (") and field separator.

Any suggestion?

exploding_data
  • 317
  • 1
  • 14
  • Both the quoted B look dodgy. The double quote should be doubled not backslash-scaped. I.e. “blah-di-blah “”B””” - see https://stackoverflow.com/questions/17808511/properly-escape-a-double-quote-in-csv#17808731 – DisappointedByUnaccountableMod Aug 16 '21 at 19:16
  • Does this answer your question? [Properly escape a double quote in CSV](https://stackoverflow.com/questions/17808511/properly-escape-a-double-quote-in-csv) – DisappointedByUnaccountableMod Aug 16 '21 at 19:20

1 Answers1

1

If you open the data file in Notepad++ with Word Wrap turned off, press Ctrl G, then select "Offset" radio button and then you can put your position # 155482 to search. This will take you to the position where the error resides/is detected. Most likely, it's an embedded double-quote in the data field (between your comma delimiters).

Dharman
  • 30,962
  • 25
  • 85
  • 135
cd2
  • 28
  • 3
  • so i am trying to replace using below command but it is not .Any suggestion:for line in reading_file: stripped_line = line.strip() new_line = stripped_line.replace('\"', '"') reading_file.close() new_file_content += new_line+"\n" writing_file = open("file", "w") writing_file.write(new_file_content) writing_file.close() – exploding_data Aug 18 '21 at 04:12
  • answer here https://stackoverflow.com/questions/68826512/replace-with-in-python-files – exploding_data Aug 18 '21 at 12:20