0

I am trying to load a csv file to Mysql. The file has a million records so I am doing batch insertions of 1000. I am using python libraries sqlalchemy and pymysql. After first batch insertion I am getting error

pymysql.err.InternalError) (1366, "Incorrect string value: '\xC2\x83\xC2\xB7\xC3\xA5...'

for column 'FHCNM' at row 50. I wanted to skip that particular batch and insert remaining batches. I doing that with try..except..finally blocks in python. Here I am getting the same error (pymysql.err.InternalError) at different batches. How do I handle single exception multiple times? Right now, I am trying like

def functnname():
   try:
         #my code to insert data to mysql
   except:
         #to handle my error first time
         pass
   finally:
         #continue with next batch insertions
   return
try:
   functnname()
except:
   #to handle my error second time
finally:
    functnname()

I am trying out like this. I don't know if it works. Any more efficient way would be appreciated. And also how to handle the above error in first place and how insert the failed batches afterwards. Python 3.x.

James Z
  • 12,209
  • 10
  • 24
  • 44
learner
  • 1

1 Answers1

0

It would be hard to do in python code without inserting each element individually, but this will hurt your performance a lot. Use mysql features to ignore / handle errors during batch insertion.

RafalS
  • 5,834
  • 1
  • 20
  • 25