I wrote a python3 script that does some web scraping and stores some information on a CSV file. The script works fine on my computer. The problem happens when I try to run the script on a docker container. The error seems to be on this part of my code (simplified further for the purposes of this question).
# default CSV module
import csv
# this is how an ACTUAL row looks like in my program, included it in case it was important
row = {'title': 'Electrochemical sensor for the determination of dopamine in presence of high concentration of ascorbic acid using a Fullerene-C60 coated gold electrode', 'url': 'https://onlinelibrary.wiley.com/doi/abs/10.1002/elan.200704073', 'author': 'Goyal, Rajendra Nath and Gupta, Vinod Kumar and Bachheti, Neeta and Sharma, Ram Avatar', 'abstract': 'A fullerene‐C60‐modified gold electrode is employed for the determination of dopamine in the excess of ascorbic acid using square‐wave voltammetry. Based on its strong catalytic function towards the oxidation of dopamine and ascorbic acid, the overlapping voltammetric …', 'eprint': 'http://www.academia.edu/download/3909892/Dopamene.pdf', 'publisher': 'Wiley Online Library', 'year': '2008', 'pages': '757--764', 'number': '7', 'volume': '20', 'journal': 'Electroanalysis: An International Journal Devoted to Fundamental and Practical Aspects of Electroanalysis', 'ENTRYTYPE': 'article', 'ID': 'goyal2008electrochemical'}
# the CSV writer object
writer = csv.DictWriter("file.csv", fieldnames=[a, b, c], dialect='toMYSQL')
# this is the source of the problem!
writer.writerow(row)
I understand the containers have only the bare bones and that means that maybe the encoding the script uses is not supported. Thus, I added this to the start of my script: (bellow the usual she-bang)
# coding=utf-8
These are the locales on my docker:
$ locale -a
C
C.UTF-8
POSIX
en_US.utf8
es_CR.utf8
I have way more on my PC, but that shouldn't change much since en_US.utf8 covers all English stuff and es_CR.utf8 covers all Spanish stuff. (most, if not all, of my results are in English.)
I'm using python3, so I know all strings are unicode characters, maybe thats related to the problem?
$ python3 --version
Python 3.6.5
Despite all that, when I run my program, I get the following error message as soon as the script tries to print the row on console:
Exception in thread Thread-6:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/Systematic-Mapping-Engine/sysmapengine/scraper.py", line 100, in build_csv
writer.writerow(clean_row)
File "/usr/lib/python3.6/csv.py", line 155, in writerow
return self.writer.writerow(self._dict_to_list(rowdict))
UnicodeEncodeError: 'ascii' codec can't encode character '\u2010' in position 262: ordinal not in range(128)
