0

Python 3.4.2

i'm using the Client interface for Scrapinghub API, which can be found here:

https://github.com/scrapinghub/python-scrapinghub

I Scrape a site and want get and print the items with

for item in job.items():
  print(item)

In a python interpreter this works fine but when i export my code in a script (f.e. 'test.py' and run it with 'python3 test.py' an error occures, which says:

Traceback (most recent call last):
File "test.py", line 24, in <module>
print(insert_statement)
UnicodeEncodeError: 'ascii' codec can't encode character '\xdf' in position 247: ordinal not in range(128)

The text of line 24 is sth like this : [ ... ] Tobias Weiß [...]

I already run 'dpkg-reconfigure locales' and set it to 'de_DE.UTF-8 UTF-8' but it does not seems to be the problem.

Do i have to convert the item dict to utf8 for every result or is there another solution i don't see.

from scrapinghub import Connection

conn = Connection('1234567890')
project = conn[123456]



jobs = project.jobs(state='finished')

for job in jobs:
    if "consumed" not in job.info['tags']:
            for item in job.items():
                   print(item)

Regards

paul trmbrth
  • 20,518
  • 4
  • 53
  • 66
fuser60596
  • 1,087
  • 1
  • 12
  • 26
  • try adding this at the beginning of the file: `# -*- coding: utf-8 -*-` – eLRuLL Mar 17 '17 at 15:02
  • Which version of python? – Ilja Everilä Mar 17 '17 at 15:03
  • The problem is that Python considers your console interface to be ASCII only. Not sure offhand how to fix that. – Mark Ransom Mar 17 '17 at 15:04
  • Possible duplicate of [How to make the python interpreter correctly handle non-ASCII characters in string operations?](http://stackoverflow.com/questions/1342000/how-to-make-the-python-interpreter-correctly-handle-non-ascii-characters-in-stri) – Taku Mar 17 '17 at 15:10
  • In the python interpreter the code runs fine but i exported it to a script file 'test.py' and run it with 'python3 test.py' - there the error occures. – fuser60596 Mar 17 '17 at 15:28

1 Answers1

2

I found a solution, which seems to work.

In the shell i had to type:

export LC_ALL=C.UTF-8
export LANG=C.UTF-8

Also i followed instrcutions from here: https://blog.scrapinghub.com/2016/08/17/introducing-scrapy-cloud-with-python-3-support/

Introducing Scrapy Cloud with Python 3 Support

Thanks to all anyways!

fuser60596
  • 1,087
  • 1
  • 12
  • 26