Characters mess up during encoding of European Characters

Question

I am currently trying to output a csv file with a few European languages like German and Spanish. When I create the file, I need to specify an encoding in order to not get messed up output. I tried iso-8859 and cp1252, but there has to be at least a few characters that are messed up (meaning that they are not from the language). Code:

output = open('output.csv', 'w', newline="", encoding='iso_8859_1')

The input that I give to the file is from OpenAI's API.

Maybe this can help: https://stackoverflow.com/questions/436220/how-to-determine-the-encoding-of-text — Luis Alejandro Vargas Ramos, Sep 23 '22 at 12:43
@LuisAlejandroVargasRamos Well, this would be helpful if I am trying to determine the type of encoding. But I think that the encodings of my languages are different, resulting in some output being messed up. — Omar Zayed, Sep 23 '22 at 12:45

score 1 · Answer 1 · answered Sep 23 '22 at 12:44

1

Just use utf-8:

output = open('output.csv', 'w', newline="", encoding='utf-8')

answered Sep 23 '22 at 12:44

Damiaan

777
4
11

I did. But the problem is that the output becomes more messed up, giving me weird signs. – Omar Zayed Sep 23 '22 at 12:46
This is some example output. – Omar Zayed Sep 23 '22 at 12:47
@OmarZayed: just set the other editors to UTF-8. All of them should be able to cope with UTF-8 which it is in fact the standard encoding now. On Windows you may want to use UTF-8 with bon so `'utf-8-sig'` as Python encoding string. It mess up other tools. – Giacomo Catenazzi Sep 23 '22 at 12:56

score -1 · Answer 2 · answered Sep 23 '22 at 12:44

-1

I don´t know if this is valid for german, but its valid for spanish. Try "latin1" in the encoding

Hope it helps

answered Sep 23 '22 at 12:44

ssanga

335
3
11

This is some example output from the file. ؟Cَmo estلs? – Omar Zayed Sep 23 '22 at 12:49
I get an arabic in there for some reason. – Omar Zayed Sep 23 '22 at 12:51
Latin-1 is just an alias for ISO-8859-1 – tripleee Sep 23 '22 at 12:58
Can you write a couple of lines of your file, that way I can try with the encoding – ssanga Sep 26 '22 at 15:56
besides the encoding specified in the line opening the file. Can you try to add this before opening the file? os.environ["PYTHONIOENCODING"] = "latin1" – ssanga Sep 26 '22 at 15:57

Characters mess up during encoding of European Characters

2 Answers2