This txt file is giving me weird problems

Question

I have a problem with the handling of txt files, the source file is encoded 'UTF 8-WITHOUT BOM' and I tried to put many "encoding = " but I can't solve this..

Here I attach an image

Right is the origen of file and left is the result

This is the code.

import io
import time

result = io.open("Edificado/edificadoResultadoSinPorBlancos.txt","w")
start = time.time()
print(f"Empece en: {start}")

with io.open("Edificado/edificco.txt","r",errors="ignore") as f:
    for line in f:
        if '|' in line:
            line = line.replace("|","-")
        result.write(line)
result.close()

end = time.time()

print(f"Termine en: {end - start}")

(the file weighs 6gb)

Any idea how I could fix it?

this is the coding from the file

Please post everything as text and explain your problem more clearly. Also try everything on a tiny file to check if size is part of the problem — Caridorc, Jan 31 '23 at 19:15
Have you tried [`utf-8-sig`](https://stackoverflow.com/a/8898439/17200348)? — B Remmelzwaal, Jan 31 '23 at 19:19
You say you tried many encoding= but your sample code doesn't have one. — Mark Ransom, Jan 31 '23 at 19:27
@Itération122442 the problem is in the left have another character than right result and I need the same txt file — imnachox2, Jan 31 '23 at 20:22
@BRemmelzwaal yes, i tried from utf-8-sig to the same and too utf-8 to utf-8 — imnachox2, Jan 31 '23 at 20:23
Your input file is not UTF-8. The odd character displayed by Notepad++ is the result of a decoding error. Try different encodings in Notepad++ until you find one where every character makes sense, then use that in Python. — Mark Ransom, Jan 31 '23 at 21:14

geekay · Accepted Answer · 2023-01-31T19:59:49.990

0

you could just use "open" , io.open is an alias for "open"

import time

result = open("Edificado/edificadoResultadoSinPorBlancos.txt","w",encoding='utf-8',errors="surrogateescape")
start = time.time()
print(f"Empece en: {start}")

with open("Edificado/edificco.txt","r",errors="ignore",encoding='utf-8',errors="surrogateescape") as f:
    for line in f:
        if '|' in line:
            line = line.replace("|","-")
        result.write(line)
result.close()

end = time.time()

print(f"Termine en: {end - start}")

updated Alternatively based on mark ransoms comment

just work in binary instead

import time

result = open("Edificado/edificadoResultadoSinPorBlancos.txt","wb")
start = time.time()
print(f"Empece en: {start}")

with open("Edificado/edificco.txt","rb") as f:
    for line in f:
        if b'|' in line:
            line = line.replace(b"|",b"-")
        result.write(line)
result.close()

end = time.time()

print(f"Termine en: {end - start}")

edited Jan 31 '23 at 19:59

answered Jan 31 '23 at 19:20

geekay

340
1
5

You won't be able to use `encode` to write the file if you didn't open it in binary mode - unless you're still using Python 2. – Mark Ransom Jan 31 '23 at 19:25
you are right infact he could just work in binary – geekay Jan 31 '23 at 19:51
The whole point of adding the `encoding` option to `open` is so that you wouldn't have to worry about encoding everything by hand all the time, and it would be consistent. – Mark Ransom Jan 31 '23 at 19:52
in my solution... errors="surrogateescape" usually solves any problems.... my suggestion was in case he still faces some random problem. but his problem statement is easily solved in binary mode.. he doesnt have to bother with encoding at all.. he needs to just replace "|" with "-" line by line... and nothing else. – geekay Jan 31 '23 at 20:02
Thanks!, errors = "surrogateescape" is the solution!, thanks. – imnachox2 Feb 01 '23 at 12:46

This txt file is giving me weird problems

1 Answers1