How to perform data cleaning for a text file?

Question

I have a text file with lots of lines, including words and numbers, here is an example:

2021-12-06 05:07:09.266 INFO: Additional  ID 1638301749791
2021-12-06 05:07:09.266 INFO: Found 
2021-12-06 05:07:09.267 INFO: ObjectStatus-ok factor 1163 factor five and six computed as it was before best weight ID 1638301749796
2021-12-06 05:07:09.267 INFO: disabled; computing power weight factor factor 19025.
2021-12-06 05:07:10.041 INFO: Wrote big factor 0.3568357342, Classificationfactortype-fail
2021-12-06 05:07:10.042 DEBUG: Duiu.0.0.2588650814
2021-12-06 05:07:10.743 INFO: Wrote .3254806495

My question is how can I keep lines that have particular word"Classificationfactortype-fail" and "ObjectStatus-ok", and delete all other lines? I would like to save the new text file in the directory.

Here is the code that I wrote:

ans = []

with open('test. txt') as rf:
    for line in rf:
        line = line.strip()
        if "Classificationfactortype-fail" in line or "ObjectStatus-ok" in line:
          ans.append(line)

with open('extracted_data.txt', 'w') as wf:
    for line in ans:
        wf.write(line)

Does this answer your question? [Python - Check If Word Is In A String](https://stackoverflow.com/questions/5319922/python-check-if-word-is-in-a-string) — Mike Scotty, Dec 13 '21 at 21:56
Does this answer your question? [Does Python have a string 'contains' substring method?](https://stackoverflow.com/questions/3437059/does-python-have-a-string-contains-substring-method) — SuperStormer, Dec 13 '21 at 23:14

score 2 · Accepted Answer · answered Dec 13 '21 at 22:00

2

If each line starts with the timecode, then str.startswith() won't work.

You can simply do:

if "Classificationfactortype-fail" in line or "ObjectStatus-ok" in line:
   ans.append(line)

in your first loop.

answered Dec 13 '21 at 22:00

Cinematic Galaxy Ita

72
1
7

Exactly. Also, your first statement should be `with open('test. txt', 'r') as rf`, and your `for` loop needs to be `for line in rf.readlines()`. – Marco Couto Dec 13 '21 at 22:03
1

@MarcoCouto indeed that's good practise, but they're not strictly necessary: **open()**'s default mode is already **"r"** and the loop through the file returns each line. But yes, it makes the code easier to understand. – Cinematic Galaxy Ita Dec 13 '21 at 22:08
It writes everything in a single line, in my new txt file. How do I make it to write separately? – nikki Dec 13 '21 at 22:28
What if you append a newline character when writing each line to the file? – Marco Couto Dec 13 '21 at 22:42
2

@MarcoCouto https://stupidpythonideas.blogspot.com/2013/06/readlines-considered-silly.html – SuperStormer Dec 13 '21 at 23:14
1

@nikki Marco Couto is right: unlike **print()**, **file.write()** does not add a **newline** character at the end, so you have to add it manually either in the first loop `ans.append(line + "\n")` **OR** when writing the new file: `wf.write(line + "\n")` – Cinematic Galaxy Ita Dec 14 '21 at 10:21

How to perform data cleaning for a text file?

1 Answers1