Removing a small number of lines from a large file

Question

I have a very large text file, where most of the lines are composed of ASCII characters, but a small fraction of lines have non-ASCII characters. What is the fastest way to create a new text file containing only the ASCII lines? Right now I am checking each character in each line to see if it's ASCII, and writing each line to the new file if all the characters are ASCII, but this method is rather slow. Also, I am using Python, but would be open to using other languages in the future.

Edit: updated with code

#!/usr/bin/python

import string

def isAscii(s):
    for c in s:
        if ord(c) > 127 or ord(c) < 0:
            return False
    return True

f = open('data.tsv')
g = open('data-ASCII-only.tsv', 'w')

linenumber = 1
for line in f:
    if isAscii(line):
        g.write(line)
    linenumber += 1

f.close()
g.close()

possible duplicate of [How to check if a string in Python is in ASCII?](http://stackoverflow.com/questions/196345/how-to-check-if-a-string-in-python-is-in-ascii) — Ruben Bermudez, Jun 24 '14 at 18:25
The way you're doing it is pretty much what you have to do. All lines must be read; all characters must be checked; and all wanted lines must be written. There aren't really any algorithmic shortcuts here. Show us your code and we can look for inefficiencies. — Tom Zych, Jun 24 '14 at 18:27
@TomZych Trying to encode the whole line to 'ascii' inside a try/except might be faster than checking each character individually. See the question RubenBermudez linked to. — dano, Jun 24 '14 at 18:29
@dano - Yes, I saw that. Of course, that's still checking all characters. You're just doing it in C. — Tom Zych, Jun 24 '14 at 22:36
It occurs to me that we cannot answer this question without knowing how the file is encoded. The simple approach of checking for characters < 0x80 will work with UTF-8; it won't work with UTF-7 or UTF-16. You can't even read the line correctly in UTF-16 if you treat it as ASCII; the newline is 0x0A 0x00. — Tom Zych, Jun 24 '14 at 23:12

score 1 · Accepted Answer · edited May 23 '17 at 12:10

1

You can use grep: "-v" keeps the opposite, -P uses perl regex syntax, and [\x80-\xFF] is the character range for non-ascii.

grep -vP "[\x80-\xFF]" data.tsv > data-ASCII-only.tsv

See this question How do I grep for all non-ASCII characters in UNIX for more about search for ascii characters with grep.

edited May 23 '17 at 12:10

Community

1
1

answered Jun 24 '14 at 18:59

Jonathan Villemaire-Krajden

1,418
10
22

1

Possibly helpful link: http://stackoverflow.com/questions/11577720/remove-lines-that-contain-non-english-ascii-characters-from-a-file – Alec Jun 24 '14 at 19:03
I was about to post a similar answer, but expanding the end range to xFFFF. – pferate Jun 24 '14 at 19:10
@pferate grep doesn't do multi-byte character merging, but the first byte should trigger this anyways. – Jonathan Villemaire-Krajden Jun 24 '14 at 19:18

Brenda J. Butler · Answer 2 · 2014-06-24T19:16:15.533

0

The following suggestion uses a command-line filter (ie, you would use it on the shell command line), this example works in a shell on linux or unix systems, maybe OSX too (I've heard OSX is BSDish):

$ cat big_file | tr -dc '\000-\177' > big_file_ascii_only

It uses the "tr" (translate) filter. In this case, we are telling tr to "delete" all characters which are outside the range octal-000 to octal-177. You may wish to tweak the charcter set - check the man page for tr to get some ideas on other ways to specify the characters you want to keep (or delete)

edited Jun 24 '14 at 19:16

answered Jun 24 '14 at 19:10

Brenda J. Butler

1,475
11
20

This will not remove the entire line as asked but just delete the non-ascii characters, leaving the rest of the line in place. – Jonathan Villemaire-Krajden Jun 24 '14 at 19:15
That's true, I misread the question. In that case the answer with grep (by Jonathan Villemaire-Krajde) is the way to go – Brenda J. Butler Jun 24 '14 at 19:17

score 0 · Answer 3 · answered Jun 24 '14 at 23:47

The other approaches given will work if, and only if, the file is encoded in such a way that "non-ASCII" is equivalent to "high bit set", such as Latin-1 or UTF-8. Here's a program in Python 3 that will work with any encoding.

#!/usr/bin/env python3

import codecs

in_fname = "utf16file"
in_encoding = "utf-16"
out_fname = "ascii_lines"
out_encoding = "ascii"

def is_ascii(s):
    try:
        s.encode("ascii")
    except UnicodeEncodeError:
        return False
    return True

f_in = codecs.open(in_fname, "r", in_encoding)
f_out = codecs.open(out_fname, "w", out_encoding)

for s in f_in:
    if is_ascii(s):
        f_out.write(s)

f_in.close()
f_out.close()

Removing a small number of lines from a large file

3 Answers3