How can I search a text file for a list of words from user input?

Question

I'm trying to make a simple word counter program in Python 3.4.1 where the user is to input a list of comma-separated words that are then analyzed for frequency in a sample text file.

I'm currently stuck on how to search for the entered list of words in the text file.

First I tried:

file = input("What file would you like to open? ")
f = open(file, 'r')
search = input("Enter the words you want to search for (separate with commas): ").lower().split(",")
search = [x.strip(' ') for x in search]
count = {}
for word in search:
    count[word] = count.get(word,0)+1
for word in sorted(count):
    print(word, count[word])

This resulted in:

What file would you like to open? twelve_days_of_fast_food.txt
Enter the words you want to search for (separate with commas): first, rings, the
first 1
rings 1
the 1

If that's anything to go by, I'm guessing this method only gave me the count of the words in the input list and not the count of the input list of words in the text file. So then I tried:

file = input("What file would you like to open? ")
f = open(file, 'r')
lines = f.readlines()
line = f.readline()
word = line.split()
search = input("Enter the words you want to search for (separate with commas): ").lower().split(",")
search = [x.strip(' ') for x in search]
count = {}
for word in lines:
    if word in search:
        count[word] = count.get(word,0)+1
for word in sorted(count):
    print(word, count[word])

This gave me nothing back. This is what happened:

What file would you like to open? twelve_days_of_fast_food.txt
Enter the words you want to search for (separate with commas): first, the, rings
>>>

What am I doing wrong? How can I fix this problem?

Martijn Pieters · Accepted Answer · 2014-11-23T03:29:29.773

You read all lines first (into lines, then tried to read just one line but the file already gave you all lines. In that case f.readline() gives you an empty line. From there on out your script is doomed to fail; you cannot count words in an empty line.

You can loop over the file instead:

file = input("What file would you like to open? ")

search = input("Enter the words you want to search for (separate with commas): ")
search = [word.strip() for word in search.lower().split(",")]

# create a dictionary for all search words, setting each count to 0
count = dict.fromkeys(search, 0)

with open(file, 'r') as f:
    for line in f:
        for word in line.lower().split():
            if word in count:
                # found a word you wanted to count, so count it
                count[word] += 1

The with statement uses the opened file object as a context manager; this just means it'll be closed again automatically when done.

The for line in f: loop iterates over each separate line in the input file; this is more efficient than using f.readlines() to read all lines into memory at once.

I also cleaned up your search word stripping a little, and set the count dictionary to one with all the search words pre-defined to 0; this makes the actual counting a little easier.

Because you now have a dictionary with all the search words, testing for matching words is best done against that dictionary. Testing against a dictionary is faster than testing against a list (the latter is a scan that takes longer the more words are in the list, while a dictionary test takes constant time on average, regardless of the number of items in the dictionary).

How about using [collections.Counter](https://docs.python.org/2/library/collections.html#collections.Counter) on file_.readlines().split(',') ? Hmm nope, still would need to iterate each line after all. But maybe collections.Counter(file_.read()) would come in handy? — brainovergrow, Nov 23 '14 at 03:32
@brainovergrow: `collections.Counter()` is a fine addition but requires an import and also would push the boundaries of the techniques the OP is already familiar with. — Martijn Pieters, Nov 23 '14 at 03:33
@brainovergrow: `collections.Counter(f.read().lower().split())` would do, then look up the counts for each search term in that. But filtering on search terms first is a fine approach too, as that takes less memory. — Martijn Pieters, Nov 23 '14 at 03:35
Right. Nice to know btw. that in Python 3.x "file" is no longer builtin :) (just leaving it here for other SO users, i'm pretty sure You already knew that.) — brainovergrow, Nov 23 '14 at 03:44
Made the edits and tested it out, worked like a charm and your explanations cleared up a lot of confusion for me. Thanks a bunch, really appreciate it! — SOrtiz122, Nov 23 '14 at 03:53

score 1 · Answer 2 · edited May 23 '17 at 12:28

1

You could try this;

import re
import collections

wanted = ["cat", "dog"]
matches = re.findall('\w+',open('hamlet.txt').read().lower())
counts = collections.Counter(matches) # Count each occurance of words
map(lambda x:(x,counts[x]),wanted) # Will print the counts for wanted words

I referenced this solution when forming the answer.

edited May 23 '17 at 12:28

Community

1
1

answered Nov 23 '14 at 03:42

Jerome Anthony

7,823
2
40
31

How can I search a text file for a list of words from user input?

2 Answers2