Python: compare two text files for same random lines

Question

I got a file wordlist.txt with 100 random words in it, each on a separate line. I currently use the following code to grab 12 random words from this file. To avoid picking exactly the same 12 words i want to builtin an extra check. The 12 random words are written to output.txt. How can i make my script compare the 12 random words (in the same order) with the 12 random words i have in output.txt (in 1 line)?

I currently use the following to read 12 random words from wordlist.txt and write them to output.txt:

teller = 0

while True:
    teller += 1

    #Choose 12 random words and write to textfile
    print "\nRound",teller
    f1=open('output.txt', 'w+')
    count = 0
    while (count<12):
        f1.write(random.choice([x.rstrip() for x in open('wordlist.txt')])+ " ")
        count += 1
    f1.close()

can you give me an example of it? or a link with explanation? — user2133342, Dec 04 '13 at 22:48
can you mail me? somehow the i can't figure out the edit... :( idokasse a(t) gmail — user2133342, Dec 04 '13 at 22:58
No, we cannot email you. Stack Overflow is about helping *other* people too. If they have the same or a similar question, they can read the answers posted *here*, but email to you can only be read by you. — Martijn Pieters, Dec 04 '13 at 23:02
Also, please don't try to try to give feedback to answers by editing the post. You can add comments *below* an answer. I've updated my answer to show an option to pick twelve new words that were not in `output.txt` before. — Martijn Pieters, Dec 04 '13 at 23:03

score 2 · Accepted Answer · edited May 23 '17 at 12:13

2

Instead of random.choice(), read all words into a list and use random.sample():

with open('wordlist.txt') as wlist:
    words = [w.strip() for w in wlist]
with open('output.txt', 'w') as output:
    for word in random.sample(words, 12):
        output.write(word + '\n')

random.sample() is guaranteed to pick 12 different words from your input list.

Because your wordlist is small (just 100 words), reading them all into a list in memory is more than fine.

If your input file is larger (megabytes to gigabytes), you may want to move to an algorithm that can pick a uniform sample out of any iterable regardless of size, only requiring memory for the output sample size.

If you need to find 12 random words that are not yet present in output.txt from a previous run, you need to read those into a set first:

with open('wordlist.txt') as wlist:
    words = [w.strip() for w in wlist]

with open('output.txt', 'r') as output:
    seen = {w.strip() for w in output}

with open('output.txt', 'a') as output:
    count = 0
    while count < 12:
        new_word = random.choice(words)
        if new_word in seen:
            words.remove(new_word)
            continue
        seen.add(new_word)
        output.write(new_word + '\n')
        count += 1

Here, I open the output.txt file with 'a' for appending instead, to add the new 12 words we had not yet seen before.

edited May 23 '17 at 12:13

Community

1
1

answered Dec 04 '13 at 22:48

Martijn Pieters

1,048,767
296
4,058
3,343

Thank you, but it seems i'm doing something wrong with indented blocks it says, i have it right now as this: http://pastebin.com/RYFnPP9H – user2133342 Dec 04 '13 at 23:14
Yes, the indentation there is entirely wrong. Don't mix tabs and spaces in your editor, and look carefully at the code; a `with` statement creates a new block, so lines below it are indented, etc. – Martijn Pieters Dec 04 '13 at 23:20
more like this? im sorry, i dont see whats wrong here.. http://pastebin.com/34YxP0t5 please tell me what i have wrong. i only have tabs, no spaces used. it tells me there is something wrong with line 20, but i cant figure it out? it is indented? – user2133342 Dec 04 '13 at 23:26
No, match the indentation in my answer. The three `with` statements are aligned. – Martijn Pieters Dec 04 '13 at 23:28
stil line 20 (words = [w.strip() for w in wlist] ) wrong when i match it like you code? – user2133342 Dec 04 '13 at 23:30

Python: compare two text files for same random lines

1 Answers1