Python Duplicate Words in Text File

Question

I have a txt file

the text file looks like this:

Still not working.

['None']['Vega~']['Vega~']['Vega~']['8^)->-<']['violence']['puker']['Zanaz']['Funk']['8^)->-<']['8^)->-<']['8^)->-<']['Vega~']['violence']['Zanaz']['Funk']['puker']['Vega~']['Vega~']['Vega~']['8^)->-<']['violence']['puker']['Zanaz']['None']['Lawn']['Lawn']['Lawn']['Leafy']['Judge69']['David']['lilwade']['Pity.']['artofwar']['Hazecloud']['Lawn']['Lawn']['Lawn']['Judge69']['Leafy']['David']['lilwade']['Hazecloud']['Lawn']['Lawn']['Lawn']['Leafy']['David']['Pity.']['lilwade']['artofwar']['Judge69']

I need to remove all the duplicates so each name should only show one time, also it must keep the order they are in.

   fo = open('C:\Python26\myfile.txt','r')
   name_cache = fo.readlines()
   typea = name_cache[0]

   def unique_list(l):
      ulist = []
      [ulist.append(x) for x in l if x not in ulist]
      return ulist

   mast =' '.join(unique_list(typea.split()))
   print mast

And what have you done till now? This is not a code-writing service. If you are stuck on anything in your trials we can help. — Anand S Kumar, Jul 06 '15 at 04:15
possible duplicate of [How can I remove duplicate words in a string with Python?](http://stackoverflow.com/questions/7794208/how-can-i-remove-duplicate-words-in-a-string-with-python) — metatoaster, Jul 06 '15 at 04:16
How can you `split()` on this input? There doesn't seem to be any whitespace between each `['someName']` entry. — James, Jul 06 '15 at 04:28
No keeping the brackets isn't an issue just the order and the names. — user3496483, Jul 06 '15 at 04:32
Is there any whitespace in the text file at all? Or is it all in one line? — James, Jul 06 '15 at 04:32

James · Answer 1 · 2015-07-06T04:44:31.033

First remove the leading [ and trailing ]. Then split on ][. For example

>>> x="['None']['Vega~']['Vega~']"
>>> x.rstrip(']').lstrip('[').split('][') 
["'None'", "'Vega~'", "'Vega~'"]

Then call your unique_list.

>>> y = x.rstrip(']').lstrip('[').split('][') 
>>> unique_list(y)
["'None'", "'Vega~'"]

Then you can easily format it to whatever you want (i.e. to a string).

Note that rstrip and lstrip are each O(n). So it might be better to do x[1:-1]. This assumes you are 100% certain that the input is of the given form (starts with [ and ends with ])

This has the same O(n) time complexity as hashing every word (adding to a python set), but maintains the original order, and gets to use your (pretty neat) unique_list function.

Amr Ayman · Answer 2 · 2015-07-06T04:42:40.997

0

You can do this:

import collections
def unique_list(l): return list(OrderedSet(l))

Also, typea is just a string with no whitespace. To split names, do this:

typea = typea.replace('[', '').split(']') # typea is now a list

edited Jul 06 '15 at 04:42

answered Jul 06 '15 at 04:35

Amr Ayman

1,129
1
8
24

score 0 · Answer 3 · answered Jul 06 '15 at 04:41

s = "['None']['Vega~']['Vega~']['Vega~']['8^)->-<']['violence']['puker']['Zanaz']['Funk']['8^)->-<']['8^)->-<']['8^)->-<']['Vega~']['violence']['Zanaz']['Funk']['puker']['Vega~']['Vega~']['Vega~']['8^)->-<']['violence']['puker']['Zanaz']['None']['Lawn']['Lawn']['Lawn']['Leafy']['Judge69']['David']['lilwade']['Pity.']['artofwar']['Hazecloud']['Lawn']['Lawn']['Lawn']['Judge69']['Leafy']['David']['lilwade']['Hazecloud']['Lawn']['Lawn']['Lawn']['Leafy']['David']['Pity.']['lilwade']['artofwar']['Judge69']"
ss = s[1:-1]
l = []
for i in ss.split(']['):
    if i not in l:
        l.append(i)
r = ' '.join(l)

With the result:

"'None' 'Vega~' '8^)->-<' 'violence' 'puker' 'Zanaz' 'Funk' 'Lawn' 'Leafy' 'Judge69' 'David' 'lilwade' 'Pity.' 'artofwar' 'Hazecloud'"

score 0 · Answer 4 · answered Jul 06 '15 at 05:05

0

Solution that keeps the brackets around the names:

fo = open('myfile.txt','r')
name_cache = fo.readlines()[0]
names = []
for name in name_cache.replace('][', '],[').split(','):
    if name not in names:
        names.append(name)

print(names)

answered Jul 06 '15 at 05:05

Sina Khelil

2,001
1
18
27

Python Duplicate Words in Text File

4 Answers4