Get unique elements after parsing a column from a file in python

Question

I am working with Python 3.6. I have a tsv file which consists of 5 columns and > 100k of rows. I have used the split function to parse the file by the delimiter from which I receive specific columns with indices. The column which I am working on looks like this:

CSF3R
DNMT3A
DNMT3A
DNMT3A
DNMT3A
CBLB
PDGFRA
KIT
TET2
TET2
CUX1
CUX1
CUX1
CUX1
CUX1
CUX1
CUX1
CUX1
EZH2
EZH2
RAD21
ABL1
NOTCH1
NOTCH1
ETV6
ETV6
ETV6
FLT3
FLT3
TP53
TP53

What I need to do is get all the unique elements present in this column and print only one of them. I have tried plenty of functions such as join, set, tried to follow many other stackoverflow posts but none of them really solve my problem.

Plus the data I receive is in the 'str' format and not in a list. Therefore I tried getting all of them in a list as well, failing to do that as well. I cannot work with python pandas because all of my fellow associates do not have any idea on the package.

Therefore the normal procedure oriented code I tried is:

file=open('filename.txt')
next(file)
    stripped=()
    pos=()
    s="-"

    for line in file:
        stripped=line.strip()
        pos=stripped.split("\t")

        pos[2]= [y for y in (x.strip() for x in pos[2].splitlines()) if y]
        print(pos[2])

The output gives a list of all of the strings, that is each string is enclosed in a separate list and not in a single list.

From the above list, my expected output is:

CSF3R
DNMT3A
CBLB
PDGFRA
KIT
TET2
CUX1
EZH2
RAD21
ABL1
NOTCH1
ETV6
FLT3
TP53

This gives me only the unique elements.

To get the unique elements, do I have to get all of these in a single list or is there any other better way to do that?

The file that I am working on.

Answer 1

From this answer reading a text file columnwise and storing in a list in python:

with open('test.txt', 'r') as file:
    rows = [[str(x) for x in line.split('\t')] for line in file]
    cols = [list(col) for col in zip(*rows)]

for i in cols:
    print(set(i))

Answer 2

filename = "path/to/Post.txt"

with open(filename) as f:
    header = next(f)
    col = 2                                                # gene column
    unique_genes = {line.split()[col] for line in f.readlines()}

print(unique_genes)
# {'KIT', 'PDGFRA', 'CUX1', 'CBLB', 'DNMT3A', 'RAD21', 'CSF3R', 'NOTCH1', 'GENE', 'ABL1', 'TET2', 'EZH2'}

The steps for getting unique items from the 3rd column in your data is as follows:

1. Open the file (with)
2. Skip the header (next())
3. Iterate over the rows of the file (readlines)
4. Split the lines by the default delimiter e.g. tab (\t)
5. Only extract data from the third column ([col])
6. Return unique values on the extracted data (set comprehension, {...}).
7. Safely close the file (with)

Select a different column by changing the col value.

Answer 3

To convert the file to a list of strings, one per line use:

with open('filename.txt') as f:
    list_from_file = [x.strip() for x in f.readlines()]

print(set(list_from_file))

And for a five column file that is tab separated try:

with open('file1') as f:
    col1, col2, col3, col4, col5 = zip(
        *(y.split('\t') for y in (x.strip() for x in f.readlines())))

Then you can use set() on the desired columns

Answer 4

I think the easiest way to do this will definitely be by using a set. As you are currently using a list of lists: [[CSF3R],[DNMT3A],[DNMT3A], ...] you will be unable to use a set. If you parse your text-file into a single list of strings: [CSF3R, DNMT3A, DNMT3A, ...], you will be able to implement a set - problem solved!

You can just look at the implementations above for help. Also, if you want better help, just post the format of your text-file, so others can poke around and maybe find even better solutions.

All the best