error in writing to a file

Question

I have written a python script that calls unix sort using subprocess module. I am trying to sort a table based on two columns(2 and 6). Here is what I have done

sort_bt=open("sort_blast.txt",'w+')
sort_file_cmd="sort -k2,2 -k6,6n {0}".format(tab.name)
subprocess.call(sort_file_cmd,stdout=sort_bt,shell=True)

The output file however contains an incomplete line which produces an error when I parse the table but when I checked the entry in the input file given to sort the line looks perfect. I guess there is some problem when sort tries to write the result to the file specified but I am not sure how to solve it though.

The line looks like this in the input file

gi|191252805|ref|NM_001128633.1| Homo sapiens RIMS binding protein 3C (RIMBP3C), mRNA gnl|BL_ORD_ID|4614 gi|124487059|ref|NP_001074857.1| RIMS-binding protein 2 [Mus musculus] 103 2877 3176 846 941 1.0102e-07 138.0

In output file however only gi|19125 is printed. How do I solve this?

Any help will be appreciated.

Ram

score 0 · Answer 1 · answered Nov 09 '13 at 09:33

Using subprocess to call an external sorting tool seems quite silly considering that python has a built in method for sorting items.

Looking at your sample data, it appears to be structured data, with a | delimiter. Here's how you could open that file, and iterate over the results in python in a sorted manner:

def custom_sorter(first, second):
    """ A Custom Sort function which compares items
    based on the value in the 2nd and 6th columns. """
    # First, we break the line into a list
    first_items, second_items = first.split(u'|'), second.split(u'|')  # Split on the pipe character.
    if len(first_items) >= 6 and len(second_items) >= 6:
        # We have enough items to compare
        if (first_items[1], first_items[5]) > (second_items[1], second_items[5]):
            return 1
        elif (first_items[1], first_items[5]) < (second_items[1], second_items[5]):
            return -1
        else:  # They are the same
            return 0  # Order doesn't matter then
    else:
        return 0

with open(src_file_path, 'r') as src_file:
    data = src_file.read()  # Read in the src file all at once. Hope the file isn't too big!
    with open(dst_sorted_file_path, 'w+') as dst_sorted_file:
        for line in sorted(data.splitlines(), cmp = custom_sorter):  # Sort the data on the fly
            dst_sorted_file.write(line)  # Write the line to the dst_file.

FYI, this code may need some jiggling. I didn't test it too well.

it is not necessary to use `cmp` parameter. [`key` is a better alternative in many cases.](http://stackoverflow.com/a/19876440/4279) It processes each line only once. `cmp` is removed completely in Python 3. — jfs, Nov 09 '13 at 13:37

score 0 · Answer 2 · edited May 23 '17 at 12:05

What you see is probably the result of trying to write to the file from multiple processes simultaneously.

To emulate: sort -k2,2 -k6,6n ${tabname} > sort_blast.txt command in Python:

from subprocess import check_call

with open("sort_blast.txt",'wb') as output_file:
     check_call("sort -k2,2 -k6,6n".split() + [tab.name], stdout=output_file)

You can write it in pure Python e.g., for a small input file:

def custom_key(line):
    fields = line.split() # split line on any whitespace
    return fields[1], float(fields[5]) # Python uses zero-based indexing

with open(tab.name) as input_file, open("sort_blast.txt", 'w') as output_file:
     L = input_file.read().splitlines() # read from the input file
     L.sort(key=custom_key)             # sort it
     output_file.write("\n".join(L))    # write to the output file

If you need to sort a file that does not fit in memory; see Sorting text file by using Python

error in writing to a file

2 Answers2