converting csv to txt (tab delimited) and iterate over files in directory python

Question

I have ~1000 files with with a two-column array with varying numbers of rows having the extension .csv. I need to read each line of the file, skip over the 1st header line, and then write all the contents as a tab-delimited .txt file. I have tried to write this myself in python. I want to change numbers.csv

X,Y
1,2
3,4
...

into

1[tab]2
3[tab]4
...

My code is below.

At the "next(csv_file)" operation, my program is reading me the error "StopIteration" The contents of the infile are erased. If I remove this line from my code, I still over-write the in-file but do nothing with an out-file.

Can someone help me get this going? `

import csv
import os

cwd = os.getcwd()
#print(cwd)

for file in os.listdir():

    file_name, file_ext = os.path.splitext(file)

    if file_ext == '.csv':
        with open(file,'r') as csv_file:
            csv_reader = csv.reader(csv_file)

            next(csv_file)

            for line in csv_reader:

            with open(file, 'w') as new_txt:    #new file has .txt extension
                txt_writer = csv.writer(line, delimiter = '\t') #writefile
                txt_writer.writerow(line)   #write the lines to file`

What is the purpose of `next(csv_file)` anyway? If it's giving `StopIteration`, then the file read pointer is at the end of the file. — jonrsharpe, Mar 06 '18 at 18:23
I was hoping to use this to skip over the first line in the file (which contains the header). — London Ashobo, Mar 06 '18 at 18:25
Typically you'd either do that by calling it on the reader (see e.g. https://stackoverflow.com/a/14257599/3001761) or on the file *before* creating the reader. But are you sure the file has content? — jonrsharpe, Mar 06 '18 at 18:28

hikerjobs · Accepted Answer · 2018-03-06T19:28:44.807

You were on the right track. I made some changes to your code:

import csv
import os

cwd = os.getcwd()
#print(cwd)

for file in os.listdir('.'):  # use the directory name here

    file_name, file_ext = os.path.splitext(file)

    if file_ext == '.csv':
        with open(file,'r') as csv_file:
            csv_reader = csv.reader(csv_file)

            csv_reader.next()  ## skip one line (the first one)

            newfile = file + '.txt'

            for line in csv_reader:
                with open(newfile, 'a') as new_txt:    #new file has .txt extn
                    txt_writer = csv.writer(new_txt, delimiter = '\t') #writefile
                    txt_writer.writerow(line)   #write the lines to file`

In writing, you need to use 'a' (append) instead of 'w' (write) , otherwise, you'll just get one line - the last line. And, if that last line was a blank line, all you'll have is a file with blanks, i.e., nothing.

score 0 · Answer 2 · answered Mar 06 '18 at 18:43

In[2]: import csv
In[3]: with open('test_file.txt', 'r') as f:
  ...:     for line in f:
  ...:         print(line)
  ...: 
X,Y

1,2

3,4

In[4]: with open('test_file.txt', 'r') as f:
  ...:     reader = csv.DictReader(f)
  ...:     fieldnames = reader.fieldnames
  ...:     result = list(reader)
  ...: 
  ...: with open('test_output.tsv', 'w') as f:
  ...:     writer = csv.DictWriter(f, fieldnames=fieldnames, delimiter='\t')
  ...:     writer.writeheader()  # remove this line if you don't want header
  ...:     writer.writerows(result)
  ...: 
In[5]: with open('test_output.tsv', 'r') as f:
  ...:     for line in f:
  ...:         print(line)
  ...: 
X   Y

1   2

3   4

converting csv to txt (tab delimited) and iterate over files in directory python

2 Answers2