Writing Output of Python Program to .txt File

Question

I've written a program that reads a CSV file and outputs the contents as insert statements. I then wrote an execute program that should take the output of the CSV parser program and write it to a .txt file but instead of writing the entire output it only writes the first statement.

Here is the code for the executor:

import sys

with open('insert.txt', 'wb') as f:
    subprocess.check_call(["python", "CSVParserMultiple.py"], stdout=f)

And the code for the parser:

import csv, os

path = 'C:/Users/user/Desktop/test/'
for file in os.listdir(path):
    if file.endswith('.csv'):
# print ('Parsing file: ' + os.path.basename(path + file))
        openFile = open(path + file)
        csvFile = csv.reader(openFile)
        getHeader = next(csvFile)
        columnHeaders = map((lambda x: "'" + x + "'"), getHeader[:-1])
        insert = 'INSERT INTO testing (' + "'ID', 'VehicleID', " + ', '.join(columnHeaders) + ') VALUES '
        for row in csvFile:
            values = map((lambda x: "'" + x.strip() + "'"), row[:-1])
            print (insert + "(" + ", ".join(values) + ");")
        openFile.close()

I'm not entirely sure that it makes sense to have them as two separate programs but I could not get them to run as defined functions in the same program for the life of me. How can I get the execution program to output all the lines of the parser program instead of just a single line? And how can I combine them into one program?

You should place those operations in functions. If you want to be able to reuse modules and call them from other scripts, you need to install them by creating a python package/module. — Brendan Abel, Jul 15 '16 at 20:27
Calling a python file via subprocess is pretty ridiculous. Don't do that. If you *need* to separate it out and a different function isn't good enough, then create a [module of your own](https://docs.python.org/3/tutorial/modules.html) — Wayne Werner, Jul 15 '16 at 20:29
Not sure why only the first line is showing up in the output file. But to combine the two scripts, just put the `with open` statement in the second script, and then use `f.write('something\n')` instead of `print`. — John Gordon, Jul 15 '16 at 20:35

score 0 · Answer 1 · edited May 23 '17 at 10:28

Not sure if this would work for you but you can use the >/>> operator to have stdout be re-routed to a file.

Edit: The difference between > and >> is that >> appends to the end of the file while > truncates the file

$python program.py >> output.txt

To combine the programs you could have the executor program be defined as the main function by defining it as

def main():
    <executor code here>
if __name__ == "__main__":
     main()

Then you can redirect stdout to a file by using

sys.stdout = open("file",'w')

Similar question: Redirect stdout to a file in Python?

WombatPM · Accepted Answer · 2016-07-15T21:38:05.880

0

You are making things more complicated than they need to be. Just nest your open statements using with. One program. And it prints to the screen and writes to a file.

import csv, os

path = 'C:/Users/user/Desktop/test/'
for file in os.listdir(path):
    if file.endswith('.csv'):
        # print ('Parsing file: ' + os.path.basename(path + file))
        with open(path + file) as infile:
            with open(path+file+".txt",'w') as outfile:
                csvFile = csv.reader(infile)
                getHeader = next(csvFile)
                columnHeaders = map((lambda x: "'" + x + "'"), getHeader[:-1])
                insert = 'INSERT INTO testing (' + "'ID', 'VehicleID', " + ', '.join(columnHeaders) + ') VALUES '
                for row in csvFile:
                    values = map((lambda x: "'" + x.strip() + "'"), row[:-1])
                    print (insert + "(" + ", ".join(values) + ");")
                    outfile.write(insert + "(" + ", ".join(values) + ");" + "\n")

edited Jul 15 '16 at 21:38

answered Jul 15 '16 at 20:43

WombatPM

2,561
2
22
22

Two small errors, otherwise this is exactly what I was looking for. The first being to change `with open(path + file + ".txt") as outfile:` to `with open(path + file + ".txt", "w") as outfile:` to allow for write privileges. And the second is that the proper output is displayed in the shell but in the .txt file only the header is printed with no subsequent values afterwards. I am not quite sure how to fix this one though, but thank you so much for the help. – ThoseKind Jul 15 '16 at 21:07
You might be having a problem with the ", ".join statement if your values are numeric and not string. I'd print out "row" and "values" to see if the conversion is going as expected. And I'd use a test file that only had 5 rows of data in it. That way any error messages wont be lost due to scrolling. – WombatPM Jul 15 '16 at 21:46
`print(row)` shows that each of the values are indeed strings (for ex. - `'11:47'`) although they contain numbers. `print(values)` results in ``. I am not sure what this means but seeing that the `(insert + "(" + ", ".join(values) + ");")` statement outputs properly in shell I would assume that both `row` and `values` are working properly. The error would then have to be within `.write` itself (whether it be syntax, although I copied it exactly from the `print` statement, or something else) or in the `with` statement that defines `outFile` above. – ThoseKind Jul 15 '16 at 22:03
For whatever reason getting rid of the `print` statement before doing `outFile.write` works. No idea why. – ThoseKind Jul 15 '16 at 22:14
I figured it out. It's because `lambda` is a temporary function. This can be solved two different ways. The first being to define `values` without using a `lambda` function or the much simpler solution is to just paste the `lambda` function again after the `.write` method has been used and then writing a print function. – ThoseKind Jul 15 '16 at 22:30

Writing Output of Python Program to .txt File

2 Answers2