How to save real-time predicted results into a csv file?

Question

I am working on an audio classifier project and currently when I run the predictor in realtime, it captures the a small segments of audio then starts the prediction process then spills out the results through the terminal like this:

[INFO] 2020-08-07 14:49:20,695: Predictions: Dog: 1.00
[INFO] 2020-08-07 14:49:20,698: Stop processing.
[INFO] 2020-08-07 14:49:21,707: Start processing.
[INFO] 2020-08-07 14:49:21,847: Predictions: Cat: 0.98
[INFO] 2020-08-07 14:49:21,849: Stop processing.

So, what I want is to have these results saved into a CSV file.

What I did so far:

Here is my script:

def _process_loop(self):

with WavProcessor() as proc:
    self._ask_data.set()
    while True:
        if self._process_buf is None:

            time.sleep(self._processor_sleep_time)                                          #Waiting for data to process
            continue

        logger.info('Start processing.')
        predictions = proc.get_predictions(

            self._sample_rate, self._process_buf)
        logger.info(

            'Predictions: {}'.format(format_predictions(predictions))
        )

        prediction = pd.DataFrame(predictions, columns=['Predicted Class', 'Accuracy']).to_csv('predictions.csv')


        logger.info('Stop processing.')
        self._process_buf = None
        self._ask_data.set()

I added this line to my code (prediction = pd.DataFrame(predictions, columns=['Predicted Class', 'Accuracy']).to_csv('predictions.csv')) as you can it in the above code, and started getting something here. The thing that I got was only one result saved inside the csv file. Specifically, it saves only the last detected result.

Predicted Class          Accuracy
Cat                      0.98

So, my current problem is how to save all predicted results inside my CSV file.

I appreciate if anyone can help me to solve this problem.

Checkout the first answer here. https://stackoverflow.com/questions/17530542/how-to-add-pandas-data-to-an-existing-csv-file — Rashid 'Lee' Ibrahim, Aug 07 '20 at 19:06

score 1 · Accepted Answer · answered Aug 07 '20 at 19:12

1

You are rewriting the file on each iteration. You can specify the write mode of the pandas to_csv function. You can append to an existing file if you set it to a.

Something like this:

pd.DataFrame(predictions, columns=['Predicted Class', 'Accuracy']).to_csv('predictions.csv', mode='a', header=False)

Some nice references:

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html

https://tutorial.eyehunts.com/python/python-file-modes-open-write-append-r-r-w-w-x-etc/

answered Aug 07 '20 at 19:12

paulopontesm

260
1
4

1

Thank you sooo much for helping me out. That definitely solved it. Also, thanks for the references. – solo Aug 07 '20 at 19:26
So, I was trying since yesterday to get the Date and Time info as shown above written next to the predicted results but couldn't make. I tried to follow through the references you provided, thankfully, and used the .at_time but that didn't get me anywhere. Do you think that is possible to do? Since the original passed data had 2 columns only, and the date and time info are generated somehow.. not sure how! – solo Aug 11 '20 at 15:20
You mean like add a column to your dataframe with `datetime.now()` ? Like this: https://www.jquery-az.com/python-datetime-now/ – paulopontesm Aug 17 '20 at 09:00
actually, it is already showing the time and date with the results as you can see in my above shell results ([INFO] 2020-08-07 14:49:20,695: Predictions: Dog: 1.0)... but when I try to add the date & time to a separate column it gives me an error saying: ValueError: 3 columns passed, passed data had 2 columns – solo Aug 17 '20 at 13:53

How to save real-time predicted results into a csv file?

1 Answers1