I have to handle a hugh data volume. Because of that I'd like to split my data into multiple csv files on my hdd (to process them separately).
The shape of my list is (1,81,141) and each of these list should represents a new row in my csv file.
Is it even possible to pour this kind of complex datastructure into csv? I've found this question, which demonstrates how to write data into csv. But the data are not multi-dimensional.
Example:
The files should contain data for single days. The shape is related to [timestamp][longitude][latitude] and refer to one calculated value.
I tried this one successfully:
import csv
import numpy as np
day1 = np.random.randint(255, size=(1, 81, 141))
with open('day1.csv', 'w', newline='') as csvfile:
writer = csv.writer(csvfile)
writer.writerows(day1)
When I open the files with Notepad++ I can see the separation with commas but not the different lines.
And when I read the data with this:
arr = [line.split(',') for line in open('day1.csv')]
print(arr[0])
I only get ['"[ 1 10 15 11 7 7 6 15 8 15 2 15 0 2 16 0 0 0 15 12 1 18 18 10 4\n'] and not the whole dataset
