1

I'm trying to append more than 2.5 million data points into a csv (which I open with Visual Studio Code). It's working for 1,738,142 data points, but no more than that...which is coincidentally the amount of rows Excel can handle (with my version/computer). Any idea why this is happening? Part of my code is below. I'm extremely new to this, so please tell me if I have to clarify. 

with open('dataset.csv', 'a', newline='') as file:
    writer = csv.writer(file)
    writer.writerow(["latitude", "longitude"])
    for y in range(upper, lower + 1):
        for x in range(left, right + 1):
            writer.writerow([(lat_difference * ((x - 3) / (right - left)) + east), \
            (long_difference * (y / (upper - lower)) + north)])
Rtrain
  • 45
  • 6
  • Are all the rows being [written to the csv](https://stackoverflow.com/questions/845058/how-to-get-line-count-of-a-large-file-cheaply-in-python)? – S.Au.Ra.B.H Dec 21 '19 at 00:17
  • If you try to read a csv into Excel that exceeds that limit on the number of row, it's not going to "work" — if that's what you mean. – martineau Dec 21 '19 at 00:33
  • The rows are not being written to the csv after those 1.7 mil data points...that's how it stops working. The line count is about 1.7 million. – Rtrain Dec 21 '19 at 00:33
  • @martineau I'm not trying to read it into excel...I open the csv directly in visual studio code. I opened it in Excel at one point just to check the number of lines, but then re-ran the code and created a different file and it's still not working. Do I automatically open it in Excel, for some reason? – Rtrain Dec 21 '19 at 00:36
  • It's awfully strange that it stops at exactly that number. Python's `csv` module module doesn't have that limitation.How can you tell it's exactly that number of rows? – martineau Dec 21 '19 at 00:36
  • @martineau I opened the file in VSCode. It only shows values for that many rows. – Rtrain Dec 21 '19 at 00:40
  • Well, since vs-code and Excel are written by the same company, it's highly likely they're both using the same software library with the same limitation as some level. Try opening the file as a plain text file — I'm not even sure what opening it in vs-code as anything else would accomplish anyway. – martineau Dec 21 '19 at 00:52
  • @martineau Interesting, I didn't know that...though a plain text file gives exactly the same rows of data. :/ Do you think it's likely a problem with my code? – Rtrain Dec 21 '19 at 00:58
  • @Rtrain: How are you opening it as “a plain text file”? You seem to have untrustworthy tools. – Davis Herring Dec 21 '19 at 02:04

0 Answers0