Matplotlib plot excessively slow

Question

I'm trying to plot 20 million data points however it's taking an extremely long time (over an hour) using matplotlib,

Is there something in my code that is making this unusually slow?

import csv
import matplotlib.pyplot as plt
import numpy as np
import Tkinter
from Tkinter import *
import tkSimpleDialog
from tkFileDialog import askopenfilename

plt.clf()

root = Tk()
root.withdraw() 
listofparts = askopenfilename()                  # asks user to select file
root.destroy()

my_list1 = []
my_list2 = []
k = 0

csv_file = open(listofparts, 'rb')

for line in open(listofparts, 'rb'):
    current_part1 = line.split(',')[0]
    current_part2 = line.split(',')[1]
    k = k + 1
    if k >= 2:                                   # skips the first line
        my_list1.append(current_part1)
        my_list2.append(current_part2)

csv_file.close()

plt.plot(my_list1 * 10, 'r')
plt.plot(my_list2 * 10, 'g')

plt.show()
plt.close()

Answer 1

There is no reason whatsoever to have a line plot of 20000000 points in matplotlib.

Let's consider printing first: The maximum figure size in matplotlib is 50 inch. Even having a high-tech plotter with 3600 dpi would give a maximum number of 50*3600 = 180000 points which are resolvable.

For screen applications it's even less: Even a high-tech 4k screen has a limited resolution of 4000 pixels. Even if one uses aliasing effects, there are a maximum of ~3 points per pixel that would still be distinguishable for the human eye. Result: maximum of 12000 points makes sense.

Therefore the question you are asking rather needs to be: How do I subsample my 20000000 data points to a set of points that still produces the same image on paper or screen.

The solution to this strongly depends on the nature of the data. If it is sufficiently smooth, you can just take every nth list entry.

sample = data[::n]

If there are high frequency components which need to be resolved, this would require more sophisticated techniques, which will again depend on how the data looks like.

One such technique might be the one shown in How can I subsample an array according to its density? (Remove frequent values, keep rare ones).

Answer 2

The following approach might give you a small improvement. It removes doing the split twice per row (by using Python's CSV library) and also removes the if statement by skipping over the two header lines before doing the loop:

import matplotlib.pyplot as plt
import csv

l1, l2 = [], []

with open('input.csv', 'rb') as f_input:
    csv_input = csv.reader(f_input)

    # Skip two header lines
    next(csv_input)
    next(csv_input)

    for cols in csv_input:
        l1.append(cols[0])
        l2.append(cols[1])

plt.plot(l1, 'r')
plt.plot(l2, 'g')
plt.show()

I would say the main slow down though will still be the plot itself.

Answer 3

I would recommend switching to pyqtgraph. I switched to it because of speed issues while I was trying to make matplotlib plot real time data. Worked like a charm. Here's my real time plotting example.